Lawyers for Andrew William Winterton, the construction site manager and director of Conquest Homes, told the Court of Appeal that the case against him was so weak it should never have gone to trial.
But Lady Justice Macur and two other judges sitting at the Court of Appeal in London dismissed his challenge on 6 November.
Winterton had been jailed for four years after being found guilty of gross negligence manslaughter at the end of a nine-week trial at Northampton Crown Court on 30 June 2017.
He also received a one-year prison sentence to run concurrently for breaching s 7 of the Health and Safety at Work Act after failing to take reasonable care for the safety of himself and others affected by his actions. In addition, he was served two eight-month prison sentences, also to run concurrently, for "causing a corporate body to commit an offence".
The case centred on the death of Shane Wilkinson, a ground worker at the Conquest development in Collyweston, Northamptonshire, where nine homes were being built.
On 4 September 2014, only a few days after Wilkinson had arrived on site, the ground worker was standing near the edge of an unshored trench when its wall collapsed, burying him under earth and rubble.
Wilkinson sustained severe blunt trauma to his head, which caused a large skull fracture. He was pronounced dead at the scene.
The original trial was told that Winterton, who was responsible for site safety, had ignored basic measures.
The Crown Prosecution Service (CPS) had argued that Winterton was on site daily and should have been aware that the trenches on the development were not being battered back or that other precautions were not being taken to prevent collapse.
The excavation digger-driver, Dean Wortley, who traded as Clearview Demolition, had not properly secured the sides of the trench. He was handed a one-year jail term.
The CPS said Wilkinson had told friends of at least two previous collapses in the trench, yet nothing had been done in response.
At the appeal, Winterton's lawyers claimed the director had never seen the finished trench and was unaware of the danger it posed.
However, Lady Justice Macur said: "There was evidence from which the jury could conclude that [Winterton] was actually aware of the method of the excavation and that it was dangerous and there was a serious risk of death."
She added that photographic evidence presented to the jury "clearly demonstrated the dangerous workmanship that posed a real and significant risk of death.
"It was a question of 'when' not 'if' the trench would collapse, and this was or should have been apparent to anybody."
