Keith Cochrane has halted his role at the Scotland Office while the firm’s downfall is being investigated.
Mr Mundell told MPs that the interim Carillion boss did not wish “to become a distraction”.
Speaking at Scotland Questions in the House of Commons, Mr Mundell said: “It is not correct to suggest that non-executive directors take policy decisions in relation to government departments.
“Keith Cochrane has done an excellent job as a non-executive director of the Scotland Office and I pay tribute to him as one of Scotland’s most respected businessmen.
“However, he has decided himself, in order not to become a distraction at a time of very important work for the Scotland Office, that he will step aside from his responsibilities until the investigation into Carillion and any subsequent enquiries are complete.”
Mr Mundell also came under fire for his failure to secure amendments to clause 11 of the EU (Withdrawal) Bill - with SNP MPs calling for him to resign.
Clause 11, which was voted through by the Commons at the end of last year, established that powers currently exercised by devolved governments within EU frameworks would be returned to Westminster.
The Scottish and Welsh governments had called the clause a power grab by London on devolved matters.
At the time, Mr Mundell pledged that amendments would be made to satisfy the devolved authorities but due to a “lack of time” the amendments were deferred to the House of Lords.
SNP MP Mhairi Black, speaking during Scotland questions, said: “What I’d like the minister to justify is how it can be okay that Michelle Mone and Alan Sugar are going to have more of an impact on this Bill than Scotland’s members, some of which are on his own side.”
Mr Mundell said: “The Bill will be debated, the clause amendment will be debated in the House of Lords. I regret that it’s been brought forward in the House of Lords but we simply did not meet the timescale we aspired to.
“There will be further opportunities to debate that amendment in this House and, of course, the Scottish Parliament.”
Fellow SNP MP Drew Hendry added: “This means Karren Brady, Sebastian Coe, Joan Bakewell and 26 Church of England bishops now have more say over Scotland’s future than their elected MPs.
“Will he finally apologise for that sad state of affairs?”
Mr Mundell told MPs that he took “full responsibility” for not meeting the timescale.
He added: “We are committed to amend the Bill and amend the Bill in agreement with the Scottish Government and a Welsh Assembly Government.”
SNP shadow Commons leader Pete Wishart called on Mr Mundell to “think about his position”.
He said: “He has accepted full responsibility for not producing these amendments, is he now at a time to think about his own position?”
Mr Mundell dismissed the question, saying Mr Wishart had struck a “completely different tone” from the “professional” tone of the Scottish Government.