A non-binding motion which called for Theresa May to rule out a No Deal Brexit was passed by 321 votes to 278 on Wednesday evening, despite a three-line whip being placed on Tory MPs.
Mr Mundell was one of 13 government ministers who defied whips by abstaining in the vote.
Under the usual Cabinet convention of collective responsibility, ministers who do not vote with the Government would resign.
Labour MP Lesley Laird, the shadow Scottish secretary, said: “David Mundell proclaimed he’d vote to take no deal off the table but appears to have caved under pressure.
“If he’s not sacked, he should resign for the simple reason that his party thought it was a good idea to inflict economic calamity on the Scottish people.”
SNP MP Drew Hendry, the party’s business spokesman, claimed Mr Mundell had abandoned the interests of his constituents.
He said: “Yet again, Scottish Tory MPs have abandoned the wishes and interests of their own constituents, by voting to keep a disastrous No-Deal Brexit on the table.
“Scotland voted overwhelmingly to remain in the EU, and we know that any form of Brexit would be devastating for jobs, living standards, our public services and the economy.
“It is shameful that Scottish Tory MPs have voted to completely ignore the wishes of their own constituents, and enable a Brexit that could cause a recession and cost 100,000 Scottish jobs.
But a UK Government source told The Scotsman: “The Secretary of State has no intention of resigning for the simple reason he fully supports the Prime Minister’s goal of leaving the EU with a deal that works for Scotland and the whole of the UK.
“Claims that he is somehow supporting a no deal Brexit are ridiculous.
“It is the SNP – who are refusing to support any kind of deal – who want a chaotic no deal Brexit in the hope it would increase support for independence.”
In a message posted on social media, Mr Mundell said: “I’ve always opposed a no deal Brexit. The House made its view clear by agreeing the Spelman amendment, I didn’t think it was right for me to oppose that.The PM has my full support in her objective of leaving the EU with a Deal to deliver an orderly Brexit.”