Boris Johnson is attempting to hold onto his premiership after more than a dozen resignations from his Government in the past 24 hours.
Levelling up, communities and housing secretary Michael Gove has also reportedly told Mr Johnson in person that it is time to resign.
Rishi Sunak resigned as Chancellor while Sajid Javid stood down as health secretary on Tuesday and were followed by a number of ministers in the ensuing hours, along with Tory MPs who revoked their support for the Prime Minister.
Mr Ross is also among those calling for the Prime Minister to step down.
He told the BBC: “The Prime Minister needs to realise he’s lost the support of many colleagues and he has to stand down as Prime Minister.
The Scottish Tories leader added: “It’s not an easy thing for many of us to tell the Prime Minister, but time is up and he needs to step aside.”
He later told STV the Prime Minister had “lost his Chancellor of the Exchequer, he’s lost his health secretary, he’s lost his Solicitor General”, adding: “I think we have seen that the Prime Minister must now know that the time has come for him to step aside.
“I think he will be looking at what happened in the chamber [during Prime Minister’s Questions] – there wasn’t a great deal of support from across our side for the Prime Minister and what he was trying to say and do.”
Mr Ross has been criticised for his shifting stance on the Prime Minister after he submitted a letter of no confidence at the turn of the year in the wake of ‘Partygate’, only to revoke it after the Russian invasion of Ukraine and then vote against Mr Johnson in last month’s confidence vote.
Mr Ross’s fellow Scottish Tory MP Andrew Bowie publicly stated he had submitted a letter to the 1922 committee, saying on Twitter that Government was “not functioning”.
First Minister Nicola Sturgeon also reiterated her calls for the Prime Minister to stand down.
“How much longer can this go on?” she tweeted.
“If Johnson has merest scrap of concern for anyone but himself, he will resign immediately.
“And then.. let’s have an election to choose an alternative. For (Scotland) the permanent alternative is independence.”
Meanwhile, SNP Westminster leader Ian Blackford urged the Prime Minister to call an election if he refused to stand down, which he hinted would be used as a “de-facto referendum” on Scottish independence.
At Prime Minister’s Questions on Wednesday, the SNP MP said: “A few weeks ago I compared the Prime Minister to Monty Python’s black knight. Actually [it] turns out I was wrong. He’s actually the dead parrot.
“Whether he knows it or not, he’s now an ex-prime minister. But he will leave behind two deeply damaging legacies. I hope the dishonesty of his leadership follows him out of the Downing Street door.
“But the other legacy is that of Brexit, and that will stay, because I’m sad to say that the Labour Party now fully supports [Brexit].
“Scotland wants a different future, not just a different prime minister. So if the Prime Minister won’t resign, will he call a general election and allow Scotland the choice of an independent future free from the control of Westminster?”
Ms Sturgeon last week announced plans to have the Supreme Court assess the legality of holding another referendum on independence, saying if the court finds the Scottish Parliament does not hold the necessary powers, the next general election will be fought on the single issue of leaving the UK.
Mr Blackford earlier told Sky News the number of resignations has reached “crisis” point.
Meanwhile, Scotland’s finance secretary Kate Forbes said “while the UK Government is consumed with resignations and deceit”, the new Treasury team should focus on helping people struggling with the cost-of-living crisis.
When asked if a change of prime minister could damage the ability of the UK to respond to the cost-of-living crisis, Ms Forbes added: “At a time of crisis, the public needs to have confidence in their political leaders that they understand what truth is, they understand what integrity is and they understand that matters more than anything else at a time of crisis.
“Right now I would argue that the public, and I certainly, do not have confidence that at the very top of the UK Government there is any concept or any understanding of those vital concepts.”