Douglas Ross urges Cabinet to 'consider impact' of failing to remove Boris Johnson
It comes as the Prime Minister said he was planning for a third term in office and potentially remaining in post until the mid-2030s, a run that would see him outlast Margaret Thatcher.
However, Mr Johnson is facing attacks from his MPs as they continue to plot about how best to remove him from post.
Mr Ross, who voted against the Prime Minister in the vote of no confidence following months of flip-flopping on the issue, said colleagues would continue having discussions about the future of the Prime Minister.
He also called on members of the Cabinet to consider the impact of Mr Johnson’s premiership on the country and the party.
Mr Ross added that the results in the by-elections were “very difficult” and echoed Oliver Dowden’s resignation letter saying it can not be “business as usual” for the party.
The Moray MP said: "It is up to the whole parliamentary party and in particular those sitting around closest to him, those sitting around the cabinet table, to look at the impact this is having not just on the party but on the country as a whole.
"I think clearly there will be discussions amongst colleagues.
"Oliver Dowden is a very well-respected colleague, he was a very good party chairman and secretary of state before that and he will have spoken to others I am sure since his resignation.
"Members, particularly those close to the Prime Minister, will have to look at what is the best situation for the country.
"We cannot continue to keep on going losing election after election as we have in these two by-elections and the difficult results the party faced in the local government elections in early May.”
However, the Scottish Tory leader did not throw his weight behind calls for the influential backbench 1922 committee of Tory MPs to change the rules around confidence votes.
As it stands, the party must wait a year until another such vote is held though a simple majority on the committee could see those rules changed.
"We know that Theresa May had more support from her parliamentary colleagues when she faced a vote of no confidence and she resigned just a few months later”, he said.
"It is not just a situation where that vote is done and dusted and everyone moves on because clearly the public in these two by-elections in particular have not moved on and we see now the resignation of the party chair who did sit in the cabinet up until Friday morning.
"This is a decision colleagues continue to look at and the Prime Minister must reflect on.”
Mr Johnson has urged Tory MPs plotting to oust him not to focus on the issues he has “stuffed up”, despite his authority being further diminished by a Mr Dowden’s resignation.
The former culture secretary stood down as Tory party co-chairman in the wake of the by-election defeats, saying he and Conservative supporters are “distressed and disappointed by recent events” and telling the Prime Minister that “someone must take responsibility”.
But the PM set his sights on being in office in the “mid 2030s”, in a run that would see him outlast Margaret Thatcher’s reign of 11 years in post.
Asked by journalists at the British high commissioner’s residence in Kigali if he would lead his party into the next election, he said: “Will I win? Yes.”
In buoyant mood, the Prime Minister added: “At the moment I’m actively thinking about the third term and what could happen then, but I will review that when I get to it.”
Asked in Rwanda if he believed questions over his leadership were settled, Mr Johnson replied: “Yes.”
Cabinet minister Brandon Lewis said the PM’s desire to look “long-term” when it comes to his leadership “has got to be a good thing”.
The Northern Ireland Secretary said he sees in Mr Johnson “drive and enthusiasm for what we want to achieve for our country”, and that kind of “zest” is to be celebrated.
He told Sky’s Sophy Ridge on Sunday programme: “We often get criticised in politics when we look short-term, at just the next day, the next election, the next vote.
“Actually we’ve got somebody as Prime Minister who wants to be looking long-term at how we structurally improve our country for generations to come. That has got to be a good thing.”
But the attacks kept on coming from his own backbenches on Saturday night, with Damian Green, who chairs the One Nation caucus of Tory MPs, warning the Government “needs to alter both its style and content” and calling on Cabinet members with leadership hopes to show their stripes.
The Sunday Times reported more than 30 MPs have submitted letters of no confidence in the Prime Minister just three weeks after the initial vote.
Other senior Conservatives are waiting for the results of the key elections to the 1922 committee before making their own moves.
The paper also reported that up to six Tory MPs are considering defecting to Labour with the recent defeats at the by-elections on Thursday potentially a new trigger.
More ministerial resignations could follow Mr Dowden’s which blindsided the Prime Minister on Friday morning.
Labour, meanwhile, challenged the Tories to call an early election, with leader Sir Keir Starmer telling Mr Johnson: “Bring it on.”
Want to hear more from The Scotsman's politics team? Check out the latest episode of our political podcast, The Steamie.
Want to join the conversation? Please or to comment on this article.