Mr Ross said he had a heated telephone call with the Prime Minister when he told him to quit in January over the Downing Street partygate scandal.
The Scottish Tory leader later U-turned on the position but voted against Mr Johnson in a confidence vote in June.
Mr Ross was asked about the issue during an Edinburgh Fringe event with the broadcaster Iain Dale, where he appeared on stage with Scottish Labour leader Anas Sarwar.
He said he had called Mr Johnson straight after Prime Minister’s Questions in Westminster.
Mr Ross said: “Because I was one of the first more senior members to call for the Prime Minister to resign, he was not particularly pleased about that.
"I think we saw from the reaction of his strongest supporters, they didn’t want it to be seen that me calling for the Prime Minister to go didn’t come with any baggage, didn’t attract any criticism of me.
"And then we had an incidence with Jacob Rees-Mogg and others, going out to defend the Prime Minister and highlighting those who dared to speak against him.”
Mr Rees-Mogg called Mr Ross a “lightweight” politician during broadcast interviews.
Asked how Mr Johnson reacted during the phone call, and whether he was “genuinely angry”, Mr Ross said: “I would say, yeah, he was angry.
"One of his MPs was saying they no longer had confidence in him. I think it would be a difficult conversation to have for any party leader with your MPs telling you that.
"Particularly when you’re Prime Minister. That ratchets it up a notch, when you’re not only just saying you think you’re unsuitable to lead the party, but to lead the country.”
Asked if Mr Johnson used any “rude Latin words”, Mr Ross said: “They were rude, they weren’t necessarily Latin from what I remember.”
The Scottish Tory leader said he always tried to “dampen” down such confrontations.
He said: “I referee football as well, and you can’t react. I’d love to react to some of the players that run up to me and swear at me and say things and that, but you just have to remain calm.
"And I think actually it undermines them – them as in politicians or players – getting overly aggressive and swearing at you. You kind of take the moral high ground.
"In the same way I think when Jacob Rees-Mogg made those comments, I said he’s entitled to that view.
"I disagree with it. I’ve never said anything about Jacob Rees-Mogg – I don’t think we’ve actually ever spoken before in the five years I’ve been in the House of Commons.”
He added: “For someone who prides himself on being very courteous to colleagues, it was a surprise.
"And I think it potentially backfired, because a number of colleagues who still supported the Prime Minister thought Jacob had gone a bit far with that.
"But for me, it’s water off a duck’s back.”
Elsewhere, Mr Ross said the Tories and Labour “have to seriously look” at an electoral pact ahead of the next general election.
First Minister Nicola Sturgeon has said the SNP will fight the election as a “de facto referendum” if a second independence vote is blocked next year.
Mr Sarwar dismissed the prospect, insisting it was a tactic “that is about keeping the Conservatives in second place” in Holyrood.