The SNP, Tories and the Liberal Democrats have returned councillors in the Tweeddale West ward of Scottish Borders Council.
The Tories’ vote share dropped by 15.7 per cent based on the 2017 election in the ward.
A Scottish Tory source said: "We're expecting third, heavy losses coming. Tory voters haven't gone to Labour, they just haven't turned out because of Boris and Partygate."
The result in Tweeddale East, the neighbouring ward, was announced soon after, with an SNP, Tory and independent councillor elected.
Speaking from the election count in Edinburgh, Scottish Tory frontbencher Miles Briggs said: “We knew it was going to be challenging, we’ve known that for a couple of weeks now. I’m personally disappointed with the results which have come in so far.
“Some seats which we won five years ago we are now holding, which is really good to see.
“There’s some good news, but it’s really disappointing.
“From the work I did yesterday with my activists here in Edinburgh, it’s quite clear that people weren’t going to go to vote.
“They were protesting voting by not going to any other party, or when they’ve gotten their postal votes they hadn’t returned them and put them in the bin.
“It has depended on how people want to send a message. You either go and vote for someone else or you don’t vote at all.”
When asked if Partygate had impacted Tory support, he said: “Yes I think that’s part of it. You can’t deny that and I think speaking to people yesterday they certainly weren’t happy with the actions of the Prime Minster and his team.”
However, Mr Briggs said he “thoroughly expects” incumbent Tory councillors in Edinburgh to be returned.
“We will lick our wounds and move on,” he said. “Let’s look back five years ago, that was a tremendous election and one of the most exciting elections I’ve been involved in with the Conservatives.”
Unlike in some places in the rest of the UK where votes were counted overnight, ballots are being tallied up in Scotland throughout Friday and the final results are due to be announced in the early evening.
Results in England have so far been positive for Labour, with the party increasing its number of seats by 34 – while the Tories dropped 122.
Senior figures in the Scottish Conservatives have told The Times they are expecting “heavy losses” due to the Partygate saga, which saw the Prime Minister, his wife and Chancellor Rishi Sunak fined by police.
But former Tory MSP Adam Tomkins described that comment as “nonsense”.
He highlighted Scottish Tory leader Douglas Ross had backed Boris Johnson to remain in post at Number 10.
On Twitter, Mr Tomkins said: “Whatever today’s results show Douglas Ross owns this, not Boris.
“It was Douglas who U-turned, Douglas who flipped, and Douglas who backed the PM. He and his team need to own the consequences, not pass the buck.”