With almost all the results in from Scotland, the SNP seems likely to win three of the six Scottish MEP seats, with Scottish Labour losing its two MEPs.
Results from the 32 councils across Scotland showed the SNP, who fought the election campaign on a strong pro-European platform, had polled 37.7 per cent.
The party's previous best Euro election result was 32.6 per cent in 1994.
Ms Sturgeon tweeted: "Formal declaration to come, but clear now that @theSNP has won the Euro election emphatically - we are on course to take 3 out of 6 seats. A historic victory. And Scotland has rejected Brexit again."
But while Ms Sturgeon's party increased its vote - it had polled 28.9 per cent in the last European elections in 2014 - Labour's vote slumped.
Five years ago it was a close second with 25.9 per cent of the votes, but with all but one of the council areas having declared, that fell to 9.3 per cent.
The Brexit Party finished on 14.8 per cent of the votes in Scotland, with the Liberal Democrats on 13.8 per cent and the Conservatives on 11.6 per cent.
Scotland's six MEPs
It means Scotland's six new MEPs are the SNP's Alyn Smith, Christian Allard and Aileen McLeod, Louis Stedman-Bruce from the Brexit Party, Sheila Ritchie of the Liberal Democrats and Baroness Nosheena Mobarik of the Conservatives.
Labour's David Martin, who had been the UK's longest serving MEP having first been elected in 1984, has lost his seat, with the party paying the price for its failure to take a clear stand on the crucial issue of Brexit.
Mr Martin tweeted: "While obviously personally disappointed to lose my seat I'm devastated for the hundreds of good Labour people who put so much into the campaign. We lost not because of lack of effort, but lack of clear message."
Mr Steadman-Bryce said: "The result is pretty amazing. Obviously five weeks ago the party didn't exist and we've gone from zero to second place in this election, which I think is phenomenal, and in the wider picture we are leading across the UK.
"This just demonstrates that there is a strong and valid number of people in Scotland who voted to leave the European Union."
Labour Edinburgh South MP Ian Murray, who has campaigned for a second European referendum, hit out on Twitter, saying: "The Labour leadership refused to listen to me, their members & their supporters. Maybe now they'll listen to the voters?"
Ms Sturgeon also tweeted about the "outstanding results" for her party, saying they were "on course for our best ever Euro election result".
She added: "Polling way ahead of any other party. Higher % in Scotland than Brexit Party's current share in rUK. Emphatically clear that Scotland doesn't want Brexit."
Mr Smith, who was the lead SNP candidate, said it was "clear Scotland has endorsed Remain".
He said: "Our Remain vote can not be ignored any longer. Scotland's for Europe."
In the City of Edinburgh Council area Labour was sixth - behind the SNP, the Liberal Democrats, the Scottish Greens, the Tories and the Brexit party.
Lothian MSP, and former Scottish Labour leader Kezia Dugdale said her party should respond to the results by "backing a People's Vote unequivocally and committing wholeheartedly to remain".
Meanwhile Scottish Conservative leader Ruth Davidson said "all indications" were her party had retained a Scottish MEP on what was a "tough night for the party UK-wide".
She tweeted: "Official confirmation to come, but all indications are @ScotTories have held our seat on a tough night for the party UK-wide. Congratulations to @NosheenaMobarik and a massive thank you to all our activists. Every vote counted!"
The Liberal Democrats had lost their Scottish MEP in 2014, when the party paid the price at the polls for its coalition with Conservatives at Westminster, but its increased share in this election leaves the party in Scotland on course for a return to the European Parliament.
Speaking on the BBC's Good Morning Scotland programme on Monday, Scottish Lib Dem leader Willie Rennie said the result is a "big win" for the party.
He said: "This was quite some result and I think it shows that the Liberal Democrats have got the momentum.
"There's lots of people who voted for us this time that have never voted for us before and we need to work hard to keep their trust that's been shown in us.
"But we've got a real opportunity to represent the majority of people in Scotland who are pro-United Kingdom and pro-European Union."