The party is backed by just 18% of voters, two points down on a week ago, according to a YouGov survey for The Times.
The Tories have climbed two points to stand at 24% while the Brexit Party is up one point to 23% and the Liberal Democrats also up one to 20%.
With Labour leader Jeremy Corbyn's Brexit strategy under intense scrutiny, the survey found that only 25% of Remain backers intend to support Labour.
This compares with 48% of Remainers who said they would vote for the party at the beginning of the year, and 40% who gave it their backing at the end of April.
Just 8% of Leave supporters say they will vote Labour, compared with 21% in January, according to the poll.
Labour's shadow health secretary Jon Ashworth told ITV's Peston: "I don't believe that would be the result at a general election.
"If that was a result at a general election it would be devastating for the Labour Party."