Just 48 per cent of Labour voters at last year’s election who support Brexit will continue to support the party, the YouGov poll for The Times found.
Nine per cent have switched to the Tories, 8 per cent to Ukip, and a quarter are not yet sure what they will do.
It highlights the division within the party over Britain’s relationship with the European Union.
Leadership challenger Owen Smith has called for a second referendum on Brexit once the terms of Britain’s exit become clear.
Leader Jeremy Corbyn says he supports Britain staying within a reformed European Union, but has faced accusations of failing to campaign hard enough prior to June’s vote.
More recently, high profile Labour MPs Rachel Reeves, Emma Reynolds and Steven Kinnock called for an end to free movement of people within the EU in response to voters’ concerns.
Fellow Labour MP John Mann also said many Labour MPs’ “shock” at the EU referendum result could be linked to a disconnect with working class voters.
Writing for Politics Home, Mr Mann said: “Any mathematician can quickly work out that in fact Labour areas were the strongest to vote Leave, and any well-organised party, sampling ballot boxes, will know that in Labour areas it was the Tory voters who tended to vote Remain and it was large blocks of Labour trade unionists who voted Leave.”
Former Labour leadership contender Chuka Umunna has been forced to clarify his position on the issue, after reports he too could see an end to free movement of people across the EU.
Mr Umunna, who also chairs campaign group Vote Leave Watch, said: “The debate about Europe, the single market and free movement is complex, and I will be setting out my views in greater length in future.
“But I have always been totally consistent in saying that Britain must be a member of the single market, on which thousands of jobs and rules protecting workers’ rights rely.
“At the same time, we need an alternative to free movement as we know it. The Government should aim for both in its EU negotiations.”
On overall voting intentions, the YouGov poll put the Conservatives on 39 per cent and Labour on 30 per cent.
Ukip polled 13 per cent, while the Lib Dems polled 8 per cent in the wake of their conference at the weekend.