Ninety minutes after polls closed, the 41-year premier conceded defeat as exit polls suggested 57% of voters had rejected his plan to streamline the country’s system of government.
Financial markets were braced for fresh volatility amid fears his fall could spark a run on Italy’s tottering banks, further weakening the euro.
The result will be seen as a further victory for the anti-establishment backlash sweeping much of Europe and America - with the opposition to his plan headed by the anti-euro Beppe Grillo’s populist Five Star Movement.
The comedian-turned politician rode the same wave of discontent which propelled Brexit to victory in the UK and Donald Trump to the White House in the United States.
Mr Renzi had staked all on his reform plan - vowing to resign if it was rejected by voters. But his gamble backfired as it turned the referendum into a vote on his record in office.
In an emotional televised address, he accepted full responsibility for the “extraordinarily clear” defeat and said he would tender his resignation after meeting his cabinet on Monday.
“The experience of my government ends here,” he said.
Earlier there was a rare moment of relief for Europe’s beleaguered political mainstream as the challenge of Austria’s far-right Freedom Party for the Austrian presidency ended in failure.
Norbert Hofer, who was seeking to become the first far-right president in Europe since the Second World War was beaten by the Green Party’s Alexander Van der Bellen.
Mr Van der Bellen said his win sent a “message to the capitals of the European Union that one can win elections with high European positions”.
But Marine Le Pen, the leader of France’s National Front, who is hoping to ride the populist tide to victory in the French presidential elections, praised Mr Hofer’s party for a campaign “fought with courage”.
“Victory will be theirs in the next legislative elections,” she said on Twitter.