A nearby boat raised the alarm after spotting the body in the water at Anstruther around 4pm and stood-by until a coastguard vessel arrived.
The Anstruther lifeboat was launched and the crew hauled the man on board, although paramedics were unable to resuscitate him.
Officers from Fife Constabulary were also called to investigate the incident.
A police spokesman said their investigation was at an early stage and that it was not yet known how the man came to be in the sea.
It is also not known how long the man had been in the water before lifeboat crew recovered his body.
A coastguard spokesman said the crew were on the scene within minutes after the man was found.