Officers responded to a report that a body had been seen in the water at around 10.20am on Thursday.
The man was recovered from the shoreline at Hawes Pier in South Queensferry.
Coastguard also attended and he was pronounced dead at the scene.
The man was not carrying any identification at the time and inquiries are now under way to establish his identity, Police Scotland said.
He is described as white, around 5ft 10ins tall, in his late 40s with blackish grey receding hair and a medium build.
He was wearing a black button-front jacket, blue hooded top with full zip and white toggles, blue denim Levi jeans, black hat, black scarf and navy gloves.
There are no suspicious circumstances surrounding his death at this stage, the force said.
Detective Sergeant Kevin Tait said: “Our investigation is continuing to establish this man’s identity and we would ask anyone who can assist with our ongoing inquiries to contact police on 101.”