The 82-year-old died at St Columba's Hospice on Sunday night.
A renowned actor and former BBC commentator, his voice had been heard over every Tattoo since 1966.
Capital events champion Councillor Steve Cardownie said: "It's hard to think of the Tattoo without Tom. The narratives he gave were second to none, and the chat he would give at the end of each Tattoo about Scotland made the hairs on the back of your neck stand on end.
"There will be thousands of people out there who have never met him but will have fond memories of him through his voice."
Mr Fleming also voiced commentaries on royal and state occasions including the Coronation, the Silver Jubilee, two royal weddings, ten funerals, the enthronement of two Popes and three state visits by Archbishops to Japan and the US.
In 1995 he was principal commentator for BBC television for the VE and VJ Day celebrations and was well known for his annual commentary on the Cenotaph Service in London.
In 1999, Mr Fleming read The Beginning of a New Song, by poet Iain Crichton Smith at the opening of the Scottish Parliament.
First Minister Alex Salmond said: "Tom Fleming was a great Scot, who will be much missed."
A deeply Christian man, Mr Fleming was a highly respected member of Canonmills Baptist Church, where he was a regular organist and preacher.
Author and friend Hamish Coghill said: "He will be sorely missed by the people of Canonmills and a wide circle of friends.
"You couldn't ask to meet a better person in your life."
Born in Edinburgh in 1927, Mr Fleming trained as an actor in the city and first appeared on stage at the Royal Lyceum Theatre in a non-speaking part.
When aged just 17, he toured India and Burma playing to the troops with Edith Evans,
and in 1948 he became the youngest ever light programme announcer for the BBC, but quit to pursue an acting career.
He went on to play the leading part in the first full-length drama production from Scotland, The Black Eye, in 1952.
The following year he co-founded the city's Gateway Theatre and joined the Royal Shakespeare Company in 1962.
Three years later, he became director of the Lyceum and was director of the Scottish Theatre Company from 1982 to 1987.
He was made an OBE in 1980, a doctor of Heriot-Watt University in 1984 and a fellow of the RSAMD in 1986, all for services to the arts in Scotland.
BBC Scotland head of programmes Donalda MacKinnon said:
"We've lost one of our finest broadcasters and our thoughts are with his family and many friends."