But now 17th century Acheson House, built for the Scottish Secretary of the time, is to become the new home of the capital's heritage guardians.
The Edinburgh World Heritage Trust will relocate from Charlotte Square to the Canongate within months under a rescue plan for the building, which includes an expansion of the adjacent Museum of Edinburgh.
Acheson House has long been seen as one of the most threatened buildings in Edinburgh's Old Town, despite a rich history. But the heritage group's move - after 120,000 worth of repairs to make the building habitable for the first time in 20 years are carried out - has secured the future of the four-storey building, which was a notorious brothel in the 1830s and 1840s,
Although ordered by Sir Archibald Acheson, it is thought he never actually lived in the building as he died the year after work began.
It became home to a wealthy merchant, Patrick Wood, the Incorporation of Bakers of the Canongate, and a "parish house" for the nearby Canongate Kirk.
However it had fallen into decline by the 18th and 19th centuries, by which time it had been divided up into tenement flats, becoming a brothel, and having to be rescued during a slum clearance programme.
The building was rescued from demolition in the 1930s by the Marquis of Bute following a campaign in the letters pages of The Scotsman.
But its seedy past famously came to light in the memoirs of local policeman, Inspector James McLevy, who recounted dramatic raids on the property.
The A-listed building is one of several in the Old Town on the "buildings at risk" register and plans to bring it back into use have been floated since 1995.
The city council hopes opening up the ground floor to the public, and allowing Edinburgh World Heritage to move on to the first floor will lead to the creation of a major "museums hub" in the area.
Culture leader Deidre Brock said: "Acheson House is one of the Old Town's most historic buildings and our long-term vision is to bring it back to life as part of a museums hub devoted to charting the fascinating development of Scotland's capital city. "Allowing Edinburgh World Heritage to occupy Acheson House will greatly assist us in the sympathetic conservation of this 17th-century building, helping guarantee it a suitably prominent role in Edinburgh's 21st-century cultural infrastructure."
Adam Wilkinson, director of Edinburgh World Heritage, added: "It is a great opportunity to help bring life back to this fascinating and beautiful building, reminding us of the historically diverse social mix of the Old Town.
"The New Town has been a generous host to us for the last 12 years and it has been a privilege to work in Charlotte Square, to which we will bid a very fond farewell."
Detailed plans to showcase the city's hidden gems in the area were first unveiled in 2000 but have stalled because of funding problems. Heritage groups have warned the council Acheson House has been "deteriorating rapidly" and large quantities of asbestos have been discovered there.
Last year the National Trust for Scotland, quit its own headquarters to move to a modern new base on the outskirts of the city.