Images of the banner which was hung on railings near Bute House, the official residence of Nicola Sturgeon, began circulating social media on Wednesday afternoon.
The sign featured a picture of Ms Sturgeon with a Hitler moustache with offensive messaging.
Police officers worked quickly to remove it from the railings.
A Police Scotland spokesperson said: "On Thursday, 25 June, 2020, police received report of an offensive banner at Charlotte Square in Edinburgh. Officers attended and the banner had been removed. Enquiries are ongoing."
The former Scottish Conservative leader, Ruth Davidson also tweeted to condemn the banner which appeared.
On Thursday she tweeted: "A picture is circulating of a banner posted on railings at Charlotte Square in my Edinburgh Central constituency. I won't give it further oxygen by linking to it.
"I condemn it in the strongest terms. Come on Scotland, we can do politics better than this."