Alderman Michael Bear, who acts as an "ambassador" for UK financial services centres, said that even though on the face of it firms don't look as though they are locating out of the Square Mile, global companies will be increasingly tempted to "book" large deals in other countries such as Hong Kong or Singapore if the UK's regulatory and tax crackdown continues.
This would not only tarnish Britain's reputation as global financial services centre but it would also reduce the UK government's tax take from the sector, he said.
According to trade body TheCityUK, financial services boosted the Treasury's coffers to the tune of 53 billion last year.
Bear, who was speaking at a Scottish Financial Enterprise (SFE) event, said: "There are choices for writing business in different jurisdictions and why would they come here if tax is higher and if it's harder to navigate through some of our regulation? We have to be alive to the fact this is a competitive threat.
"The secret for us is to get in between those capital flows. If it becomes too complicated or too costly (for international firms to book business in the UK] that will stop."
Chris Cummings, chief executive of TheCityUK, who was also in Scotland yesterday, agreed that the UK could be damaged by this "invisible threat".
"The buildings stay here, many people stay here but the tax take simply fails to arrive," he said.
Although a number of high-profile firms, including HSBC, have raised the prospect of relocating their headquarters, Bear said there isn't any evidence yet that there has been an exodus from the Square Mile.
He warned that Britain was spending too much time "picking over the bones of the past" while other countries, particularly China, weren't worried whether their banks are "too big or too international".
"Rather the reverse," he said.