Director Nick Barley said serious "reputational damage” was already being done to the event after it faced protracted problems securing visas for 12 authors in 2018.
He suggested the “knock-on” impact of the increased hassle and demands from authors would see potential star guests at the event decide attempting to get into the UK was “not worth bothering about.”
And he raised the prospect of “very smart” festival organisers in the likes of India and Canada taking advantage of Edinburgh’s problems.
Some writers invited by Mr Barley last year were told to produce bank statements and birth certificates and even undergo biometric testing last year as a requirement of appearing at the event.
Mr Barley said its global standing was set to decline further in the wake of Brexit after a failure to make “positive progress” with the UK Government despite extensive lobbying of ministers.“
He said: "The real problem is the reputational damage that is being done. Beyond the people whose visas were rejected again and again are all the people who have heard about it, who know it could be a problem and are deciding it may not be worth their while to come to Britain after all.
“Among other things, they will have heard that they have to supply three years’ worth of bank statements to show they are financially viable.
“I never talk about people who are not in the programme. I’m not going to talk about who hasn’t come to the festival.
"But I fear that what happened last year is having a knock-on impact on people’s decisions on whether or not to come to the festival. If it carries on for much longer it will get much, much worse.
"Although we had a dozen authors whose visas were rejected last years, in the end we managed to solve every one of those problems.
"There is a real risk that if this is not resolved in future then we will have people who cannot get a visa and we will have people who decided it is not worth bothering.
“I will not name them, but last year I had one author tell me that they were considering pulling out because they did not want to submit three years worth of bank statements and did not want to accept an invitation.
“We have got a great reputation as a festival. It’s not being attacked yet, but I’m acting to try to prevent that happening in future.
“Unless we can get the visa problems solved there is a risk that people will go to events elsewhere. There are some brilliant festivals in other parts of the world. Unless we get this sorted Edinburgh could lose out to other festivals elsewhere.
“There are lots of very smart festival organisers who are developing and rapidly growing events in places like Jaipur in India and Monteal, in Canada.
"There are competitors out there who will put our festival at risk unless we make sure it as easy as possible for artists and authors to appear in Edinburgh.”
Julia Amour, the director of Festivals Edinburgh, and its chair Sorcha Carey, met with Immigration Minister Caroline Nokes for talks days after the Scottish Government offfered to host a summit on the issue in August.
Mr Barley added: “We’re not able to say at this stage whether anyone’s visa has been refused. It’s too early to say as the application process is ongoing.
"There is positive dialogue going on with the Government but we’re not able to say we have had positive progress. We’ll not say that until we have evidence that writers are getting their visas.
“At the moment, the problems we’ve had relate to authors outside Europe. Our major concern is that this might begin to affect authors from within Europe after Brexit. That is the urgency of it.
“When I took on this job I didn’t expect to be spending a lot of time talking to politicians about visas. We also have to try to put together a festival.
“I’m very concerned at the number of high-level officials in the British Council, in embassies around the world and in government who are spending time trying to resolve these problems. We’re using a sledgehammer to crack a nut.
“I'm very concerned at the number of high-level officials in the British Council, in embassies around the world and in government who are spending time trying to resolve these problems. We are using a sledgehammer to crack a nut.”
Ms Amour said: " The festivals are working with agencies, parliamentarians and governments to shape policy at both Scottish and UK level - including Immigration Minister Caroline Nokes – helping to find solutions to recurring problems with complex visa rules that can deter artists from accepting our invitations and threaten the free flow of culture and ideas at a time when the country needs to be even more globally engaged.
"We welcomed this opportunity to meet the Immigration Minister and will continue to work with partners to help improve the system."