The country could also struggle to maintain its system of free education for home students, according to the report by Edinburgh University.
There are currently more than 4,800 undergraduate students from the rest of the UK paying fees of up to £9,000 a year to study in Scotland.
• Get the latest referendum news, opinion and analysis from across Scotland and beyond on our new Scottish Independence website
The study interviewed 50 higher education policy-makers and others in Scotland and the rest of the UK, as well as 148 young people aged 14 to 19 in schools in Scotland and the north of England.
Professor Sheila Riddell, of the University of Edinburgh’s Moray House School of Education, said: “The higher education systems in the four countries of the UK are tightly inter-meshed, and decisions made in one country have major knock-on consequences for the others.”
Scottish policy-makers questioned the sustainability of free higher education in the light of ongoing austerity in the public sector. They believe the issue was “dormant” rather than settled.
Young people in the north of England believed that the current policy of Scottish universities charging fees to students from the rest of the UK was unfair, the report found. As UK citizens, they believe they should enjoy equivalent entitlements.
In Scotland, young people supported the principle of free tuition. However, other issues such as the economy were more important in influencing their decision on how to vote in the referendum.
Scottish Conservative education spokeswoman Mary Scanlon said: “An independent Scotland would not be able to discriminate between different national groups, and that has been confirmed in this study.”
Lib Dem education spokesman Liam McArthur said a fall in the number of students coming to Scotland from abroad paying fees would leave a “significant funding gap” for universities.
More than one in ten students at Scottish Universities is English-domiciled. The fear is that even a slight increase in this number could make it harder for Scottish youngsters to get a place.
The Scottish Government has argued that it will be able to continue to charge fees on the basis of “objective justification” after a Yes vote. They claim Scotland’s relative size, shared land border and language, would allow it to continue with fees.
A spokeswoman said last night: “The Scottish Government is firmly committed to free tuition and we need to ensure that there are enough funded places for Scottish-domiciled students. This is important for them, their families and the economy.”
She added: “The ability to benefit from free tuition would provide a very strong incentive for students being asked to pay up to £27,000 over three years to come to Scotland in increased numbers. That is why our policy on this matter is in place.”