A report into the implications of Scotland re-joining the EU if it became independent said that Scotland would become a customs and regulatory border for the EU and as a result would need to close the border with England.
The move would create problems for trade as Scotland exports substantially more to the rest of the UK than to the EU.
The report, from the Institute for Government, also said that as a new member state, it would be unlikely that Scotland could negotiate the same UK opt-outs and would need to adopt the euro and re-join common fisheries and agriculture policies.
The document said: “Joining the EU would mean Scotland joining the single market and customs union – and as a result the Anglo-Scottish border would become a new external customs and regulatory frontier for the EU.
"Even a looser model of integration with the EU, such as Scotland joining the European Economic Area (EEA), could not grant frictionless access to both the EU and the UK markets, so long as the UK–EU relationship continues to be governed by the UK-EU Trade and Co-operation Agreement.”
It said the process required to join the EU could take ten years, as an application process could not begin until Scotland was fully independent from the UK and typically takes years to conclude.
The report said: “The process of Scotland’s separation from the UK could easily last longer than the five years it took for the UK to exit the EU and the EU accession process would likely last at least two further years.”
However, the report said it “seems more probable that Scotland might be able to avoid any commitment, even in principle, to joining the Schengen area, within which there are no border checks”.
Along with the UK, the Republic of Ireland opted out of the Schengen area, instead maintaining the existing Common Travel Area that allows for the free movement between the UK, Ireland, the Channel Islands and the Isle of Man.
The authors said the SNP needed to be "open about the trade-offs" and the "costs as well as benefits" that EU membership would bring and warned that Scotland should “make that choice in the knowledge that it will not be able to maintain open borders with both the EU and with the rest of the UK.”
Pamela Nash, chief executive of Scotland in Union, said: “The reality of leaving the UK is scrapping the pound and building a hard border with our main trading partner, with devastating consequences for jobs.“It would also mean a border between friends and families.”
Jenny Gilruth, SNP Europe Minister, said: “It is the UK Government which is putting up borders because of its Brexit policy but there is no reason an independent Scotland would not remain in the Common Travel Area, meaning people here would enjoy freedom of movement within the British Isles and EU.
“The EU’s founding values – such as democracy, equality, freedom and the rule of law – are Scotland’s values and we are keen to make our contribution to the shared endeavour that the EU represents."