A report by charity Full Fact and think-tank UK in a Changing Europe claimed the most immediate challenge for an incoming prime minister on the Union was that of independence.
Entitled "The Conservative Leadership Contest: A Guide to the Policy Landscape", the paper aims to provide information to help people make up their minds on the key issues.
First Minister Nicola Sturgeon is planning to hold another independence referendum on October 19 next year.
In a section written by Nicola McEwen, professor of territorial politics at Edinburgh University, the report suggests there would still be pressure on the UK Government to map out the future of the Union even if the Supreme Court rules referendum legislation is beyond the Scottish Parliament’s legal powers.
The report said: "The most immediate challenge facing the next prime minister will be how to engage with the Supreme Court reference and respond to its outcome. This is a legal process, but one that is loaded with political significance.
The court may conclude that the reference is premature and await a further reference from UK law officers once the proposed legislation has passed through Parliament. Or it may take up the request to determine in advance of the legislation where competence lies.
"Ultimately, the future of the Union will be determined by politics, not law. If there is no legal pathway to an independence referendum, Nicola Sturgeon declared that she would regard the next UK general election as a de-facto referendum.
"Such a scenario might suit the Conservatives in the short term, as the party most likely to benefit electorally from opposition to independence within Scotland.
"And so long as Scotland remains divided on independence, there is neither a legal nor a political imperative to respond to any claims of an SNP mandate.
"But such prolonged constitutional limbo could have a debilitating effect on politics and the Union in the longer term and do little to restore intergovernmental relationships already damaged by Brexit."
The report also criticised outgoing PM Boris Johnson’s so-called "muscular unionism”, which was aimed at strengthening the UK Government’s role and visibility in devolved territories.
Prof McEwen wrote: “Centred on policy areas that fall under the responsibility of the devolved institutions, this approach has prompted repeated accusations that devolution is being undermined.
"For example, the three devolved finance ministers combined forces to charge that new powers to spend directly in devolved areas, which the UK Government gifted to itself in the controversial United Kingdom Internal Market Act, threaten and reverse the devolution settlements.
“Together, these developments have led to a significant erosion in trust between the UK and devolved governments.”
The paper also criticised Mr Johnson’s impact on trust in politics, and advised leadership contenders Rishi Sunak and Liz Truss that whoever won would have to address the problem.
"One of the defining legacies of Boris Johnson’s premiership has been its bulldozing of political trust and erosion of citizens’ faith in democratic politics and politicians,” the paper said.
"Building trust at a time when real pay is falling and public services are struggling will be a severe test for the new prime minister.”