The Electoral Reform Society (ERS) Scotland identified a move away from “moderate pluralism” towards a “predominant party system” since the elections of 2003 and 2007.
The organisation said the SNP’s success “risks undermining the diversity, openness and spirit of co-operation” of the Scottish Parliament.
Proposed reforms include a new written Scottish constitution developed through a national debate and a “People’s Assembly”.
This would take the form of a second parliamentary chamber, with citizens randomly selected and allowed up to a year of “democratic leave” to propose, scrutinise and amend legislation.
ERS Scotland advocates a move from the current “d’Hondt” method of counting parliamentary seats to the “Saint-Lague” approach, which would treat large and small parties equally.
It also wants elected Holyrood committee conveners and proportional representation in UK general elections.
“There is a paradox at the heart of Scottish politics,” the report said.
“The preference Scots seem to have for the institutions of Scottish democracy over British ones has manifested itself in the emergence of a dangerously lop-sided Scottish party system - the very thing that has traditionally connected democratic institutions to the people.
“The SNP have benefited enormously from being seen to support, defend and indeed symbolise Scotland’s relative political autonomy from an unpopular and failing British political system.
“But the SNP’s success has been so overwhelming that it risks undermining the diversity, openness and spirit of co-operation that were supposed to characterise the Scottish Parliament at its inception.”
ERS Scotland said its proposals would contribute to “a more open, diverse and self-critical “ politics.
The report said: “We are not suggesting that Scots should or shouldn’t vote for one party or another; only that the principles, hopes and expectations which underpin most people’s voting decisions may not be best-served by a system where one party has a near-guaranteed place in government.”
It follows an ERS Scotland briefing by Professor John Curtice suggesting the SNP was poised to do so well on May 5 that a second list vote for the party may be “wasted”.
Director Willie Sullivan said: “Labour were predominant for years, which let them take voters and their future for granted.
“They got too comfortable to respond properly to massive changes in the world around them in the 1980s and 1990s.
“If the SNP become too dominant, there’s a risk that our politicians become overly comfortable - and, potentially, secure enough to ignore the next wave of social change. Politics risks become subject to partisan ‘boom and bust’ if one party becomes as dominant as Labour was in the past.
“We hope this report and our proposals can help start a debate about the direction in which Scottish politics is going and how we can foster a more self-critical and broad range of voices in a parliament designed for diversity.
“Political openness and the ‘rainbow politics’ of the 2003 Parliament is healthy and, rather than something to be feared, should be welcomed.”
Meanwhile, new data from ScotCen’s Scottish Social Attitudes survey shows interest in politics in Scotland is at its highest level since 1999.
Researchers found that 40% of people have “a great deal” or “quite a lot” of interest in politics, up five points on the figure in summer 2014 before the independence referendum.