Delegates supported a call on Friday to study "other more successful models for dealing with criminal offences" by 435 votes to ten.
A motion said Scotland' s incarceration rate was "substantially higher" than most other EU countries, while 96 per cent of prisoners are men and 32 per cent are aged under 30.
It added: "Conference therefore proposes a study to consider other more successful models for dealing with criminal offences, such as the Finnish model, which utilises open prisons and has focused on rehabilitation, reducing imprisonment rates to amongst the lowest in Europe without any increase in crime".
Inmates in Finland’s open prisons can leave the facility, study and have paid jobs.
Colin Storrier, of the SNP's Edinburgh Central branch, told delegates: "When we have a system that is fundamentally broken, we should have the humility to look at other approaches that have been more successful."
He said a "not insignificant" number of people were imprisoned for non-violent crimes in Scotland, such as housebreaking and drug offences.
Mr Storrier said a third of all Finnish inmates are housed in open prisons and are statistically less likely to be arrested again.
SNP MP Owen Thomson said the Scottish Government was already working on a new community justice strategy to promote alternatives to prison, "moving the focus towards prevention".
He said the proposed study would support this, adding: "Let's look for new ideas."
SNP ministers are consulting on how to improve the way bail and prison custody operates in Scotland.
It comes ahead of new legislation set to be introduced in Holyrood.
Earlier this month, justice secretary Keith Brown said: “Our overarching aim for the justice system in Scotland is to improve public safety, support victims and reduce rates of victimisation. The proposals in this consultation support that aim.
"We cannot simply keep using imprisonment to address wider societal harms. Indeed, in some cases such use can exacerbate the harm.”
He added: “Keeping our communities protected is our number one priority and these principles underpin the reforms we are consulting on.”