Michael Matheson made the comments as legislation to axe the move for those sentenced to four years or more behind bars came into force.
Long-term prisoners were previously entitled to be released after serving two-thirds of their sentence.
Under the change, those offenders who the courts have ruled must be supervised upon release will now have to serve the whole of their jail term.
Criminals serving four years or more who do not require supervision after release will be freed six months before the end of their sentence.
However, long-term prisoners will still be able to apply for early release after they have completed half their sentence, although whether this is granted or not is at the discretion of the Parole Board.
Tories have previously complained the change will only apply to 3 per cent of all prisoners. Neither the Conservatives nor the Labour Party supported the legislation when it was before Holyrood.
Mr Matheson said: “I am delighted that from today long-term prisoners who pose an unacceptable risk to public safety will now serve more of their sentence in custody.
“This is a huge step in the right direction to end a system introduced by the then-UK government in 1993.
“Recorded crime is at its lowest level in 41 years but public safety is still our key priority. This law is proof of our commitment to keep communities safe and reduce the likelihood of prisoners reoffending.
“We listened to feedback on the Bill throughout the process and as a result we increased the numbers of prisoners no longer eligible for automatic early release and also ensured that rehabilitation of offenders is key to the process by ensuring mandatory supervision.”
In addition to ending automatic early release for the most serious offenders, the legislation also brings in increased flexibility over prisoner release dates, so that these can changed by up to two days to avoid inmates being released on a Friday, over the weekend or on a public holiday, when it can be more difficult for them to get help from support services.
The Scottish Government stressed this is discretionary and is “not a blanket end to Friday liberations”.