Mr Matheson made the announcement as the demolition of Scotland’s only female jail at Cornton Vale near Stirling got under way.
Plans to replace the prison with a new 300-cell facility were scrapped in 2015 in favour of a smaller 80-capacity prison at the site and five new units each accommodating up to 20 women across the country.
The smaller units for female offenders aim to place them closer to their families and provide targeted support to address issues such as alcohol, drugs, mental health or domestic abuse.
The announcement followed a report into how best to deal with female offenders by former lord advocate Dame Elish Angiolini.
Mr Matheson said: “With the demolition of Cornton Vale under way, there is no turning back on our plans for a smarter, more progressive approach to managing women in custody, ensuring they are prepared for life after release.
“That is why I am also pleased to announce that the first two community custodial units for west and east Scotland will be in Glasgow and in Fife or Dundee.
“By housing women in smaller, community-based units closer to their families, and providing additional support to address their needs, such as drug and alcohol advice or mental health support, we can further reduce reoffending - and so keep crime down and our communities safe.”
The Scottish Prison Service aims to have the new national facility at Cornton Vale and the two community units complete by the end of 2020.
The demolition work coincided with the publication of a new document setting out the Scottish Government’s vision for justice.
Among the priorities is a continuation of a shift in spending towards the prevention of crime.
The document states: “We already have a track record in shifting towards prevention, including Scotland’s recent success with youth justice.
“As things stand, the cost of enforcement still outweighs the amount spent by the justice portfolio on primary prevention and early intervention many times over.
“The cost of the criminal justice system itself is about £2.5 billion per annum.
“In times of reduced public spending, a shift towards preventative spending is challenging, but it is more likely to be effective in the long-term.”
Mr Matheson added: “Over the last decade Scotland has become a safer place, with less crime, including violence, falling drug use, improved fire safety and better support when people are victims of crime or other serious incidents.
“But we strive for greater progress, not least while inequality continues to influence the likelihood of someone being a victim of crime or being drawn into offending.
“This is among the challenges outlined in the vision paper, which also highlights the relatively poor physical and mental health of people in contact with the justice system.”