The warning is the starkest yet from the Scottish Government to the Dutch-owned operator following months of misery for passengers.
Mr Matheson said performance had seen a "marked improvement" this week following hundreds of cancellations because of late-arriving new trains causing a staff training backlog.
However, he said his issuing of the remedial plan on 24 December had "very serious implications" for ScotRail.
He expected the plan to be submitted around 18 February, which would be closely monitored to judge its effectiveness.
It relates to the level of cancellations on Edinburgh area suburban services on the Bathgate, Dunblane, Fife Circle, North Berwick and Tweedbank routes.
The move was also ordered because ScotRail's overall performance over the four weeks to last Saturday is expected to fall below "breach" level.
Once ministers have agreed to the plan, it will remain in force "until it’s delivered in its entirety", a Transport Scotland official said.
Mr Matheson said he expected it to "set out how they will address the performance issue".
He said if the plan failed, Abellio - an offshoot of state-run Dutch railways - would be in breach of contract.
The minister said that could lead to fines, increased charges or early removal of the ten-year contract, which is due to end in 2025.
He said: "This has got very serious implications for the contract holder.
"The remedial plan is one of the most serious things we can do.
"They [Abellio] are very clear about how serious this is for them as franchise holder.
"That could ultimately result in them losing the franchise."
The move follows two ScotRail-wide improvement plans since 2016, although this is more serious as it takes the operator closer to losing the franchise.
However, Mr Matheson also told MSPs that up to 70 per cent of delays had been caused by ScotRail's alliance partner, Network Rail.
He called again for full powers over the track body to be transferred to the Scottish Parliament.
The minister said: "Fixing ScotRail's train crew problem will not be enough."
He said only with Network Rail being fully accountable to Holyrood would it be possible to managed the Scottish rail system properly.
There were only a handful of cancellations today but 78 services had fewer than normal carriages because of faults.
ScotRail said most were because several of its newly-acquired 40-year-old InterCity diesel trains were being used for training.
Only some ten of the 26 trains have arrived, and only two of these have been refurbished, because of delays by US-owned refitter Wabtec.
This has both delayed and added to the training programme since the unrefurbished trains have features such as old-fashioned "slam doors".
Two electric trains being maintained caused "short formed" services on other routes.
ScotRail has only received around half its 70 brand new electric trains because of delays by Japanese manufacturer Hitachi.
Scottish Labour transport spokesman Colin Smyth said: "SNP Cabinet Secretary Michael Matheson must personally take responsibility for the miserable level of service Scotland’s hard-pressed rail passengers are receiving and apologise to them.
“By issuing a waiver to dodge their performance targets in September instead of demanding improvements, Michael Matheson personally gave ScotRail a licence to fail and since then performance has continued to plummet.
“The SNP Government joined forces with the Tories in the Scottish Parliament in November to vote down a Labour’s proposal to strip Abellio of the ScotRail franchise, so few people will believe Michael Matheson’s is serious.
"The reality is Michael Matheson has turned a blind eye to ScotRail’s appalling record of failure, and as a result passengers are suffering growing delays, cancellations and overcrowding and have been hit by yet another fare rise.
"With prices rising and performances plunging, it is no surprise more and more people are backing Labour's call to bring the end of Abellio’s franchise now and to bring our railways into public ownership so we have a railway system run for passengers, not profits."
Scottish Greens transport spokesman John Finnie MSP said: "It’s extraordinary it is has taken month after month of mismanagement by Abellio for the transport secretary to take this situation seriously.
"Talk of improvement plans will be of little comfort to commuters who’ve heard such promises before and who cannot rely on the rail services they need to get them to and from their work.
"We’re long overdue putting this public service back in public hands.”
Scottish Liberal Democrat transport Mike Rumbles MSP said: “Two years ago,the previous transport minister promised he had a plan in place for getting Scotland’s railways rolling.
"Now, his successor stands up and makes a remarkably similar promise.
“The public can have no faith in this SNP Government’s command of our railways when plans and promises come and go but performance continues to stall.
“We need a railway system that delivers for passengers who are paying through the nose for a substandard service.
"If there isn’t an immediate improvement, Abellio should be stripped off the franchise at the first possible opportunity.”
Scottish Conservatives transport spokesman Jamie Greene said: "The transport secretary is in a permanent state of apology over Scotland’s rail.
"Performance Improvement and remedial plans seem to be the SNP’s battle cry, as well as blaming everyone but themselves for failing the travelling public.
“Finally, Mr Matheson puts some meat on the bone over what sanctions are open to him under the conditions of the current franchise agreement.
"I think it is right to give the ScotRail Alliance the chance to meet its contractual obligations, but it must know that all options are now on the table, with much at stake if they can't.
“I am far less concerned about who owns the railway but instead how it performs and how it is managed.
"The Cabinet secretary has laid down the gauntlet to the current operator and, like many, I wait with baited breath to see if it will rise to the challenge."
ScotRail Alliance managing director Alex Hynes said: “We know the service our customers have received hasn’t been good enough recently.
“Everybody at the ScotRail Alliance is working flat out to deliver the improvements our customers expect and deserve.
"ScotRail will submit a remedial plan to Transport Scotland within the timescale outlined.”