Mr Matheson also told MSPs he no longer believed the train operator franchise was "fit for purpose".
The Transport Secretary was being quizzed by MSPs about the £18 million remedial plan which troubled operator ScotRail has put in place after being in breach of key targets on performance and cancellations.
ScotRail fell to a rating of 79 per cent overall in recent National Rail Passenger Survey results, missing its target of 88.5 per cent. If it fails to improve its performance the firm could have its contract, which is due to end on 2025, removed.
The remedial plan sets out deadlines for performance improvements by next year and includes hiring an extra 55 drivers and 30 conductors this year, as well as the addition of three classic high-speed trains to provide extra resilience.
Asked when passengers might see a difference in the train service, as there have already been two improvement plans laying out 269 measures, as well as the new remedial plan, Michael Matheson said: "I expect to see Scotrail delivering sustained improvements. That needs to happen now to ensure customers see improved reliability, fewer cancellations and more seats on affected routes.
"The remedial plan is specifically to address the issues relating to where ScotRail is in breach of the franchise agreement and to take very specific measures to address that breach."
He added: "The time-frame is to May 2020 because it takes around a year for the calculation of figures to work through the system by the very nature of how the franchise operates, which I no longer believe is fit for purpose or serves the travelling public well.
"But if ScotRail fails to deliver on the commitments set out in the remedial plan which is now part of the contract - these are contracted commitments that ScotRail has now given as a result of the remedial plan, then they will be in default of the franchise, and if they are in default then at the end of this remedial plan we will be in a position where we can terminate the contract.
"The remedial plan creates a contractual obligation on ScotRail to deliver these improvements and if they fail to do so they will be in default and at that point the government can make a decision to terminate the contract."
However Scottish Liberal Democrat transport spokesperson Mike Rumbles said a "cast iron guarantee" was needed that the franchise will be removed from ScotRail if performance targets are not met at the break point in the contract next April.
He said: “Passengers are sick and tired of putting up with late trains, trains where you can’t get a seat and the substandard service provided. Ministers are now warning that performance is unlikely to reach acceptable levels until May 2020.
“Rather than endless initiatives and little real improvement, the public want a cast iron guarantee that if performance levels continue to be breached at the contract break-point in April next year, the government will remove the franchise from ScotRail.”