Humza Yousaf made the comments just weeks before the results of a wide-ranging consultation on raising the age of eligibilty.
He said it was important to find a balance with the scheme and ensuring affordability, due to Scotland’s ageing population.
Mr Yousaf said there was “concern around the longer-term sustainability” of the scheme, with Scottish ministers considering raising the age of eligibilty.
He added: “We know that we have an ageing population, an ageing demographic – as most of western Europe does – and therefore we have to find a balance between making this scheme fair, realising the benefits of it, and making it sustainable in the long term.
“That’s why we’ve had the consultation. We do have to look at that sustainability.”
Free bus passes were first introduced for over-60s and the disabled in 2006 under the last Labour-led Scottish Government.
Over 1million people currently benefit from free bus travel in Scotland, representing a third of all bus journeys made each year.
Speaking in front of a Holyrood committee, Mr Yousaf said there had been 3,000 responses to the Scottish Government’s call for views.
Scottish Labour insisted they would fight cuts to bus passes.
Rural economy spokesman Colin Smyth MSP said: “A Labour-led government introduced the free bus pass, allowing older people – particularly those in rural areas – and disabled people the freedom to travel where they choose.
“We will fight the SNP’s cuts to the bus pass. The SNP has already cut £10 million from the bus pass budget and launched a consultation on changing the age limit.
“Now Humza Yousaf seems content to freely admit the SNP plan to raise the age of eligibility for the bus pass. The number of bus users is already plummeting under the SNP and raising the age of eligibility will simply cut that number further leading to more and more services being axed.
“In its 2016 manifesto the SNP promised that the bus pass would continue and said nothing about cutting it back. The SNP has absolutely no mandate to make these cuts. To do so would be a betrayal of voters’ trust.
“Humza Yousaf should go and read his own 2016 manifesto and ditch his plans to cut the bus pass, which have never been endorsed by the electorate.”