Appearing before the education committee, Mr Swinney said he was "wrestling" with the issue and had met chair Lady Smith to discuss it.
Survivors have repeatedly called for the remit to be extended to cover all abused children and not just those who were in residential care.
The inquiry has been repeatedly criticised by campaigners unhappy about the remit and the issue of financial redress.
Mr Swinney appointed judge Lady Smith to lead the inquiry last month after Susan O'Brien resigned from her position earlier this year, accusing the Scottish Government of interference.
Responding to a question from Labour MSP Johann Lamont, the committee's deputy convener, Mr Swinney said he was considering the remit and "looking very carefully" at the issue of financial address for those who suffered abuse.
But he said there were a number of survivors who held concerns that extending the remit of the inquiry would delay its conclusion.
He said: "I have personally discussed the question of extending the remit with Lady Smith. I take it very seriously but I have to be mindful of a number questions in relation to the remit.
"If I extend the remit of the inquiry, I would inevitably be extending the length of the inquiry. I have to mindful of the view of survivors who want this exercise to be proceeded with and not to become something that is longer than it needs to be because they want to get progress on these questions.
"The dilemmas on this are not easy because they ultimately come down to the fact it's unavoidable the length of the inquiry will be extended if I extend the remit. That's a significant issue with which I am wrestling."
Ms Lamont said a "lack of confidence" in the inquiry was "sapping the energy" of survivors and those who want to see justice.
And she called for the committee to look in more detail at the departure of Ms O'Brien.
Mr Swinney said he had moved to address concerns over a support fund for survivors and said the Scottish Government had yet to deny the inquiry anything on the basis of cost.
He said: "In all circumstances, the request for approval of financial measures, they have been approved. None have been rejected by the government. We may have asked some questions, but ultimately they have all been accepted."
Speaking after the meeting, Alan Draper, of In-Care Abuse Survivors (Incas) said the Scottish Government's position on the remit and the issue of redress was becoming "increasingly tiresome".
He said: "The government continue, in my opinion, to prevaricate. They're being dishonest because they're dragging it along, which is pretty appalling for survivors, when I think they have already made up their minds about not extending the remit."