It comes after the latest tranche of partygate fines issued by the Metropolitan Police left Boris Johnson and the Cabinet Secretary unscathed.
Amid the episode's fallout and following bruising losses in last week's local elections, Mr Hunt told The Times Magazine it was not the "right time" for a leadership change due to the war in Ukraine.
"But I would be very open with you that I don't rule out a return in the future," he added.
Speaking to Times Radio, he also said the "setbacks" the party suffered in the local elections were not just "mid-term blues" but reflected the cost-of-living crisis.
"Underneath it, I think the reason that we got such a kicking was economic concerns that many families had," he said.
"We are faced with a situation now where we have very, very low underlying growth in the economy.
"To win an election, the Conservative Party has to promise a well-funded NHS and the prospect of tax cuts. If we make people choose between one or the other, we're not going to win the election."
Earlier on Thursday, it was announced the Met had made around 50 further fixed penalty notice referrals as part of its investigation into possible lockdown-breaking parties in Downing Street and across Whitehall.
It brought the number of fines to more than 100 - with Mr Johnson, his wife Carrie and Chancellor Rishi Sunak hit with fixed penalty notices in April over a birthday party held for the Prime Minister in No 10's Cabinet Room in June 2020.
Deputy Labour leader Angela Rayner said Mr Johnson broke his own rules on a "record-breaking scale", adding that "Britain deserves better".
Mr Johnson's official spokesman said following the Met's update that neither the PM nor Cabinet Secretary Simon Case were among those fined in the latest batch.
The Prime Minister confirmed he was not one of the most recent recipients, nor was his wife, following a Cabinet away day in Stoke-on-Trent.
Mr Johnson said he was "sure we'll have plenty to say about that when the thing's finished" after being asked about the extent of law-breaking in Downing Street as he arrived for a Cabinet meeting in Staffordshire on Thursday.
Ministers dodged questions on whether they still had faith in the PM when asked about the latest development at the same event.
Home Secretary Priti Patel failed to say whether or not she still supported Mr Johnson. Health Secretary Sajid Javid also chose not to answer when asked why the PM would not resign, despite Sir Keir Starmer's promise to do so if he is handed a fixed penalty notice over similar allegations.
There have been ongoing calls for the PM to resign over the saga from opposition MPs, as well as his own backbenches, but he has repeatedly said he is determined to get on with the job.
In addition to the Scotland Yard probe and the investigation by Sue Gray, a senior civil servant, into the partygate claims, the PM is facing a third inquiry by the Privileges Committee into whether he misled Parliament with his repeated assurances that Covid rules were followed in No 10.
Asked in Stoke-on-Trent if his previous statements about not knowing rules were broken could be true, given the extent of the police fines, Mr Johnson said: "As soon as I have any more to say about that, you will be among the very first to know."
Brexit opportunities minister Jacob Rees-Mogg made clear his continued backing of the Prime Minister, saying Mr Johnson "deserves" his support.
Culture Secretary Nadine Dorries and Education Secretary Nadhim Zahawi also said they still had faith in their leader when asked about the fines as they attended the Staffordshire meeting.