He said he was sorry if his words caused any offence to Liverpool supporters and relatives of the 96 people killed in the incident in April 1989.
Mr Hunt made his remark in an interview in which he was answering questions about England's poor performance in the World Cup.
He said he was "incredibly encouraged by the example set by the England fans. I mean, not a single arrest for a football-related offence, and the terrible problems that we had in Heysel and Hillsborough in the 1980s seem now to be behind us".
Margaret Aspinall, chairman of the Hillsborough Family Support Group, reacted furiously, describing Mr Hunt as "an absolute disgrace".
Mrs Aspinall, who lost her 18-year-old son James at Hillsborough, said: "After all these years of fighting for justice I am very angry that he has shown such ignorance of the facts."
In a statement, Mr Hunt said later: "I know that fan unrest played no part in the terrible events of April 1989 and I apologise to Liverpool fans and the families of those killed and injured if my comments caused any offence."
Mr Hunt was said to be "horrified" at the way his words came out during an interview he gave while attending an event to launch a schools sports competition linked to the 2012 London Olympics.
But Mrs Aspinall said she would not accept Mr Hunt's apology unless she was allowed to meet him and "explain the facts". She added: "I want him to understand that he has reopened old wounds which should have healed many years ago."
"The problem we have is that Hunt has influence and people listen to him."
The 96 fans died and hundreds more were injured when a lethal crush developed in the Leppings Lane end of Sheffield Wednesday's Hillsborough stadium during an FA Cup semi-final match between Liverpool and Nottingham Forest.
Lord Justice Taylor's official inquiry into the disaster, which reported in 1990, criticised senior police officers on duty at the match for a "failure of control" and recommended the introduction of all-seater stadiums.
A group set up to study previously secret documents relating to the Hillsborough tragedy met for the first time in February.
The Hillsborough Independent Panel will oversee the public disclosure of archives held by the emergency services, the city council and other public bodies in Sheffield.