Heavy smoke and flames were seen at the military facility on Friday afternoon amid blaring alarms.
Asylum seekers who have been living at the barracks have been moved outside but do not know if it will be safe to return to their accommodation on Friday, the PA news agency has been told.
Home Secretary Priti Patel said the "damage and destruction" at the barracks was "deeply offensive to the taxpayers of this country" and vowed to support police.
A charity accused her of playing politics with what has happened and called on the Home Office to shut down the site.
Eight fire engines were sent to the scene along with police officers, and the blaze is now under control with firefighters damping down remaining hot spots.
Kent Police said no arrests have been made.
Police say there have been no reported injuries and enquiries remain ongoing to determine the cause of the fire.
One Napier Barracks resident told PA that when the fire broke out he saw that one of the accommodation blocks was full of smoke with people inside.
He and some friends tried to get them out but firefighters arrived after around 15 minutes and helped rescue them.
In that time the blaze spread to another building, he said.
"Everyone is panicked," he told PA through a translator, adding that he fears something similar may happen again in future.
The blaze came on the same day residents were reportedly told by the Home Office that no more people would be moved from the controversial barracks.
"We (were told we) are going to stay at this camp until we have a decision about our (asylum) claims and we should be isolated for ten days in our rooms."
There have been reports of suicide attempts at the barracks and more than 18,000 people signed a petition to shut the site down amid concerns over conditions inside.
The asylum seeker, whose identity has been protected, said he came from a country where he was being persecuted by the Government.
Asked about what living at Napier Barracks has been like, the man added: "I have suffered here more than I have ever suffered on earth.
"Now I am suffering more than I was."
He told PA that the "only dream" for many of them was to "get out of Napier".
Ms Patel issued a statement on Friday evening following the incident.
She said: "The damage and destruction at Napier Barracks is not only appalling but deeply offensive to the taxpayers of this country who are providing this accommodation while asylum claims are being processed."
In response, Kolbassia Haoussou from Freedom From Torture, said: "We are horrified at the news of a fire at the former army barracks in Folkestone being used to house asylum seekers.
"We hope all residents, staff and emergency workers are safe.
"By mocking the vulnerability of asylum seekers, the Home Secretary would rather shirk responsibility and play politics with people's lives.
"Many of the people trapped here suffer from severe mental health issues and low immune systems linked to the abuse they have fled.
"The camps are unsafe, unsanitary and unfit to house vulnerable people."