Dr Fox insisted the Cabinet has not agreed a deal on immigration after withdrawal from the EU.
The remarks came as Chancellor Philip Hammond signalled there was “broad acceptance” in Cabinet of a wide-ranging post-Brexit transitional period lasting up to three years.
Dr Fox said: “We made it clear that control of our own borders was one of the elements we wanted in the referendum, and unregulated free movement would seem to me not to keep faith with that decision.”
The International Trade Secretary said he had not been involved in any Cabinet talks on extending free movement for up to three years after Brexit, stating: “If there have been discussions on that I have not been party to them. I have not been involved in any discussions on that, nor have I signified my agreement to anything like that.”
In remarks that are likely to be seen as directed at the Chancellor, Dr Fox said: “I am very happy to discuss whatever transitional arrangements and whatever implementation agreement we might want, but that has to be an agreement by the Cabinet.
“It can’t just be made by an individual or any group within the Cabinet.”
Mr Hammond said on Friday the transitional period up to 2022 would mean “many arrangements remaining very similar to how they were the day before we exited the European Union”.
The Chancellor said there would be a registration system in place for people coming to work in the UK after Brexit, during the transitional period.
“If they come here to work after we leave the European Union, during that transitional period, the sensible approach will be to seek to register people so that we know who’s coming and who’s going,” he said.
A government source said freedom of movement will end when Brexit occurs in March 2019, and a time-limited implementation period would then follow.
The source said that the idea of mass, uncontrolled migration is “not a vision of Brexit that we will undertake”.
However, the EU has made it clear that continued access to the single market is dependent on free movement of labour.
Tory tensions were also exposed when former Brexit minister David Jones branded Mr Hammond’s transition initiative “deeply dangerous” as he accused the Chancellor of “going on manoeuvres” while Prime Minister Theresa May was abroad on holiday.
“All this agitation by the Chancellor and his allies is hugely discourteous to her and undermines her authority,” Mr Jones said.
Pro-EU Tory MP Sir Nicholas Soames praised the Chancellor, saying: “He has restored discipline to the political infant class who want Brexit at any cost. Instead of mindlessly criticising him, they should thank him for putting a stop to what was in danger of becoming a pub brawl.”
The war of words came as an ally of Foreign Secretary Boris Johnson also criticised Mr Hammond’s plans.
Gerard Lyons, who was an economic adviser to Mr Johnson when he was London mayor, said a two-year transition period would work better.
He said: “Many of the ‘risks’ being highlighted about Brexit are perceived risks, not real risks. And a two-year transition would alleviate many concerns.”
Mr Johnson has not commented publicly on Mr Hammond’s transition plans.
Liberal Democrat leader Sir Vince Cable said Dr Fox’s remarks point to a Cabinet “civil war”.
He said: “There is no Cabinet consensus for moderation. And the rumours of Boris Johnson being about to resign fuel the uncertainty.”