Immigration minister Robert Goodwill sparked a backlash after he raised the prospect of extending a scheme that will apply to non-EU workers from later this year.
He said the levy “may be something that’s been suggested to us that could apply to EU”.
Mr Goodwill also suggested a seasonal work scheme for fruit-pickers was also being considered.
The Prime Minister’s spokeswoman said the EU worker levy was “not on the agenda”, but not before the idea was widely condemned.
The Institute of Directors said the plan would damage the economy, while economists Graeme Roy and David Bell warned a levy would increase red tape and act as a barrier to skilled migration.
The SNP said the idea was an indication the government was pursuing the “hardest of hard Brexits”.