Dominic Raab said he wanted to “manage expectations” about the plan to give people deemed to have arrived in the UK illegally a one-way ticket to the east African nation.
When announced last month, Boris Johnson said tens of thousands of people could be flown there under the deal in the years ahead.
Asked when the first deportation flights to Rwanda will take place, Mr Raab told BBC Radio 4’s Today programme that it would happen “as soon as possible”.
“I think that we’ll have to wait and see how operationally it works in practice,” he said.
“I think I’d be careful about managing expectations.
“It is not going to deal with the whole problem.”
Asked if hundreds or thousands of people would be removed every year, Mr Raab added: “I would have thought it was more likely to be in the hundreds.”
The Home Office previously disputed suggestions that modelling by its own officials indicated that only 300 people a year could be sent to Rwanda.
Asked about the report in The Times, the department said it did not recognise the figure and there was no cap on the number of people removed under the arrangement.
Since the start of this year, 8, 803 people have reached the UK after navigating busy shipping lanes from France in small boats, according to analysis of Government data by the PA news agency.
There were 106 people who arrived in three boats on Thursday, with more crossings thought to be under way on Friday.
Campaigners who have lodged legal challenges against the Rwanda policy said they received notice that first flights will now not take place until at least June 6.
Last weekend the Prime Minister said 50 asylum seekers had been told they were due on a flight within a fortnight, which would be the end of May, but he anticipated opposition to the move.
According to the Home Office, Home Secretary Priti Patel and Rwandan foreign minister Vincent Biruta reinforced their “commitment to working in collaboration with UN agencies” on the deportation plan and “emphasised” that claims will be processed in accordance with the UN Refugee Convention, in meetings in Geneva with the body’s high commissioner for refugees Filippo Grandi.
But afterwards Mr Grandi said on Twitter he had reiterated his concerns about the deal and that the UNHCR “will continue proposing concrete solutions that respect international law”, adding: “Shifting asylum responsibilities is not the solution.”
Meanwhile it emerged Ms Patel will visit Linton-on-Ouse to hear concerns from locals about a reception centre for 1,500 asylum seekers.
Home Office officials were told the North Yorkshire village was “in crisis” amid hours of questioning by residents at a parish meeting on Thursday night over the use of the disused RAF base.
Around 60 asylum seekers will be arriving at the site by the end of the month, the department’s director of asylum accommodation centres Cheryl Avery said.
Hambleton District Council, which is seeking a judicial review over the plans, said it has called on the Government to pause the proposal “immediately”.