Leader of the Commons David Lidington warned there would be “strong” reaction to the remarks as he stressed the Speaker must retain the confidence of the whole House.
Mr Bercow was plunged into fresh controversy after a video emerged of him talking to students at Reading University on February 3 in which he said: “Personally, I voted to Remain. I thought it was better to stay in the European Union than not.”
In the video, obtained by the Sunday Telegraph, Mr Bercow says that immigration has been a good thing for Britain
Mr Bercow also referred to “untruths” during the Brexit campaign, and how “promises were made that could not be kept”, and expressed hopes that Parliament would maintain changes to working hours and health and safety protections after Brexit.
With Mr Bercow already facing a vote of no confidence because of his branding US President Donald Trump a “racist and sexist” as he effectively banned him from addressing Parliament during his state visit, a number of Tory MPs say his position as Speaker is no longer tenable.
Mr Lidington told the BBC’s Andrew Marr Show: “There will be strong reaction among some MPs to what he said at Reading, particularly after what he said about the state visit earlier in the week. Ultimately, the Speaker has to command the confidence of the House of Commons as a whole.
“John has his very strong supporters as well as his strong critics in the House of Commons, but we shall have to see how members as a whole respond.
“It is really important for the very independence of the Speaker’s office that the Speaker, whether they start as a Conservative MP, a Labour MP, or whatever, is independent of Government. Speakers, if anything, should be towards the people who are not in Government, as, actually John Bercow probably has done in the way that he has used urgent questions that we have found inconvenient.”
The parliamentary website states: “The Speaker is the chief officer and highest authority of the House of Commons and must remain politically impartial at all times.
“On election the new Speaker must resign from their political party and remain separate from political issues even in retirement.”
Mr Bercow’s spokeswoman said that how the Speaker cast his ballot in the Brexit poll, or Strictly Come Dancing, had no impact on his ability to deal fairly with all MPs.
She told the Press Association: “Mr Bercow voted in the EU referendum, along with millions of others.
“The record shows that he has rigorously facilitated the raising of concerns of those on both sides of this argument, as he does on every other issue.
“The Speaker’s impartiality is required on matters of debate before the House, and he has been scrupulous in ensuring that both sides of the argument are always heard.”
The spokeswoman said Mr Bercow’s record showed he was neutral in the chamber “irrespective of how he voted in a referendum, general - one would hope for himself - local, or Strictly Come Dancing.”
James Duddridge is among Tory MPs saying it is not possible for Mr Bercow to now act impartiality over Brexit debates in the Commons.
The row came as it was reported that Mr Trump may address a mass rally during his state visit after being blocked from speaking to both houses of Parliament.
An event in Birmingham may see proceeds raised by ticket sales go to the Royal British Legion, it was reported.
Mr Duddridge, who tabled the motion of no confidence in the Speaker over his Trump comments, told John Pienaar the speaker is “no longer impartial”.
Speaking on BBC Radio 5live’s Pienaar’s Politics, he said: “I think there will be a vote of no confidence and I think he will go.
“There’s absolutely no way Speaker Bercow can sit in the chair on European issues.
“When you become Speaker you must be impartial. He’s no longer impartial, he’s no longer able to continue to do the role, which is why I think the House will vote him down in a vote of no confidence. In reality he may see the lie of the land and go before he’s pushed.”
He added he had been “amazed” by the number of people to privately voice to him their support for his motion.