Sir John told guests at a private dinner that the views of the 48% who voted to remain in the EU must be taken into account in the forthcoming negotiations.
His comments, reported in The Times, were roundly condemned by Iain Duncan Smith - one of the leading Eurosceptic rebels on the Conservative benches when Sir John was prime minister.
“You can’t claim democracy when you want it and reject it when you don’t. We had a vote, that vote now has to be acted on, that’s all,” he told BBC Radio 5 Live.
“The idea we delay everything just simply because they disagree with the original result does seem to me an absolute dismissal of democracy. And that’s what I thought John Major’s comments were today. The tyranny of the majority? What’s the tyranny?”
Peter Lilley, one of the-then Eurosceptic cabinet ministers famously dubbed “bastards” by Sir John, told BBC Radio 4 ‘s The World at One: “It is odd for a democrat to be against majority rule.
“Obviously, John Major has changed his views since he used to get elected on majorities.”
While Sir John was said have accepted the UK would not remain in EU, he expressed the hope it would stay as close as possible to the other 27 member states, adding there was a “perfectly credible case” for a second referendum.
“I hear the argument that the 48% of people who voted to stay should have no say in what happens,” he said.
“I find that very difficult to accept. The tyranny of the majority has never applied in a democracy and it should not apply in this particular democracy.”
The disclosure of comments came after his successor in Downing Street, Tony Blair, suggested the Brexit process could be halted if voters decide the “pain-gain cost-benefit analysis doesn’t stack up”.
The two ex-premiers were supported by former deputy prime minister Nick Clegg who said the Leave camp had deliberately refused to spell out the terms on which the UK would withdraw from the EU during the referendum campaign.
“Of course we should involve a needlessly self-harming version of Brexit,” he told The World at One.
“Underpinning the almost hysterical fervour of the hard Brexiteers is an ideological wish to turn the United Kingdom as it leaves the EU into a low regulation, low tax, enlarged offshore Singapore for which they have no mandate.”
The latest row erupted as the Maltese Prime Minister Joseph Muscat - whose country takes over the EU presidency in January - said the other 27 member states were adamant the UK could not remain in the EU single market unless it continues to accept free movement of labour.
“All of us have been pretty clear in our approach that we want a fair deal for the UK but that kind of fair deal can’t translate itself into a superior deal,” he told the BBC.
“I know that there is absolutely no bluffing from the European side, at least in the council meetings I have attended, saying ‘We will start in this position and then we will soften up’. No, this is really and truly our position.”
Responding to his comments, a Downing Street spokesman said: “This is a negotiation that will take place next year and the Government will set out its negotiating strategy in the fullness of time.
“The aim of that negotiation is to get the best possible deal for Britain, for British companies to access and work with and within the single market and for European businesses to have the same access here.”