The Tory ex-leader said while Boris Johnson "probably" has the legal right to block a fresh independence ballot, he should be "wary how he uses that power".
The possibility of a second referendum is one of the key issues in the current Scottish election campaign.
Nicola Sturgeon's SNP is pushing for another vote, with former first minister Alex Salmond also now bidding to make a dramatic return to Holyrood as the leader of the new pro-independence Alba Party.
It is campaigning on the regional list section of the Scottish Parliament ballot, in the hopes that this will help create a "supermajority" of MSPs supporting Scotland coming out of the UK.
Sir John, writing in the Financial Times, stressed that Scotland "cannot be kept forever in an arrangement if her people wish to end it".
Instead he said: "To save the Union, Scots must be persuaded by hard facts that it is in their interest to do so."
The former PM continued: "Prime Minister Boris Johnson probably has a legal right to refuse to sanction a second independence referendum. But he should be wary how he uses that power.
"It is unwise to dismiss Scottish ambitions, or to delay any vote, without action to expose the reality of separation - and remedy shortcomings in the UK's devolution settlement."
Mr Johnson has so far rejected Ms Sturgeon's calls to give the go-ahead for another independence referendum, with the Prime Minister noting SNP leaders had said the 2014 vote would be a "once in a generation" event.
Sir John cautioned: "A blunt refusal would be a still greater error if accompanied by the provocative assertion that Scots should wait another generation before voting again on the issue.
"Such a hardline approach is more likely to provoke a break-up than prevent it. Scotland is a definable and proud nation. She is perfectly capable of self-government and, whether or not it would be a wise decision, has a right to seek independence."
He argued that "with facts and reason, it should be possible to persuade a majority of Scots that it is better to remain in the Union than to leave it".
But Sir John stressed: "The debate must be about the lives and livelihoods of the Scottish people. It must not be decided by Scottish distaste for Conservative rule, or by a trivial popularity contest between the Prime Minister and Scotland's First Minister.
"Fanciful promises did not persuade the Scots to back Brexit. Only facts will persuade them to rally to the Union. The Government must engage Scottish opinion so that every family and business is informed about what the Union does, and what would be lost by leaving it."
Scottish Conservative leader Douglas Ross argued against holding another referendum.