Mr Jack said that Nicola Sturgeon “doesn’t speak for Scotland” and that a physical border with Scotland might split families.
"We have a geographical border between England and Wales, and England and Scotland,” he said.
"To create a physical border and an economic border would be a terrible mistake. There are people crossing the border every day to go to work or school.
"There are children in my Dumfries and Galloway constituency who go to school in Carlisle. We all have family and friends who live in different parts of the UK. To break that up would be a terrible sadness."
In an interview in The Express, the Cabinet minister said he believed his party would do well at the Holyrood elections. He said: "Now is not the time to have a constitutional upheaval.
"The division and uncertainty would damage business. It's not in our thinking at the moment to do anything other than help the country recover from the pandemic and save jobs."
Asked if Whitehall would launch legal action against a “wildcat” poll on independence, Mr Jack said: "Yes. There are many reserved matters and the constitution is one of them.
"It's entirely a matter for the UK Government "
He also advised voters to boycott any unofficial poll, saying: "The SNP is a campaigning organisation to break up the UK masquerading as a party of government.
"We believe in strengthening devolution, they believe in separation."
Last month The Scotsman revealed SNP ministers had not sought or received any legal advice on their draft independence referendum bill that could pave the way for a potential poll.
Reports had suggested the Scottish Government was set to announce the timings and question for a second independence referendum by the end of March. But a Freedom of Information request confirmed the Bill had not been checked by external or civil service lawyers.
Mr Jack also said he worried that Scottish independence could lead to the collapse of the UK. "If Scotland went, would Northern Ireland want a border poll?” he said. “Would Wales feel that they could go? It certainly would not strengthen the UK. The UK is stronger than the sum of its parts. It is an enduring partnership that is over 300 years old."
However, he said he believed the success of the vaccination programme was dampening demand for independence.
SNP Depute Leader Keith Brown said Mr Jack’s "attempt at Trump-style denial of democracy” was “completely unsustainable”.
He added: “It is increasingly clear that the Tories know there will be an independence referendum if the people of Scotland vote for one in May.
“Boris Johnson wants a free hand to strip powers from the Scottish Parliament and impose his disastrous hard Brexit. In an independent Scotland, the people who live here – not Westminster governments we don’t vote for – will decide how we rebuild our country after the pandemic.”
A message from the Editor:Thank you for reading this article. We're more reliant on your support than ever as the shift in consumer habits brought about by Coronavirus impacts our advertisers.