Michael Matheson has today warned the crossing would "waste significant money" which should instead be devoted to other "practical, deliverable" projects.
The criticism is set out in a letter to UK Transport Secretary Grant Shapps which also points out that transport is devolved to both Scotland and Northern Ireland.
A feasibility study has been ordered into the possibility of a 20 mile bridge which would connect Portpatrick in the South west coast of Scotland with Larne in Northern Ireland.
Nicola Sturgeon has been sceptical in her response, but said she was not opposed to looking into the plan.
But, Mr Matheson's letter today takes a harder line.
"I am writing to you to express my concerns about these proposals," he states.
"In both Northern Ireland and Scotland, budget constraints from successive UK Governments, have for a number of years restricted necessary investment in public transport and vital infrastructure and held back progress for our communities."
It adds: "I strongly believe that if £20 billion is available for investment in infrastructure in Scotland and Northern Ireland that rather than indulging the Prime Minister with this vanity project,
such funding should be made available to our respective governments so it could be better spent on meeting the priorities of the people we represent."
A number of proposals are already "on the table" in Scotland which could benefit from such funding, Mr Matheson added.
One proposed idea could see a bridge-tunnel hybrid scheme put in place, which would fuse at Beaufort's Dyke, a few miles off the Scottish coast, but this was a major world war II munitions dump.
"I am concerned that pursuit of this in light of the already identified technical, logistical and safety challenges such as the Beaufort’s Dyke munitions dump could waste significant money and resource that could be put to better use on practical, deliverable projects," Matheson adds.