The busy route is a single carriageway from the outskirts of Inverness to Inverurie, more than 80 miles away, apart from the occasional crawler lane.
Only three years ago Scottish Executive officials insisted not enough vehicles used the road to justify dualling it. They said it could even handle 50 per cent more traffic.
But Mr McConnell has revealed that the Aberdeen to Inverness road is likely to be the next big upgrade in Scotland.
Speaking at an "Ask Jack" forum hosted by the Aberdeen-based radio station Northsound, Mr McConnell said the state of the road between the two cities was a "huge issue". And he stated that while other routes had a case to be given priority, his own view was that the Aberdeen to Inverness road was "the most acute example".
He continued: "It's probably the most expensive, given the nature of the road.
"There's a lot of pressure just now for the A9 between Inverness and Perth to be the next big investment. But I suspect Aberdeen to Inverness is probably - given the economy of that part of the north of Scotland and the challenges there, and the state of the road - the next big project that's required."
Mr McConnell's backing for an upgrade was welcomed by politicians and campaigners at both ends of the A96.
Alison McInnes, the chair of Nestrans, the north-east regional transport partnership, said: "We've consistently highlighted the need for improvements to the A96, and this is a welcome signal that it's becoming a top priority.
"Nestrans has been involved in studies to support the case for road and rail enhancements between Aberdeen and Inverness, and we'll continue to push for investment for those."
Howard Brindley, the co-ordinator of Hitrans, the transport partnership for the Highlands, said: "Hitrans welcomes any improvements to the transport infrastructure serving the north of Scotland and the A96 is an important route.
"However, the A96 is just one of three routes which connect us to the Central Belt. We have already identified the importance of upgrading the A9 and the A82 and welcomed the recent announcement that 30 million was set to be invested in improvements to the A9 and a similar amount in the A82."
Richard Lochhead, the Scottish National Party MSP for Moray, also welcomed the First Minister's comments.
He said: "The people of Moray will welcome the growing cross-party support and momentum for upgrading the A96 and dragging this vital transport link into the 21st century.
"I do hope Mr McConnell's belated recognition of the need to improve the A96 is not just him getting carried away in the run-up to May's elections."
Nanette Milne, Tory MSP for the North-east, said: "This is very encouraging and exceptionally long-overdue news. It is one of the most dangerous roads in Scotland."