He told a businessmen’s lunch that Scottish companies and multinationals based in Scotland were looking at moving south or out of the UK.
His comments will fuel the debate over new water charges which have left some companies facing increases of up to 500%. CBI Scotland is among a number of organisations calling for Scottish Water to be privatised.
Jones said his members were deeply concerned about the increased cost of water, compared to charges in England which he said were "contributing to their competitive disadvantage".
He also attacked business rates which he said were also a "stumbling block to positioning Scotland as a truly good place to do business" as the rate poundage was significantly higher than in England.
"All Scottish business wants is a level playing field on which to compete with business across the Border, much as we had between 1995 and 2000 (when the uniform business rate applied).
"I have been disturbed on my visit over the past few days that overseas companies and, yes, Scottish companies, are saying that moving south or away from the UK altogether is becoming seriously likely because they fear the environment for doing business here in Scotland is less easy, less competitive. That’s a serious worry."
Speaking after his address to the CBI annual lunch in Edinburgh, he said he had heard threats of leaving from two indigenous Scottish firms and two overseas companies with big operations in Scotland.
He said: "Another worry is that companies may not move away but will choose to switch investment plans away from Scotland, and that would impact on new jobs and existing ones."
In his address, he said: "We realise Scottish Water faces a difficult task in dealing with accumulated indebtedness and an infrastructure which has suffered from public sector under-investment for decades. We appreciate that the new operation has only been there for a year. But we cannot go on like this. It is time for the Scottish Executive to look again at the ownership and funding questions."
Jones’s comments turn up the heat on the Executive following outrage over water bills. BP Grangemouth is paying several million pounds more than a similar-sized refinery in England. Small firms across all sectors have also objected to current water charges.
Jones was in Scotland last week visiting members in the Shetland and Aberdeen oilfields. He rounded off his tour with his speech to the first CBI Scotland annual lunch where he raised a number of concerns.
He was highly-critical of Executive policy on planning including third-party rights of appeal - and red tape, though he avoided the sort of antagonistic language that got him into bother after last year’s CBI Scotland annual dinner in Glasgow. On that occasion, Ross Finnie, the agriculture minister, called him an "English prat" after he accused the Scots of being anti-English.
On Friday, Jones praised Scottish sectors of industry, such as financial services, for their record of performance and their status in the world economy. He welcomed moves to tackle the transport crisis, while stressing that CBI members would be demanding more than "warm words".
But he said his members should keep a close eye on delivery. "We need to watch what politicians do, not what they say," he said.
His comments will reawaken concern expressed in the run-up to devolution, when a number of companies said they would consider leaving Scotland, particularly if there was any likelihood of higher taxes that would undermine their competitive position. Companies such as Weir Group, the Glasgow engineering company, led the charge, while others preferred to raise their concerns anonymously through CBI Scotland.
Murdo Fraser, the Scottish Conservatives’ enterprise spokesman, said Jones’s comments were of great concern. "It confirms our own suspicions of the matter in our talks with the business community," he said.
He has already supported CBI Scotland’s calls for a rethink of the ownership of Scottish Water and said it should be mutualised. "Water charges have replaced business rates as the number one concern. Since this year’s water charges went out I have received dozens of complaints from small firms," he said.