The move comes following concerns raised by rival operators and a formal complaint from an SNP MEP that government aid to NorthLink and Caledonian MacBrayne may amount to unfair competition.
It has come despite ministers in both the current and previous Scottish administrations complying with EC demands for CalMac services to be put out to competitive tender. CalMac was the sole bidder.
The inquiry was welcomed by the SNP Scottish Government, which said it related to the previous Labour-Liberal Democrat administration.
Jacques Barrot, the European transport commissioner, said: "I am well aware of the importance of lifeline ferry services to ensure the survival and prosperity of isolated island communities in Scotland.
"This is actually a very important reason why we should ensure full compatibility with European community law and provide legal certainty for the way these are organised."
The commissioner insisted the cash from the Scottish Government to ferry companies may turn out to be "fair compensation" for the provision of ferry services and fully in line with EU state aid rules. But he said the commission required more information to check the payment method in recent years.
A spokesman for the commission said the decision did not call into question the need for a "regular and affordable lifeline ferry service" for local communities, or threaten the continued provision of such essential services.
The UK authorities have been given two months to respond.
The complaints relate to subsidies paid to NorthLink, whose ferries operate between the mainland and Orkney and Shetland, and Caledonian MacBrayne, which operates to the Hebrides and across the Clyde. Both companies are now part of the same group, and both receive subsidies to cover loss-making routes.
Peter Timms, its group chairman, welcomed the inquiry. He said: "We fully understand the issues involved and that the Commission requires more information to check that mechanisms used in recent years do not fall foul of funding rules."
Gordon Ross, managing director of Western Ferries, which competes with CalMac between Gourock and Dunoon, said: "There will be transparency, accountability and hopefully the resolution of the anomalous situation on this route."
It is thought the result of the commission investigation will enable ministers to re-tender the CalMac service on the route after it attracted no bids.
SNP MEP Alyn Smith, who called for an inquiry, said it would be vital in giving clarity to talks on how ferry services are maintained. He said: "This will give everyone clarity, a clear timescale and process to argue this case. No doubt the SNP will argue this robustly with a view to obtaining a stable future for ferry users and companies."
However, Tavish Scott, the Lib Dem MSP for Shetland and the previous transport minister, said the investigation would "confirm that these services are compliant with the appropriate rules".
A Scottish Government spokesman said he hoped the inquiry would help resolve complaints over the payments.
He said: "We are absolutely committed to delivering first-class ferry services for our remote and island communities."
Q & A: SUBSIDY INVESTIGATION
Why has the European Commission launched this investigation?
It has received complaints that allegedly illegal government subsidies were paid to state-owned Scottish ferry companies which gave them an unfair advantage over rivals.
What do these involve?
One is over a rival freight service taking business from NorthLink Ferries, leading to the firm being given extra subsidies. The other is about Caledonian MacBrayne receiving subsidies for the Gourock-Dunoon route, where it competes with an unsubsidised private rival, Western Ferries.
What is at stake?
If the EC rules that illegal subsidies were paid, ministers could have to repay them. This could amount to tens of millions of pounds, because CalMac receives more than 2 million a year to cover its losses on the Gourock-Dunoon route.
Haven't ministers already complied with previous EC directives over subsidies paid to state-run ferry firms?
They have, but these complaints relate to specific issues.
Why is everyone involved welcoming this inquiry?
Because it is seen as a chance to settle this long-running issue for once and for all. The EC is expected to rule over whether subsidies were used for their proper purpose, to support "lifeline" ferry services, or to stifle competition.
Will this inquiry cost the Scottish taxpayer?
No. The EC is expected to bear the cost of its own investigation.
Why do Scottish ferry issues seem to generate so much argument?
Because there is often no viable alternative transport between their islands and the mainland, so ferry services can have a major effect on people's livelihoods. Large amounts of subsidy are also required on many routes because of their relatively small passenger numbers.