There were even squabbles over the materials being used and, of course, the ever-escalating cost and constant delays. But the problems did not stop when the building was officially opened in October 2004.
Since then, the parliament has been plagued by leaky roofs, pigeon problems and a falling beam. The cost of maintaining the complex building is five times as much as expected and hundreds of thousands of pounds have been spent on new security measures.
Now one of the leading magazines for construction professionals has delivered a withering assessment after carrying out a "five years on" assessment of the building. Its overall rating is a paltry eight out of 15 marks.
Building magazine's review of the building is entitled "Miralles' magnificent mess" and rates the parliament three out of five for both functionality and build quality, and just two out of five for image.
The assessment says the Holyrood building failed as a national icon and brands it "an indecipherable jumble of forms". It praises the interior of the parliament as "exhilarating, inspiring and resplendent" but says the exterior granite and concrete are "dark, gloomy and growing even more dingy with weathering".
It also says the building's 100-year design life has been "severely undermined" by the roof beam collapse, persistent leaking roofs and the pigeon problem.
Independent Lothians MSP Margo MacDonald, who was a leading critic of the Holyrood project throughout its construction, backs the magazine's verdict.
She says: "It's a fair assessment and it underlines the fact 'we woz robbed' when it came to the cost and 'we woz conned' when it came to the description of the building as cutting edge in various ways."
She believes most of the public takes the same view of the place as she does.
"Every time more factual information comes out, you feel a shaft of anger and frustration but, for the rest of the time, you accept that for the moment anyway it's the only parliament building we've got so we've just got to get on with it."
The Holyrood saga saw the price of the new building soar from the 10m-40m quoted by Donald Dewar in the devolution white paper to 414m by the time it opened in October 2004.
After the opening, the problems just carried on coming. Four weeks after the opening ceremony, 154 windows had to be replaced because of a problem with the sealant.
Less than a year later, 12 specially-crafted oak doors were replaced at a cost of 1,000 each after they started to buckle.
Then, in March 2006, just as the controversy over the building seemed to be fading, a roof beam in the main parliamentary chamber came adrift from its moorings above the heads of MSPs during a debate.
The rows about the building returned with a vengeance and the chamber was out of action for two months.
By the time presiding officer George Reid announced official "closure" of the project, negotiations with contractors had reduced the cost from 431m to 414m. But even as he spoke, a bucket was catching the drips from a leaky roof in the lobby outside the debating chamber and officials admitted they were still wrangling with contractors over a serious water leak in the car park and a permanent puddle outside the committee towers.
Since then there has been massive extra expenditure on security – 223,000 for a triangular "roundabout" at the car park entrance, 300,000 on turnstiles to enter and exit the building and 1.5m on bollards and concrete benches now being installed in front of the building and up the Royal Mile.
Edinburgh Pentlands MSP David McLetchie says the magazine's overall assessment of the building is fair, though he would give it four out of five for functionality. "I don't feel squeezed in my office," he says. "It seems to me to have all that's required of a parliament."
He agrees, though, that the inside of the building is better than the outside.
"When you look at the design of the building, it might have a 100-year life in Barcelona, but it is going to struggle to have that in Scotland without high repair and maintenance costs, as we have seen," he adds.
The building has, however, won a number of awards, including the RIAS Andrew Doolan Award for Architecture and the 2005 Stirling Prize, the UK's most prestigious architecture award.
SNP MSP Linda Fabiani, who served on the Holyrood progress group overseeing the construction, is still proud of the parliament.
"As a user, I would score it higher," she says. "It is a marvellous space to work in and the international design prizes speak for themselves."
She claims visitors are largely positive about the place, too. There are people who don't like the outside but when they get inside they love it."
Whatever people's opinions of the building, it seems pretty certain the costs and the controversy will continue.
Holyrood bosses recently signed a new maintenance contract at a cost of 13m for the next eight years.