He warned that the site of the last full-scale battle on British soil could effectively become an “island” unless it was treated as “hallowed ground”.
Oliver, president of the National Trust for Scotland (NTS), which is responsible for the site, said the Government had set an “unfortunate precedent” in overturning a planning decision by Highland Council and allowing 16 homes to be built on neighbouring land.
Some historians claim the site of the 1745 battle should have been protected because it is effectively a war grave.
Speaking at the Edinburgh International Book Festival, Oliver said the Government had approved the housing “without a backward glance”.
He said: “That means anyone is now more likely to get permission to build in the vicinity of the battlefield. It’s the scene of the last pitched battle on British soil and yet it could well be compromised by modern development. Do we love the landscape or not? If we do love it, and are not just saying it, we have to act like we do.”
He later told The Scotsman: “It has created an unfortunate precedent. This housing development is not on the battlefield but it is on the fringes of it. Without a shadow of a doubt, during the battle and in its aftermath it would have been walked, run and trampled over by people involved in the engagement.
“The battlefield has been really looked after. People come from all over world to the landscape and it has a beautiful new visitor centre. A lot of money and a lot of thought has gone into it.
“But if it is to make it more likely that there will be further development then the special integrity of the site could be compromised. It would become an island. It must make further developments more likely.
“Culloden battlefield is a special place. It should be regarded as something that has to be looked after. The battlefield and the context in which it sits should be looked after and preserved forever.
“It’s a jewel and you wouldn’t want to see that compromised in any way.”
The NTS has called for an overhaul of the country’s planning laws in the wake of the case, branding approval for the project “the wrong decision in the wrong place”.
Oliver added: “I’m not a legislator. I’m not involved in the day-to-day running of the National Trust for Scotland. But I do feel you either care about the landscape or you don’t. It’s not something you just say. It’s about the way that you act.
“We either have places in the landscape that we consider to be hallowed ground, special places of historical significance that can be looked after for future generations, or we don’t.”