The club were given the go-ahead for their new 20,000-seater home earlier this month after a judicial review at the Court of Session.
But that left locals living near the Kingsford site on the outskirts of the city furious.
They claimed the plot chosen by Pittodrie chiefs to replace their home of 120 years was an incursion into green belt land and insisted alternative locations were available.
But the No Kingsford Stadium group have now announced with “considerable regret” they will not pursue further legal avenues to block the project.
Instead they plan to focus their energies on making sure the club adhere to the strict planning conditions placed on the blueprints when they were approved.
In a statement, the group said: “We appreciate this is perhaps a difficult pill to swallow. We don’t want the stadium at all, but the facts are that the stadium is being allowed to progress, therefore we believe the best avenue for us to take now is to ensure that the conditions are rigorously enforced by Aberdeen City Council.”
But Milne has promised to work with all parties to ensure disruption for local communities is kept to a minimum.
In a statement posted on the club’s website, he said: “This is very welcome news which will come as a relief to our fans and the wider public who have supported the plans for Kingsford. We can now focus on phase two of the development - the stadium and associated infrastructure.
“We are working with the councils to clear the conditions attached to our planning consent. Our aim is to deliver a modern stadium that dramatically improves the matchday experience for our fans and is safe and easy to get to and from, with minimal disruption to neighbouring communities.
“This may be the end of the road for further legal challenge, but it’s the start of a journey for us and we will continue to engage with the local community to ensure that we address their concerns while delivering fantastic community and sports facilities.”