East Lothian Council called a halt to work on the cricket pavilion at Lewisvale Park after workmen uncovered two Roman altar stones, with one dating from the second century.
The find has been described as "the most significant of its type in the last hundred years".
Permission to undertake excavations for the building foundations and to lay services into the pavilion was required from Historic Scotland due to the fact that the pavilion lies partially within a Scheduled Ancient Monument area.
This permission was approved on the condition that a programme of archaeological work was carried out.
During the work, the two stones were unearthed along with a number of other archaeological features, including a lead bowl, fine pottery and some rough handmade pottery.
The artifacts have been removed by a specialist company for further investigation following the discovery last month.
Councillor Paul McLennan, cabinet member for community wellbeing, said: "The discovery of these remains is particularly exciting as it is not often that Roman altar stones are discovered during an archaeological excavation in Scotland.
"This helps with the emerging picture of life in and around the Roman fort at Inveresk during the second century. 2010 will be an exciting time to be involved with the heritage of Musselburgh, given the opening of the new Musselburgh Museum and the increased awareness of the Battle of Pinkie."
Both altars have been carved along the edges and on one face, with the position of the carved face suggesting that the stones were toppled at some point. Work is in progress to remove the soil to allow the ornate carvings to be analysed.
At least one of the altars is 2nd Century in date, and was dedicated to the Roman God Jupiter.
A spokeswoman for Historic Scotland said: "Inveresk is one of the most important Roman sites in Scotland, containing a large Roman fort and a civil settlement with a bathhouse and amphitheatre.
"The discovery of two almost intact carved Roman altars in Lewisvale Park is the most significant find of its type in the last 100 years of investigation and discoveries at Inveresk."
Work on the new cricket pavilion has now been allowed to restart following the discovery.