Dr William Seale, author and historian, said that Scots were sought out for the project given their expertise in stonemasonry and that a group of men who had been working on Edinburgh New Town were hired for the job.
They had been found through Edinburgh Lodge Number 1, the oldest masonic lodge in the world, which had a chapter solely made up of working stonemasons.
Around eight to 12 men went to Washington from Edinburgh in the early 1790s, led by master mason, Collen Williamson of Dyke, Moray.
“The Scots were the greatest stonemasons in the world,” said Dr Seale, whose book, A White House of Stone, has just been published.
“It was obvious to the commission [leading the building project] to try Scotland and they had a local merchant go to Edinburgh, where they were building the New Town.”
Dr Seale, speaking in an interview with the White House Historical Association, added: “There were about eight to 12 men and they were at the top of their field. They were working in the New Town, which exploded in the late 18th Century.”
They were to join a work force of which up to 25 per cent were slaves, Dr Seale added.
The men and their families lived in a workers village that sprung up in the shadows of the construction site. People were married there, babies were born and a community forged.
“The village was hive of human activity,” Dr Seale said.
Master mason Williamson was already in America by the time the team of Scots stonemasons arrived and first met the Washington commission driving the White House project in 1793.
His cousin ran the Fountain Inn, a busy tavern where men of influence met, and his role was secured.
“Williamson was difficult to get along with, but he was highly skilled at his job,” Dr Seale said,
He clashed repeatedly with James Hoban, the Irish architect hired to design the White House, with Williamson eventually sacked in 1795.
“Hoban was a young man, only 30, and Williamson was in 60s. He just didn’t think a man as young as Hoban knew what he knew.
“Williamson was very vocal but Hoban was quiet, he didn’t make a show.
“The commission eventually got sick of listening to Williamson.”
Dr Seale said the Craigleith stone used for Edinburgh landmarks such as Register House and Charlotte Square was similar to that used in the White House.
Used to working with porous material back home, they whitewashed the stone sourced from a Virginia quarry to protect it from water damage – and were thus instrumental in putting the white into the White House.
The White House was later rebuilt using the original stone after it was torched by British troops in 1814. White paint was then used to cover soot marks left by the fire.
Some of the “exquisite carvings” created by the Edinburgh stonemasons, including a “festoon of flowers” over the front door, can still be seen at the White House – as well as a special token of home.
“The one Scottish symbol that is incorporated in the White House by the Scots is a rose,” Dr Seale added.
“You find it in the caps of the Ionic columns. This is the most voluptuous rose you every saw. In 1780, Scots gardeners propagated the Scottish double rose. It took Europe by storm.
“The Scots were extremely proud of it. These men knew what they were doing, putting this rose in the columns. The Scots territory was established.”
Stewart McLaurin, president of the White House Historical Association, visited Edinburgh earlier this month to present a lecture hosted by Edinburgh World Heritage and Dr Brian Lang on the Scottish stone masons of the White House.