Dr William Seale, author and historian, said that Scots were sought out for the project given their expertise in stonemasonry and that a group of men who had been working on Edinburgh New Town were hired for the job.
They had been found through Edinburgh Lodge Number 1, the oldest masonic lodge in the world, which had a chapter solely made up of working stonemasons.
Around eight to 12 men went to Washington from Edinburgh in the early 1790s and were led by master mason, Collen Williamson, of Dyke, Moray.
“The Scots were the greatest stonemasons in the world,” said Dr Seale, whose book, A White House of Stone, has just been published.
“It was obvious to the commission (leading the building project) to try Scotland and they had a local merchant go to Edinburgh where they were building the New Town.”
Dr Seale, speaking in an interview with the White House Historical Association, said up to a dozen men left Scotland for Washington.
“There were about eight to 12 men and they were at the top of their field. They were working in the New Town which exploded in the late 18th Century.”
The building work was, however, to slow down with the outbreak of the Anglo-French wars, Dr Seale said.
“These men were about to go broke. They took the offer to built the White House , they illegally left the country from the west shore and arrived in Norfolk (Virginia) and very quickly got to work,” he said.
They were to join a work force of which up to 25 per cent were indentured slaves, Dr Seale added.
The men and their families lived in a workers village that sprung up in the shadows of the construction site. People were married there, babies were born and a community forged. Gardens separated rows of huts, people came from across Washington to shop at its markets, and horse races were held.
Churches of various faiths were created an a masonic lodge, which still exists but at different premises, was founded.
“The village was hive of human activity,” Dr Seale said.
Master mason Williamson was already in America by the time team of Scots stonemasons arrived having emigrated after his business in Scotland fell on hard times.
He first met with the Washington commission driving the White House project in 1793.
His cousin ran the Fountain Inn, a busy tavern where men of influence met, and his role was secured.
“Williamson was difficult to get a along with, but he was highly skilled at his job,” Dr Seale said,
He clashed repeatedly with James Hoban, the Irish architect hired to design the White House, with Williamson eventually sacked.
“Hoban was a young man, only 30, and Williamson was in 60s. He just didn’t think a man as young as Hoban knew what he knew. Williamson was very vocal but Hoban was quiet, he didn’t make a show.
“The commission eventually got sick of listening to Williamson.”
Williamson left the White House in 1795 and was replaced by the Englishman James Blagden.
Dr Seale said the Craigleith stone used for Edinburgh landmarks such as Register Houe and Charlotte Square was similar to that used in the White House, which took more than eight years to build.
As a result, the Scots were instrumental in putting the white into the White House.
Used to working with porous material back home, they whitewashed the stone sourced from a Virginia quarry to protect it from water damage.
“The Scots were familiar with white washing stone. The white wash was thick and made with flattened beer and limestone. It was painted on with brooms.
“The idea was that years would pass, the rain would wash off the surface, but leave it in the nooks and crannies so that water wouldn’t get in and freeze and split the stone,” Dr Seale said.
The White House was later rebuilt using the original stone after it was torched by British troops in 1814. White paint was then used to cover soot marks left by the fire.
Some of the “exquisite carvings” created by the Edinburgh stonemasons, including a “festoon of flowers” over the front door, can still be seen at the White House - as well as a special token of home.
“The one Scottish symbol that is incorporated in the White House by the Scots is a rose,” he added. You find it in the caps of the Ionic columns. This is the most voluptuous rose you every saw.
“In 1780, Scots gardeners propagated the Scottish double rose. It took Europe by storm.
“The Scots were extremely proud of it. These men knew what they were doing, putting this rose in the columns. The Scots territory was established.”
Dr Seale said it was not clear what happened to the Scots stonemasons once the White House was finished.
“I wish I knew. These were substantial members of the community. I wish they had told their stories, but I suppose after so many generations it is lost to the wind,” he said.