The Cairngorm stone – the most valuable of its kind – was given to Victoria during her stay at the Scottish palace in 1851 by local man James Grant.
His family were from the area in Aberdeenshire and were moved during the Highland clearances.
James found the quartz near Ryvoan in the Cairngorms and carried it to Balmoral over the mountains on horseback.
He was rewarded with £50, “£1 per weight”, which amounts to about £3,700 in today’s money.
In recent years, James Grant’s descendants believed the stone to be lost, until it was discovered to be safely stored at the Queen’s summer home.
The family, who are from the Stonehaven area, Aberdeenshire, were invited to attend a church service with the Royal family last Sunday along with other descendants of James Grant.
Kathryn Briggs, 31, her brother Finlay, 33, parents John, 63, and Sue, 60, and daughter Lily, six, all took the opportunity to meet some of their relatives for the first time.
Kathryn said: “James Grant was my great-great-grandfather.
“The family line has branched off in several directions, and we all met up, many of us for the first time.
“The stone itself has been a tale that has been passed down over generations.
“We didn’t know where the stone was but it transpired no one had ever contacted the monarchy to find out where it had went.”
25 members of the Grant family attended the church service on Sunday morning, where they were joined by Queen Elizabeth II, Prince Philip, Prince Andrew, the Duke and Duchess of Cornwall, Prince Andrew, Sophie, Countess of Wessex, and their children, Louise and James.
After a service lasting an hour and a half, the family were invited to stay behind to view the Cairngorm stone at the church.
Kathryn said: “It was an amazing day, and we were seated just metres away from the Royals.
“We didn’t know that the Royal family were going to be there but we thought they might be.
“But the emphasis was on this stone that our direct ancestor had given to Queen Victoria herself.
“It was the first time many had seen the stone. For me it was a real honour because I love the Royal family.
“I’m a self-confessed royalist, so for me it is quite an amazing thing to have a connection.
“It was a really great experience because not only did we meet family members we had never met, we got to see this stone that had been a tale in the family for years.
“After the service my brother Finlay, who had travelled over specially from his home in Norway, took the opportunity to hold the stone.
“We realised that he was the first Grant to hold the stone since it was delivered to Balmoral 165 years ago.”
It was originally thought that a piece of the stone was chipped off to produce a brooch for Prince Albert, Kathryn said this was inaccurate, and the stone remains intact as it was presented 165 years ago.
It is thought that the Queen will be in residence at Balmoral until September.