The duke, flanked by his daughters Princesses Beatrice and Eugenie, thanked well-wishers for lining the route back to Balmoral after the royal family attended a prayer service.
It was the first time the members of the royal family had been seen together in public since the Queen’s death on Thursday.
Asked by a mourner how things were, Andrew said: “We’ve been allowed one day, now we start the process of handing her on.”
He added: “It’s nice to see you, thank you for coming.”
On Sunday, the Queen’s coffin is expected to be taken by road to the Palace of Holyroodhouse in Edinburgh.
Andrew and his two daughters were joined by two of the Queen’s children, the Princess Royal and the Earl of Wessex, the Countess of Wessex, Anne’s husband Vice Admiral Sir Tim Laurence and the Queen’s grandchildren Peter Phillips, Zara Tindall and Lady Louise Windsor at the service on Saturday.
It was a prayer service held at the nearby Crathie Kirk church, a place the Queen visited regularly during her stays at Balmoral.
Shortly after the service finished at about 2.30pm, the family began a sombre walk across the bridge over the River Dee towards the castle.
They were seen nodding and waving to the members of the public who had lined the route towards the castle.
After Andrew spoke to the public, the family walked over towards the vast collection of flowers and tributes that had been left at the gates.
The crowd fell silent as they intently read messages and discussed the bouquets with each other.
At one point, Princess Eugenie laid a bunch of flowers, and was seen being comforted by her father as she wiped tears away from her eyes.
After being at the scene for around six minutes, the family stood at the castle gates facing the public and waved, prompting them to break into a spontaneous round of applause.
Andrew also put his hands together in a prayer position and briefly bowed to the crowd to give his thanks.
One woman then said: “How lovely.”
A large crowd had gathered at Balmoral ahead of the visit, with locals including parents and children, some carrying roses, convening at the castle gates to add their floral tributes.
Ross Homer, who serves as a royal engineer in the army, spoke with members of the royal family during the visit.
The 42-year-old had come to Balmoral from Kinloss with his partner Kate, 38, and two daughters Laila, 14, and Roxy, 12.
Speaking about the interaction, he said: “Everyone got out the car and started walking up the bridge.
“There was Andrew, Edward, Sophie and their daughter at the front. We said ‘sorry for your loss’ as they walked past and they said ‘thank you’, and as they carried on walking they spoke to people further up than us.”
Mr Homer added: “You could see the tears in their eyes, they looked very sad, you could see they had been crying.
“They showed their appreciation for everybody, standing with their flowers, it was lovely to see. A very heartfelt moment.
“It’s been a nice day to see how much she’s loved.”