The 96-year-old was attending St Magdalene Church on New Year’s Eve accompanied by the Princess Royal, when it is reported he made the comment as he joked with onlookers.
Witnesses who claimed to have heard the comment said the man had a ginger beard, The Sun and the Mail Online reported.
The Duke traditionally marks the festive period with the Queen at her Sandringham estate in Norfolk, attending a church service on Christmas morning and New Year’s Eve with the royal family.
Philip, who retired in August, is notorious for making controversial comments.
The royal once famously asked a group of Aboriginal Australians during a visit in 2002: “Do you still throw spears at each other?”
He also once asked a disabled 60-year-old man on a mobility scooter: “How many people have you knocked over this morning on that thing?”
Buckingham Palace declined to comment.