Armed Forces Minister James Heappey said Andrew had “caused enormous challenges for the royal family”, but avoided saying whether it was right for the Queen to strip her son of his military roles.
Mr Heappey said that as a minister, he did not want to comment further as he “might risk being too colourful”.
But he told LBC on Wednesday that Andrew had “caused enormous challenges for the royal family in a year when we should be celebrating the extraordinary service of Her Majesty the Queen as she reaches her platinum jubilee”.
It comes as Andrew awaits a civil sex case in the US, with the trial scheduled to take place between September and December.
Virginia Giuffre is suing the duke for allegedly sexually assaulting her when she was a teenager and claims she was trafficked by Andrew’s friend, the late convicted sex offender Jeffrey Epstein, to have sex with the duke when she was 17 and a minor under US law.
The duke has strenuously denied the allegations.
Ms Giuffre claims Andrew had sex with her against her will at Ghislaine Maxwell’s London home and at Epstein’s mansion on the Upper East Side of Manhattan.
Maxwell, Epstein’s former girlfriend and a friend of Andrew, was convicted in the US on December 29 of procuring teenage girls for Epstein to abuse and will be sentenced this summer.
The duke is also alleged to have abused Ms Giuffre on another occasion during a visit to Epstein’s private island, Little St James, and on a separate occasion at Epstein’s Manhattan mansion.
The Queen has stripped Andrew of his remaining patronages and honorary military roles as the monarchy distanced itself from the duke ahead of potentially damaging developments in his lawsuit.
Security sources have said the abrupt changes to the Duke of York’s status following the removal of his military titles as well as him relinquishing the use of his “HRH” styling will trigger a review of his requirements by the secretive government committee in charge of deciding royal protection deployment.