The Health Secretary said he will not apologise for spending time with his family as they have seen very little of him over the last seven months.
Mr Yousaf’s trip to the studios near London was reported in the Scottish Sun on Wednesday.
Scottish Conservative chief whip Stephen Kerr said the Health Secretary should have reconsidered his holiday.
Mr Yousaf posted a series of tweets on Wednesday morning.
He said: “Don’t usually comment on stories about me, but I’ll make an exception.
“Most important job I have is being a good father, step-father & husband to my wife and kids.
“In the last seven months they’ve had virtually no time from me.
“Pandemic has been tough for everyone and I am luckier than most but it has been tough for my 12yr old step-daughter who missed her friends during her P7 year.
“As a treat, I promised this Harry Potter fan a trip to HP Studios months ago (if restrictions allowed) when school finished.
“Every day I have taken ‘off’ I have worked. Speaking to clinicians, vaccines and testing teams, clearing submissions etc.
“My family don’t see me much at all, and while they understand why, it is difficult for us all, so I won’t apologise for giving them the v limited time I do.”
Mr Yousaf also claimed he had “bumped into” Scottish Conservative public health spokesman Sandesh Gulhane while at the Warner Bros studio near London.