Dumfries and Galloway councillor Jim Dempster has been suspended from the party pending an investigation after making the remark about Mr Yousaf.
Mr Dempster reportedly said that if Mr Yousaf had visited the region, “he may have been at Springholm but no-one would have seen him under his burka”.
He made the comments in a meeting with Transport Scotland officials.
Mr Yousaf said he felt “insulted and deeply shocked” at the remark and is now calling for Mr Dempster to resign.
He told BBC Good Morning Scotland: “For this kind of slur which is so brazen, not something which simply slips off the tongue, to make that kind of Islamophobic slur from a senior elected councillor in front of my transport officials, so knowing that it would easily come back to me, and in front of members of the public I should say as well, just goes to show how far we’ve got to go and just how emboldened some feel about making Islamophobic remarks.
“Councillor Dempster, who I understand now admits making these remarks, has no option but to resign.”
Mr Yousaf said he had received an apology from Mr Dempster but does not accept it.
He said: “If it is completely inexcusable then the only logical conclusion for him is to of course resign, there’s no excuse for it.”
He added: “You don’t need diversity training to know not to say to a Muslim you are hiding under a burka.”
A Scottish Labour spokesman said: “James Dempster has been suspended pending an investigation by the Scottish Labour Party. The party has a zero tolerance policy towards racism.”
Mr Yousaf said that Labour leader Richard Leonard has contacted him to say that Mr Dempster has been suspended.
He said: “The only logical conclusion for Richard Leonard would be to expel Councillor Dempster and Councillor Dempster to do the honourable thing and just resign.”
Labour MSP Anas Sarwar has tweeted his support to Mr Yousaf, saying: “Solidarity with @HumzaYousaf. This comment was crass, stupid, offensive & unacceptable.
“Right to call this out & challenge it, no matter who or where it is from. Everyday racism and Islamophobia is real. Our society & institutions are not immune to it. Lots of work to do.”
It is the latest racism incident to hit the Scottish Labour Party.
Last month Labour MP Hugh Gaffney apologised for using “deeply offensive” language during a Burns Supper speech.
Mr Gaffney, who represents Coatbridge, Chryston and Bellshill, has been reprimanded by the party and is to attend equality and diversity training after making comments about the Chinese and LGBT communities.
And Davie McLachlan, former leader of the Labour group on South Lanarkshire Council, was suspended by the party in January pending an investigation after allegedly making a racist remark about Mr Sarwar, which he denies.
Mr Sarwar drew up an eight-point plan on how the party can tackle everyday racism and Islamophobia after speaking out on the abuse he received during the Scottish leadership campaign.