Dumfries and Galloway councillor Jim Dempster has been suspended from the party pending an investigation into his remark about Humza Yousaf.
Mr Dempster reportedly said that if Mr Yousaf had visited the region, “he may have been at Springholm but no-one would have seen him under his burka”.
Speaking on the BBC’s Sunday Politics Scotland, Labour MSP Mr Sarwar said: “Dinosaurs make crass, stupid, offensive, unacceptable remarks.
“It’s right he apologises. It’s right that Labour has suspended him but I think the Labour Party has got to make a decision in the long-term based upon on how we want to reflect in terms of society, whether he’s truly remorseful or not.
“It’s one thing after an incident saying you are sorry, but it’s about changing yourself and changing your behaviour and how that reflects on wider society.”
He added: “I think the Labour Party’s got to make a decision, and got to take one very quickly.”
Mr Sarwar appeared on the programme with Mr Yousaf to discuss the racism and Islamophobia they have both experienced, including death threats.
The transport minister has called for Mr Dempster to resign and refused to accept an apology the councillor offered for his remark.
Speaking about his own experience of racism and Islamophobia, Mr Sarwar said: “People will send threats about burning down my offices, which obviously has an impact on my staff. Targeting me, targeting my family.
“Questioning my loyalty to Scotland, questioning my loyalty to the UK. Saying that I’m part of some undercover mission to impose Sharia law on Scotland or the UK. Questioning whether we belong.”
Mr Yousaf said he had faced similar abuse.
Mr Sarwar is heading a campaign to tackle everyday racism and has drawn up an eight-point plan to about how his party can deal with the issue.
Davie McLachlan, former leader of the Labour group on South Lanarkshire Council, was suspended by Scottish Labour in January pending an investigation after allegedly making a racist remark about Mr Sarwar, which he denies.
Last month, Labour MP Hugh Gaffney apologised for using “deeply offensive” language during a Burns Supper speech.