The Loose Women host, who was Britain’s first transgender national television newsreader, said it was “just horrible” to be painted as “dangerous to society and unwanted” by non-trans commentators.
It comes amid ongoing debate in Scotland over laws allowing transgender people to self-identify and obtain a certificate confirming their preferred gender.
Speaking on BBC’s Question Time on the subject of whether safe-guarding measures on women-only spaces should be reviewed, Willoughby said: “This has been a debate now that has raged for a minimum of three or four years.
“It’s been incredibly toxic and it’s been awful as a trans person… to be sat at home and see people who aren’t actually trans discuss your life and paint you as a whole en masse as dangerous to society and unwanted is just horrible.”
Another debate currently raging in Scotland concerns where trans women should be imprisoned.
The placement of transgender woman Isla Bryson, who raped two women while identifying as a man and was sent to Cornton Vale women’s prison near Stirling, has caused public uproar.
Speaking on Question Time Willoughby added that stories of transgender prisoners being placed in spaces with other women was “not true”.
She added: “There are bad apples in all walks of life. There are bad protestants, there are bad Catholics, bad lesbians, or bad gays, and yes, there are bad trans people.
“And all of those people if they do something criminal and are found guilty they deserve to go to jail and to be punished in a jail that is suitable for them and where everybody else is safe.
“What you’ve been subjected to, I’m afraid, over the last couple of weeks which is now par for the course is a lot of misinformation by a media that is completely transphobic.
“There is categorically no way somebody who has committed rape would be moved into the women’s estate.”
As well as her regular appearance on Loose Women, Willoughby is known for appearing in the 2018 series of Celebrity Big Brother.