A survey of more than 1,300 people carried out for the housing association Places for People Scotland found 40% believed this to be case.
The same research also found almost a third of people believe the quality of social housing is poor.
The overwhelming majority (96%) of people said they would not be put off from being friends with someone if they lived in either a council or housing association property.
While almost nine out ten people (89%) said they would always prefer to own their home rather than rent, 85% agreed the cost-of-living crisis means Scotland needs more properties for social rent.
Tom Norris, managing director of Places for People Scotland, is to speak out about the impact stigma has on communities and tenants when he addresses the Scottish Federation of Housing Association’s annual conference on Wednesday.
Speaking ahead of that, he insisted action was needed from social housing providers, developers and both national and local governments in order to “change the narrative around social housing”.
He stated: “We must collaborate and work in partnership to end the stigma of social and affordable housing not just through words, but through action.”
Mr Norris said: “Scotland is a welcoming and progressive nation, but, despite our best efforts, the stigma of social housing can sometimes rear its ugly head.
“It diminishes the good work we do as a sector, undermines social cohesion and promotes social isolation.”
He continued: “Our research backs up evidence from across the UK which shows how negative attitudes towards social housing and people living in it, can scar the lives of people and communities across Scotland, hampering our collective efforts to help communities to thrive.
“Stigma is preventing Scotland from becoming a fairer and more equal society. It holds back people and it holds back communities.
“Wwe need to build more of the homes Scotland needs, and at pace – so that people from all walks of life, and of all ages, can benefit from an affordable and stable home now and long into the future.”