The force said the raid on two properties on Friday morning was part of an intelligence-led human trafficking operation in the town.
It said those arrested - three men, aged 37, 39 and 53, and a 53-year-old woman - were due to appear in court on Monday.
Officers said the action, codenamed Operation Barramundi, was launched after information was received from the UK Government's national referral mechanism, which identifies and refers potential victims of modern slavery.
They said a team of officers from Tayside, Police Scotland’s specialist crime division, operational support unit and the national human trafficking unit had been working with Perth and Kinross Council, Migrant Help and the Gangmasters Labour Abuse Authority.
Detective Inspector Marc Lorente, said: “I would like to thank the local communities where this activity was carried out for their patience, as well as our partners for their ongoing support.
“I would urge anyone with information about potential human trafficking offences in their area to contact the police so further enquiries can be carried out.
"Police, other enforcement agencies and partners cannot tackle the wider issue of human trafficking alone - we also need the public to work with us if we are to identify and help vulnerable individuals being exploited.
“If you suspect exploitation is happening in your community, please report it.”
Police Scotland has launched a national campaign to raise awareness about modern slavery, asking people to be aware and vigilant of the signs of exploitation and explaining what people should do if they have concerns.