The major security blunder has affected at least six people who were waiting for new or returned passports.
The error came to light when one woman took to social media in an attempt to track her passport.
An investigation has been launched into the mix-up, which has sparked concerns over the security of personal data.
Charmaine Firth, a mortgage advisor from Dalry in Dumfries and Galloway, had been due to receive her passport last Friday.
However, when she opened the envelope the passport inside belonged to a man from nearby Twynholm.
Ms Firth could not get in touch with HM Passport Office to report the problem, so she used Facebook to trace the Twynholm man.
He told her he had been sent the passport of a woman from Newton Stewart.
A Home Office spokesman said the security of personal documents was of “paramount importance” and it was urgently investigating the issue.
Ms Firth has yet to trace her documents.
She told BBC Scotland she was concerned her details were at risk of being stolen.
“Someone out there has received my passport,” she said.
“It’s a total breach of the data protection act. Our passport details can be used fraudulently, so that’s obviously a really huge concern for me.
“I can’t understand how there has been such a big mix-up.”
Kippford resident Glanville Jones, 94, was meanwhile sent the passport of Evie Whyte, 18, who lives in Port William.
He said the issue had caused a lot of confusion.
“A girl phoned me yesterday, she hadn’t got her passport, but somebody had told her somebody had got mine,” he said.
“And she wondered whether I’d got hers, but I hadn’t, it was a different girl’s that I’d got.
“I am bothered about it. I don’t think it’s right really. I thought about phoning the MP actually.”
The official advice to people who have found themselves in this situation is to contact HM Passport Office to arrange for the passport to be returned.
A total of 472 passports were lost during delivery in 2016 – up from 312 in 2012, according to the UK government.