Brown, who joined Warriors in the summer as breakdown and contact area coach, will quit at the end of the month.
His family have struggled to settle in Glasgow during the pandemic and will move back to the London where he stayed while playing for Saracens.
“Relocating during the pandemic has been a real challenge,” said Brown, a former Scotland captain. “We moved north in the summer and since arriving my family have struggled to settle in Glasgow.
“An opportunity has come up down south and for the wellbeing of my family we have taken the decision to return.
“My family’s happiness is my number one priority and we have taken the difficult decision to return to London to be closer to where my children grew up and where their friends are.
“Glasgow Warriors will always have a special place in my heart, and I’d like to thank Glasgow and Scottish Rugby for the opportunity and understanding they have shown. It’s a great club to be have been involved with and I believe they have a strong future ahead.
“I’d like to wish Danny, and everyone at the Warriors, all the best for the rest of this season, and beyond.”
Glasgow head coach Danny Wilson said: “We are understanding of Kelly’s situation and family has to come first. This pandemic has been challenging for everyone. It’s a shame to see Kelly go, especially after such a short period of time with us, however, we are completely understanding of the position he is in and wish him well.”