The 33-year-old Edinburgh flanker won 76 caps for his country during a 12-year Test career that included the recent World Cup in Japan.
"All good things must come to an end," Barclay said on his Instagram account.
"This isn't an easy decision to make; playing for Scotland is the culmination of years of hard work and dedication and undoubtedly one the best things I will achieve in my life. But for me and my family it feels like the right time.
"It has truly been a privilege to wear the jersey. Not many people can say they got to live their boyhood dream, but I was lucky enough to have that honour and it is something I am unbelievably proud of.
"Not the end of my rugby journey, but the end of one chapter. Thanks for the memories."