Questions have been raised over the head coach's Dark Blues future as his side's miserable 2019 ended in World Cup failure.
The Scots followed up a dismal Guinness Six Nations campaign - where they finished second bottom - by crashing out at the pool stage for just the second time.
Former Scotland scrum-half Andy Nicol claims his old team-mate is under pressure but Townsend has already declared his desire to fight on and turn his team around.
And he has got the support of Glasgow prop Fagerson.
The forward said: "I'm 100 per cent backing Gregor, definitely. "If you read everything (that's written about the team), you wouldn't play rugby in the morning.
"It's one of these things. If you read too many tweets it's not great, there's a lot of keyboard warriors out there.
"We've not been good enough. We know that. There's been some pretty harsh internal reviews. We'll take our learnings from it. If we learn from it, it will be worthwhile."
Scotland made the worst possible start to Pool A as they were thumped by Ireland in Yokohama but kept their hopes of reaching the last eight alive with back-to-back wins over Russia and Samoa.
'We've grown as a squad'
But Townsend's team had no answer to the relentless energy of Japan during the climax to their group on Sunday, with the hosts blowing them out of the competition with a 28-21 triumph.
"We're all devastated as you can imagine," said Fagerson, who gave Scotland some hope of a late comeback when he scored a second-half try.
"We showed glimpses of what we can do but if we'd done that for the full 80 minutes we could have won the game. This is international rugby - it's cut-throat and if you give these teams a sniff they will punish you.
"We need to be more clinical but we've definitely learned a lot of lessons this World Cup and grown as a squad.
"I'm devastated right now but I really believe there are bigger things to come."
Townsend started with his tried-and-trusted campaigners against the Irish but turned to a younger crop including the likes of Jamie Ritchie, Magnus Bradbury and Darcy Graham after seeing his veterans flop against Joe Schmidt's men.
'We weren't good enough on Sunday but we'll learn'
The experience could yet be the making of those World Cup rookies but Fagerson - himself just 23 - refused to use a lack of caps as an explanation for Scotland's exit.
"We were here to do a job but we didn't get it done," he said. "I hate this old age thing, if you're good enough to play, you play. It doesn't matter how old you are.
"That's not an excuse, we just weren't good enough on Sunday. There were positives, we'll find some somewhere.
"But it's going to be a hard review and we need to learn from this. If we learn from it then it won't have been a waste of time."