The Scots kicked off their Autumn Nations Series campaign with a 60-14 victory over Tonga but stiffer tests await.
The Wallabies will arrive in Edinburgh in a rich vein of form having beaten South Africa twice in the Rugby Championship on consecutive weekends last month.
Although pleased to welcome fans back to Murrayfield with a ten-try win, Townsend felt there were times Scotland were caught out by the Tongans who scored a second-half try through prop David Lolohea.
“Defensively we’ll have to be at our very best,” the Scotland coach said of next Sunday’s game. “Australia have shown they are a good attacking side and they are full of confidence. They’ve won five Test matches in a row and they’ve beaten the world champions twice, the No 1 team in the world.
“We set very high standards in our defence and at times today it wasn’t at those standards.
“We also have to see how quickly we can integrate the players who are going to come back into the team during the week.
“We’ve got an extra day to prepare which should help us but we know it’s going to be a massive challenge on Sunday.”
With the match taking place outside the recognised international window, Scotland faced Tonga with a side made up entirely of players from Glasgow and Edinburgh and gave eight players their debuts.
The likes of Stuart Hogg, Finn Russell, Duhan van der Merwe, Chris Harris, Adam Hastings, Rory Sutherland, Huw Jones and Sam Skinner will return to the squad for the Wallabies match, but Townsend was happy with how the new boys fared against Tonga.
“I’m very pleased that players making their debut for their country did so in a winning performance and contributed to that performance,” said the coach.
“I felt we energised the crowd by the way we played. We aimed to play at speed and we did that in the first half. I thought Tonga played really well. We didn’t know what to expect from them but I felt when they had the ball they caused us problems so we’ll have a few things to work on this week.”