The Secretary of State for business, energy and industrial strategy made the comments during an exchange in Parliament, in which he appears to misunderstand the political situation in Edinburgh, where Labour took control of the city council with the support of the Conservatives.
SNP business spokesperson Stephen Flynn said: "Isn't it great to see the Secretary of State in this House rather than standing in the street filming a statement to the public surrounded by boarded-up shops and rubbish. What an edifying spectacle for a man who believes in the pre-eminence of this here Parliament."
He asked how much the Government's energy package would cost, if large companies like Amazon would benefit from "a scheme that's going to built on the back of public-sector borrowing", and if he can explain to "energy-rich Scotland" why "Westminster has failed us so terribly badly".
Business secretary Mr Rees-Mogg said: "He refers to rubbish in the streets of Westminster and I would point out to him that as soon as an administration turns from Conservative to socialist the rubbish piles up in the streets, as I think it's also been doing with the SNP in Edinburgh."
Met with protestations from the SNP benches, Mr Rees-Mogg continued, saying "this scheme is fair to taxpayers", adding there would be a review in three months "to ensure the support goes to those people who need it most".
Mr Rees-Mogg insisted further energy bill support would be made available to businesses beyond April 1, with the current "broad brush" policy replaced by a targeted approach.
SNP MP Alison Thewliss said local firm Calder Millerfield has been quoted a 345 per cent increase in their electricity costs, asking what he could offer them.
Mr Rees-Mogg, in his reply, said: "We have done this on a completely broad brush basis because at the moment that's the right thing to do, it's needed urgently.
"However, we do need to examine exactly who ought to benefit in a review and then have that announced in plenty of time for April 1 to see what level of support is needed for the longer term."