The taxpayer-backed lender said the provision takes its total to £6.7 billion ($8.3 billion), but it remains in talks with the US Department of Justice over a settlement.
RBS said it was still uncertain when a deal will be reached and added that further “substantial” additional provisions may still be needed.
The mammoth provisions, which will be included in the bank’s results for the fourth quarter of 2016, will push RBS deeper into the red when it posts full-year figures next month.
It is likely to see the group - which is 72 per cent owned by taxpayers - post one of its largest losses since its government bailout at the height of the financial crisis.
RBS chief executive Ross McEwan said: “Putting our legacy litigation issues behind us ... remains a key part of our strategy.
“It is our priority to seek the best outcome for our shareholders, customers and employees.”
The provisions come after German investment banking giant Deutsche Bank and Swiss group Credit Suisse agreed to settle similar claims late last year for their role in the mis-selling of toxic mortgage-backed securities, paying a combined £9.9 billion ($12.5 billion).
But Barclays refused to agree a settlement and the Department of Justice is now taking legal action against the British bank.
Mr McEwan said the bank had suffered from “misplaced ambition” in the past.
“It’s clear to me that RBS became detached from the consumer-focused values that has to be at the heart of any bank,” he said.
He added that the provisions are “another painful example of the cost of that legacy”.
“Once these issues are behind us, we can focus 100 per cent of our efforts on our core bank and serving our customers better,” he said.
The settlement with US authorities is one of the key hurdles RBS must clear before the Government can look to sell it back fully into private hands.
Its share price has been weighed on heavily over the past year amid fears over the size of the fine.
But bosses at the group cast doubt over the timing and size of the ultimate settlement, confirming only that they continue to co-operate with the Department of Justice, but are not currently in active talks.
The bank said: “The duration and outcome of these investigations and other RMBS (mortgage-backed security) litigation matters remain uncertain, including in respect of whether settlements for all or any of such matters may be reached.”
RBS already sunk into the red by £2.5 billion for the first nine months of 2016 and the provision will see full-year losses widen substantially.
The group has posted losses of more than £50 billion over the past eight years.