The US Department of Justice (DoJ) said RBS would pay a $US35 million penalty (£26.7 million), and compensation of around $US9 million (£6.8 million) to more than 30 victims after entering into a non-prosecution agreement.
The probe targeted a fraud scheme stretching from 2008 to 2013, which aimed to boost profits by defrauding customers over trades in residential mortgage-backed securities (RMBS) and collateralised loan obligations (CLOs).
US Attorney Deirdre Daly said: “For years, RBS fostered a culture of securities fraud.
“Those in a position of authority taught and encouraged fraudulent trading practices. Worse, those supervisors and compliance personnel then took steps to prevent victims and honest RBS employees from discovering and exposing the scheme.
“After our joint investigation into fixed income trading began, RBS saw the error of its ways.
“RBS was able to avoid criminal charges in this case only because of its voluntary self-reporting and extraordinary co-operative efforts.
“By entering into this agreement, RBS has admitted the seriousness of its past criminal conduct and made a clean break.”
Among the victims of the fraud scheme were Barclays, Citigroup, Elliott Associates, Morgan Stanley and Goldman Sachs.
While the non-prosecution agreement focuses on mortgage securities traded in the secondary market, the bank is still awaiting a potential settlement with US DoJ over claims it mis-sold mortgage bonds in the primary market leading up to the 2008 banking crisis.
A spokesman for RBS said: “Having identified misconduct and self-reported the matter to the authorities, RBS has extensively co-operated with this investigation.
“Two former managing directors have pleaded guilty and RBS has zero tolerance for market misconduct.
“We will pay restitution to all impacted customers. We are pleased to be able to resolve this issue as we continue to build a simpler, stronger bank that is fully focused on serving our customers well.”
The announcement comes after the Financial Conduct Authority (FCA) said on Monday it may take action against RBS despite dismissing “serious allegations” about how the lender treated small business customers inside the Global Restructuring Group (GRG).
Shares in RBS closed up 0.3 per cent on the London Stock Exchange.