Royal Bank of Scotland, Bank of Scotland and Clydesdale all saw a net reduction in customers between April and June last year, according to separate data.
Figures from payments body Bacs showed just over a million switches took place in 2016, about 23,500 fewer than in 2015. Just 208,387 switches occurred during the fourth quarter of last year.
Consumer groups said that more needed to be done to encourage people to switch, saying that a particular bank does not need to be a “lifetime companion”.
An investigation by the UK competition body the Competitions and Markets Authority (CMA) has previously found that consumers could typically save £92 a year by moving to a deal which is better suited to them.
Tashema Jackson, money expert at uSwitch.com, said: “Awareness of the Current Account Switch Service may be high, but the numbers show that in reality each year fewer people are using the service.
“Clearly more needs to be done to open people’s eyes to the fact that banks need not be a lifetime companion.”
She added: “The recent CMA announcement on the changes Bacs will be making to the service, including the commitment to provide a decision on overdraft provision prior to switching, could give a much needed boost to improve this traditionally stagnant market.
“Greater transparency coupled with a more trust-worthy system will empower consumers to better manage their finances and make the most of where they keep their money.”
However, the organisation claimed that awareness levels of the Current Account Switch Service reach a record high of 78 per cent – representing an increase of 20 percentage points since its launch.
Under the Current Account Switch service, customers can complete a move in seven working days – a substantial improvement on the previous process that could take between 18 and 30 working days - while all payments going out and in will be moved from the old account to the new account.
Meanwhile, payments accidentally made to or requested from the old account are automatically redirected to the new account and customers will atomatically receive a refund of interest and charges on their old and new current accounts if anything goes wrong with the switch.
Royal Bank of Scotland saw nearly 9,300 customers switch to alternative services, figures from banks and buildings societies showed, while Bank of Scotland suffered a net loss of 1,571. Clydesdale saw customer numbers reduce by just over 6,000.
Santander, Nationwide and Halifax gained the most customers, the figures showed, followed by Tesco Bank and TSB.