They want work started on more double-track sections of the largely single track route between Edinburgh and Tweedbank so trains can more easily pass each other.
The supporters of the line, who lobbied for years for its re-opening, have also urged planned electrification of the line to get underway.
Borders Rail Monitor said: "While much media attention has focused on aspirations to extend the Borders Railway, a pre-requisite of any such plan must be to enhance the capability of the existing railway.
"This should be a key objective, irrespective of whether the line is extended or not.”
The group said ScotRail’s trains had been cancelled almost every week since 2015 and in some cases several days a week.
Founder Bill Jamieson, who played a key part in the 20-year campaign to restore the line, said: “The inadequate infrastructure specification ordered by Transport Scotland, with double track cut back to just 9½ miles out of 30½ miles of new railway, has dealt ScotRail and Network Rail a very poor hand.
"The Borders Railway train service can only perform adequately with the most intense management scrutiny and/or high levels of performance by east coast main line trains [such as LNER and CrossCountry] with which Borders trains share tracks between Waverley station and Portobello junction [in Edinburgh].
"Lead times are long for rail upgrading, and it is now over five years since the railway opened – sufficient time to have reached robust conclusions about the strengths and weaknesses of the Borders Railway.
"Planning the remedial actions needed – both including, and in addition to, electrification – must start now.
"This has even more urgency as a result of the response to coronavirus, with the rail industry facing the challenging task of rebuilding passenger patronage as a key element of the Scottish Government’s decarbonisation plan.
Mr Jamieson said double tracks should be extended northwards through Gorebridge and improvements made between Waverley station and an upgraded Portobello junction.
He said double track should continue through Brunstane “and ideally create a continuous double-track Borders Railway through Newcraighall and Shawfair”.
A ScotRail spokesperson said: “We don’t accept these figures and believe they have been incorrectly calculated.”
However, the operator declined to respond to campaigners who challenged its claim.
But it said: “Since the Borders Railway, it has reconnected the Scottish Borders and Midlothian with Edinburgh after a break of almost 50 years.
"ScotRail takes great pride in our commitment to the area and the work we are doing with communities and businesses throughout the south of Scotland.”
Transport Scotland, which was responsible for the £353 million project to reopen line, said it disputed the statistics on which the calls for improvements were made.
A spokesperson said: “We do not recognise these figures as they appear to have been taken from unofficial sources which may have incorrectly calculated the public performance measure (PPM) [the official yardstick].
"ScotRail’s official PPM statistics actually show that Borders performance is slightly higher than the rest of a franchise as a whole.”