Workers such as train conductors and station staff have also been instructed not to work on days off (rest days) after rejecting a 5 per cent offer plus other benefits.
The move, announced on Friday, comes on top of a one-day strike called for Monday, October 10, which is expected to affect most routes outside the Glasgow area.
The likely impact of the overtime ban is not yet clear, such as whether a temporary reduced timetable would be required, like the one during the drivers’ dispute this summer that reduced services by one third.
However, it is understood Scottish Government-owned ScotRail is far less dependent on RMT members to work overtime to keep trains running than on drivers.
The RMT said the overtime ban would particularly affect services on Sundays as it was not part of the working week.
Scottish organiser Mick Hogg said: “Where ScotRail has not filled vacancies, staff will not be working overtime.”
ScotRail said it expected “some level of disruption”, but was still assessing the likely impact and said it would provide an update next week.
RMT general secretary Mick Lynch told members: “Following a successful meeting of your representatives this week, it was agreed that it would be advantageous to escalate the dispute after the 24-hour strike on October 10.
"You are instructed not work any overtime or work on a rest day from 00.01 on Friday, October 14 until further notice.
“I have advised management that this union remains available for meaningful talks in order to resolve this dispute.”
The RMT said it would discuss the dispute with transport minister Jenny Gilruth in Edinburgh on Tuesday.
Hogg said: “There has been some movement which will now be subject to more discussion.”
ScotRail said it was working on contingency plans for services during the strike and “will update customers as soon as the plans are finalised”.
A Scottish Government spokesperson said: "There can be no increase on the current pay offer.
“Any industrial action will undoubtedly have negative impacts on passengers and employees.
"It is also likely to mean continued timetable changes and more short-term cancellations disadvantaging everyone.”
The RMT has also called more Britain-wide strikes in a separate pay dispute that includes Network Rail signallers, which will halt most ScotRail services on Saturdays October 1 and 8.
In addition, a strike by the Aslef train drivers union will hit services run by several cross-Border operators, including LNER and Avanti West Coast, this Wednesday.
Network Rail said similar numbers of ScotRail trains would run on Saturday as in the RMT strikes in August, when only 379 of the normal 2,150 services operated – fewer than one in five.
Reduced frequency services will operate on 11 Central Belt routes between 7:30am and 6:30pm.
The lines operating will be: Edinburgh-Glasgow Queen Street via Falkirk High, Edinburgh-Glasgow Central via Shotts, Edinburgh-Helensburgh Central via Airdrie/Bathgate, Edinburgh-Cowdenbeath, Edinburgh-Tweedbank, Edinburgh-Larbert, Glasgow Queen Street-Larbert, Glasgow Queen Street-Falkirk Grahamston, Milngavie-Springburn, Glasgow Central-Hamilton/Larkhall and Glasgow Central-Lanark.
However, no trains will not operate on the Edinburgh-North Berwick, unlike during previous strikes, because of engineering work.