Giving evidence to MSPs on Holyrood's justice committee, Adrian Hanstock, deputy chief constable of British Transport Police (BTP), said his force would be reluctant to provide services in Scotland free of charge should Police Scotland take over rail policing north of the border.
The committee also heard warnings that railway policing was likely to be afforded a lower priority under the new arrangement, with the service "diluted" to give priority to other crime types.
The devolution of BTP function was a recommendation of the Smith Commission.
However, the Scottish Government’s plans to integrate the service with Police Scotland have proved controversial with critics who claim it is another example of imposing centralised control over policing.
Asked about how railway policing would operate across the border under the new model, Mr Hanstock said: "If (BTP) do not have responsibility for Scotland, why would we provide that? It would have to be a commissioned service that Police Scotland would have to pay for."
The committee also heard from Nigel Goodband, chairman of the BTP Federation, who said there would be an inevitable "dilution" of the service offered on the railways if policing was subsumed within Police Scotland.
He said: "When you are just another cog in a bigger wheel, the service you deliver...I think that would be diluted. The priorities of Police Scotland are totally different to what the train companies and passengers want."
Police Scotland said the integration of railway policing into the national force would be "massively complicated", but said the challenges were not "insurmountable".
Assistant Chief Constable Bernie Higgins said the current 225 BTP officers would be "ring fenced", but he added that in "times of crisis" those officers may have to be diverted away from the railway to deal with other incidents.