Lib Dem leader Willie Rennie has insisted the spending plans as currently drafted “fall substantially short” of what is needed for his party to endorse them.
With the SNP no longer having a majority at Holyrood, Mr Mackay is forced to seek support from opposition parties if he wants his Budget for 2017-18 to be passed.
Negotiations between the Finance Secretary and the other parties have been taking place ahead of a crucial vote on the plans in Holyrood.
For the Liberal Democrats to back the Scottish Government, Mr Rennie said extra cash would be needed to help school pupils who come from disadvantaged areas, warning that a “£70m shortfall” compared to spending south of the border must be addressed.
Improved funding for colleges must also be part of the Budget, Mr Rennie said, claiming the draft plans for 2017-18 leave the sector £93m poorer than 2010-11, when spending in this area peaked.
The Lib Dem leader said: “We believe that it is important to return investment back to former funding levels and meet future skills needs. Improved college funding will need to be part of the final Budget.”
He went on to claim the planned spending levels for mental health did not “get anywhere near meeting the need” of the services.
Mr Rennie insisted “at least £1.2 billion” would need to be earmarked in this area, calling for spending on Child and Adolescent Mental Health Services (CAMHS) to be doubled to bring funding in line with England.
He claimed funding for the police is “at least £20m short of what we think is important to promote a successful police service”.
To help Scotland’s islands, the Lib Dem also demanded the return of the Air Discount Scheme and for ferry fares to Orkney and Shetland to be cut by 50 per cent.
In a letter to the Finance Secretary, Mr Rennie said: “Our support for the Budget can only be considered if significant and substantial change is included for colleges, the pupil premium, mental health, the police and the island economies.”
Mr Rennie said his party’s suggestions would bring about a “step change in mental health and a transformation in education that will help in the road to a liberal Scotland”.
He added: “I have had a number of meetings and discussions with the Finance Secretary so far and am looking forward to receiving his response to our plan.”