Stephen House, the Chief Constable of Strathclyde Police, said his force alone would send 5,000 more people to jail a year if everyone found to be carrying a knife was sent to prison.
Insisting he was not "liberal" on knife possession, he nevertheless said the policy was "on the wrong tack completely", criminalising people who did not deserve to be put in jail.
House said even somebody carrying a knife for a fishing expedition could potentially fall foul of the policy, which is being vigorously backed by Labour and the Conservatives.
His comments come as the political parties begin to flood into the Glasgow North-East seat ahead of the by-election now set for 12 November.
Labour candidate Willie Bain, who launches his campaign tomorrow, has made the pledge on automatic jail sentences a key part of his bid for the seat, having said that people caught with a knife should get four years in jail.
In an interview with Scotland on Sunday, House also called on SNP ministers to guarantee funding for police forces ahead of the coming public sector squeeze.
But it will be his comments on knife crime that will attract political attention. He said: "I cannot sign up to (the idea that] everybody who carries a knife should be locked up.
"We would provide probably 5,000 extra prisoners a year to the prison service because there are a lot of people who carry knives. If you are going to lock everybody up, then there's not enough prison spaces."
He used the example of a teenager caught with a knife for the first time because of peer pressure from gang members. "I don't think that person should go to prison," he said. He argued it was better to talk to teenagers and their parents and only consider a prison sentence if they are caught again.
Commenting on House's views, David Kerr, the SNP candidate in the by-election said: "Willie Bain's keystone policy in this by-election has crumbled. Not only is it unworkable, it is also totally inconsistent with the position of the UK Labour government, who are not proposing mandatory sentencing for carrying a blade."
But a Labour Party spokesman last night questioned House's figures, pointing to official statistics showing that, across the whole of Scotland, there were only 3,148 people convicted of weapons offences.
Scottish Labour's community justice spokesperson Paul Martin said: "The views of the police are always valuable, but it is the role of elected politicians to legislate. A mandatory prison sentence would deter people."