Scottish Liberal Democrat leader Willie Rennie said Phil Gormley should seek a leave of absence from the post while the Police Investigations and Review Commissioner (Pirc) looks into the allegation.
No information has been officially released regarding the nature of the complaint against him, but if a serious breach of standards is found, Mr Gormley could face dismissal.
Allegations surfaced last night that another senior Police Scotland officer has accused Mr Gormley of bullying.
The chief constable said: “I am co-operating fully with the Pirc and will provide all necessary assistance to bring this matter to a timely and satisfactory conclusion.
“In fairness to others who may be involved, it is not appropriate for me to comment further at this time.
“I would like to stress that I remain focused on leading Police Scotland, ensuring that we continue to serve and protect the people of this country.”
However, Mr Rennie said that in order for the investigation to be conducted effectively, Mr Gormley should temporarily step aside.
“These allegations of gross misconduct are incredibly serious and require a thorough and prompt investigation,” he said.
“For that investigation to be conducted effectively, it will be necessary for the chief constable to seek leave of absence from his post. Any leave of absence should not imply acceptance of guilt.
“Previous cases in Scotland and other parts of the UK have set a precedent, where the person who has been under investigation has temporarily stepped aside.”
Scottish Labour justice spokeswoman Claire Baker said: “All allegations must be fully investigated and I’d urge the Police Investigations and Review Commissioner to be as transparent as possible.
“With the most senior police officer under investigation, it is vital that whatever the outcome the public maintains confidence in Police Scotland.”
Green MSP John Finnie said: “Allegations of this nature can be damaging to public confidence in the police and it’s therefore vital that a thorough investigation is undertaken and the full findings are published.”
The investigation followed a referral to the Pirc by the Scottish Police Authority (SPA).
A spokesman for the Pirc said: “Following a referral by the SPA, the commissioner has assessed that the conduct which is the subject of the allegation would, if proved, amount to gross misconduct.
“Once the investigation is concluded, the commissioner must determine whether, in the investigator’s opinion, the senior officer has a case to answer in relation to the misconduct allegation.
“The commissioner must submit a report to the SPA containing a summary of the evidence and the investigator’s opinion on whether the allegation should be referred to a misconduct hearing.
“Where the authority determines that there is a case to answer for either misconduct or gross misconduct, it must refer the misconduct allegation to a misconduct hearing.
“As this is a live investigation, it would be inappropriate to comment further at this time.”
A Scottish Government spokeswoman said: “We note the Pirc investigation and that they will provide a report to the Scottish Police Authority.
“It would not be appropriate to comment on any current investigation.”