The 31-year-old died after being restrained by officers responding to a call in Kirkcaldy, Fife, on 3 May 2015.
Almost three years to the day of the death, family solicitor Aamer Anwar criticised Police Scotland, the Crown Office and the Police Investigations and Review Commissioner (Pirc).
He said the family had been left without answers.
Mr Bayoh’s family, including partner Collette Bell and his sister Kadi Johnson, are now suing Police Scotland for £1.85m, claiming the death could have been avoided.
It is said to be the first case of its kind in Scotland and will reach the Court of Session by 18 May.
Ms Bell said: “We will fight to make sure Shek gets the justice he deserves.
“I ask you, how long would you fight if your loved one was unlawfully killed? What would you want in place to ensure your loved one had justice?
“You would continue to fight and campaign.”
Mr Bayoh’s sister said: “We have kept an open mind and kept faith in the justice system, but waiting for three years with no answers has been soul destroying.
“We want an end to this, we want to grieve. Our patience has been exhausted and our trust failed.”
Mr Anwar listed six allegations in the civil action, including that the manner of restraint used by the officers was “not reasonable, proportionate or necessary and resulted in Sheku suffering positional asphyxiation”.
He said the civil action would be paused if any charges are brought over the death.
Mr Anwar said: “Sheku’s family have always said if he broke the law then arrest him, but any use of force had to be lawful, proportionate and necessary in the circumstances, but he did not deserve to die.
“Last Friday a summons was served on the present chief constable for the actions of officers three years ago under the leadership of then chief constable Stephen House. The action for damages in the Court of Session is for £1.85m in the name of his family.”
Mr Anwar also called for the resignation of Pirc commissioner Kate Frame, saying the body had failed “to adhere to its values of integrity, impartiality and respect”.
The solicitor added: “I wish to state on behalf of Sheku’s family that the investigation into the death of Sheku Bayoh was a national disgrace, to date no officer was suspended without prejudice and despite a final report delivered by Pirc to the Lord Advocate in 2016 he is yet to take action.
“The passage of time means that memories fade and evidence disappears or deteriorates. Sheku’s loved ones never wanted to go to court, but they will not give up.”
Last week Police Scotland confirmed legal papers had been received from the family.
The Crown Office said it had been a “complex and challenging investigation”. No timescale has been put in place for a ruling on potential charges.
A spokesman said: “We recognise that this has been a difficult time for the family of Sheku Bayoh.
“The family, and their legal representative, have been kept informed of any significant developments and senior Crown Counsel along with Crown Office officials met recently with the family and will continue to do so where appropriate.
“This has been a complex and challenging investigation and while there is still some further work to do, considerable progress has been made in the last few months.
“It is not appropriate at this stage to commit to any timescales given the nature of the investigation, however it’s anticipated a full report will be submitted for the consideration of Crown Counsel, the most senior lawyers in the Crown Office, in due course.”
Pirc said it had carried out an independent investigation under the direction of the Lord Advocate, who has responsibility for the investigation of deaths in Scotland.
A spokeswoman added: “The Pirc conducted an extensive and detailed investigation into Mr Bayoh’s death.
“More than 500 statements were obtained by Pirc investigators who also gathered evidence from a wide range of experts, including those suggested by the Bayoh family and their legal representatives, to give a greater understanding of events leading to Mr Bayoh’s death.”