The demonstration in Baltimore, Maryland, was peaceful for hours, but later pockets of protesters smashed police car windows and shopfronts.
The protests over the death the 25-year-old came a day after deputy police commissioner Kevin Davis said Mr Gray should have received medical attention at the spot where he was arrested, before he was put inside a police van handcuffed and without a seat belt – against force policy.
Mr Gray died on 19 April after suffering a fatal spinal injury while in custody and his death has intensified a national debate over police treatment of African-Americans.
Authorities have not explained how or when Mr Gray’s spine was injured. Video showed him being dragged into a police van and police have said he was in it for about 30 minutes before paramedics were called.
Mr Gray’s death has been compared to those of other unarmed black men who died at the hands of police in New York City and Ferguson, Missouri.
Baltimore residents voiced their anger at how the police and the city are handling the investigation into Mr Gray’s death.
Protesters threw cans and plastic bottles in the direction of police officers. One protester broke the window of a police car, grabbed a police hat inside and wore it while standing on top of the vehicle with several others.
At least two people were hurt in the mayhem and more than a dozen were arrested.
In her first public comments since Mr Gray’s death, his twin sister Fredricka Gray appealed for calm.
“My family wants to say, can you all please, please stop the violence?” she said at a news conference with the mayor. “Freddie Gray would not want this.”
Earlier the crowd paused for a moment of silence in front of Shock Trauma, the hospital where Mr Gray died.
Carrying slogans such as “Jail Killer Police!” and “Unite Here!”, demonstrators filled two city blocks on Saturday and marched to City Hall, where the crowd filled a grassy plaza.
Tanya Peacher, 36, said she had never attended a protest in the city before, but watching a video of Mr Gray’s arrest motivated her.
Ms Peacher said she “looked at my son”, and marched to stop other custody deaths.
A dozen marchers lay down in the street during an impromptu “die-in” at a major junction.
Both police commissioner Anthony Batts and Baltimore mayor Stephanie Rawlings-Blake, who took office in 2010, are African-American. They came in making promises to both the inner-city residents and police who spent decades staring each other down in neighbourhoods ravaged by crack and heroin.
But with each death of a black man in custody, their efforts to overcome mistrust have hit walls of scepticism and outrage.
Mr Batts said he had sacked 50 police employees and reduced officer-involved shootings. The “use-of-force” reports police must file dropped from 598 in 2012 to 435 in 2014.
But he acknowledged that some cases had “tarnished this badge and the reputation of the department”. Mr Gray is at least the fifth black man to die after police encounters since Mr Batts took charge.
A Baltimore Sun investigation revealed last year that the city has paid about $5.7 million (£3.8m) in brutality settlements since 2011, involving 102 instances of excessive force.
Mr Batts then asked the US justice department to review the department’s policies and procedures. Now the department has opened a second probe, examining Mr Gray’s death.
Baltimore had one of America’s busiest markets for heroin and crack cocaine when Martin O’Malley ran for mayor in 1999. The future Maryland governor and Democratic presidential candidate imposed a zero-tolerance policy that reduced crime, but also resulted in thousands of arrests without due cause.