The dead men are all believed to be Spanish nationals, originally from Gambia, who emergency services said were working when a 15ft (4.5m) concrete bay wall collapsed on them.
Blocks weighing about one-and-a-half tons each came down, along with tons of scrap metal being stored in an outside yard at the vast recycling site in a heavily industrialised part of Birmingham.
All five workers were pronounced dead at the scene following the incident at Hawkeswood Metal Recycling in the Nechells area of the city at about 8:40am yesterday.
A sixth man is currently in hospital after suffering a leg injury, although it is not thought to be life-threatening.
In a briefing at the site, which is bordered by a railway line, Detective Superintendent Mark Payne, of West Midlands Police, said: “We believe the victims are foreign nationals. We believe the men are from Africa.”
Friends and relatives of the men, who were at the scene awaiting more information, said the victims were all Spanish, but of Gambian heritage.
Manka Sawo, who was among the group, said they were alerted to the accident by the injured man who rang them at about 9am to tell them what had happened.
During a press conference at the site, Mr Payne said a joint Health and Safety Executive (HSE) and police investigation was now being carried out to establish the cause, warning that the process could take weeks.
He added that one line of of inquiry would be to find out whether “any issues of negligence or malpractice” may have contributed to the collapse.
Mr Payne said: “We’re simply trying to recover the bodies of the men and do it in a way which will help understand exactly how that wall came to fall down.
“Clearly we are investigating together with the HSE, whether there are any issues of negligence or malpractice that have contributed to that wall falling down.”
Emergency services said they have accounted for all staff who were on site and that there was no prospect of anyone having survived under the many tons of rubble.
Mr Payne said: “It’s a wall constructed of blocks that weigh about 1.5 tons each - concrete blocks.
“That wall was supporting a body of scrap metal.
“It appears that that wall has collapsed on top of the men and then the scrap metal behind the wall has fallen on top.
“You can imagine it’s a very difficult and complex scene and we’re working to try and understand exactly how that wall came down.
“We’re trying to do that in a way that’s safe for everybody involved, in a way that allows us to recover the deceased and understand exactly what happened.”
He added that the men had been working at the site which employs dozens of staff at the time they were killed.
Mr Payne said it had moved very quickly from being a rescue situation to a recovery operation, which was “likely to take the rest of the day”.
He added that the emergency services hoped to have completed recovery of the bodies overnight.
Speaking at the scene, Mr Sawo said: “It is devastating.
“Five of them have lost their lives.”
He added: “We are looking for information, we don’t know what has happened yet.
“Everybody here is of a different relation - family, friends, cousins, we are all very close.”
Mr Sawo said the injured man had suffered “a broken leg”, and had phoned relatives from hospital.
He added: “I knew one of the men very closely; yesterday we celebrated Eid together. It’s very, very sad.”
West Midlands Fire Service said it would work as quickly and as safely as possible to recover the bodes of the five men from the extremely challenging scene.