The Scottish Government announced on Sunday a pause to placing trans prisoners who have a history of abusing women - including sexual violence - in female jails. The announcement came after double rapist Isla Bryson, 31, was housed in segregation in Cornton Vale prison near Stirling following conviction before being moved to the male estate.
MSPs also voiced their outrage after reports emerged on Sunday that Tiffany Scott - who was convicted of stalking a 13-year-old girl before her transition - was due to be moved to the female estate. Speaking to BBC Radio Scotland on Monday, Mr Brown said a decision on Scott's placement "wasn't taken" and has not been given approval from the risk management authority, adding: "There is so much in the public domain which is incorrect."
He also said he understands that none of the five trans women currently in the female estate has a history of violence against women.
"This changes, as you can imagine, on a regular basis, but my understanding is there are no transgender women in the female estate that... have violence against women convictions," he told Good Morning Scotland.
Mr Brown added that, in exceptional circumstances, a trans woman could be moved to the female estate, but that would require the approval of Scottish ministers before a decision is taken.
"It will always be the case, has always been the case, similar in relation to a gender recognition certificate, that these things are not determinative of where somebody is placed - you cannot insist on that right if you're a transgender woman or man, to be placed in the estate where you want to be placed," he said.
"It will only happen as a result of a multi-party assessment, a rigorous risk assessment that goes on and that is what determines where people are placed."
Asked why this cannot be made Scottish Government policy now, the Justice Secretary stressed the importance of the ongoing Scottish Prison Service (SPS) review into the placement of trans prisoners.
"I believe that we do (have to wait for the review), I think (the SPS) have got a fantastic track record in dealing with this," he said.
Mr Brown said the SPS is "taking their time to make sure they get this right", adding that the Scottish Government will look at creating a specific unit for trans prisoners if that is what the review recommends.
Tensions around the issue of gender recognition have been heightened in recent months by the passage of Scotland's Gender Recognition Reform (Scotland) Bill in December - legislation that would make it easier for trans people to obtain a gender recognition certificate.
Opponents of the Bill raised concerns about the impact on women and girls, while its supporters said it is a minor administrative change.
The UK Government moved to block the Bill from becoming law by using, for the first time, Section 35 of the Scotland Act.
Mr Brown said Scottish Secretary Alister Jack - who issued the order to block the legislation - has not said what changes should be made to the Bill that would allow its passage, describing the move as a "democratic outrage".