The former Labour leader made the comments in a new video as he announced he has joined the Jewish Labour Movement (JLM) as an affiliated member.
Mr Brown said the party had been too slow in dealing with cases of anti-Semitism and urged members to follow him in joining the JLM, which is due to vote on its membership of the party at its annual general meeting next Sunday.
In the video, filmed in Liverpool, Mr Brown paid tribute to the late Sir Nicholas Winton, who joined the Labour Party in 1936 and who organised the rescue of 669 Jewish children from Nazi-occupied Czechoslovakia on the eve of the Second World War before helping them to begin new lives in Britain.
Mr Brown said: "The Labour Party has always had a long, proud and noble tradition of standing up against all forms of prejudice and racism. But in the last few years it has let the Jewish community down.
"They should never have allowed legitimate criticism of the current Israeli government to act as a cover for the demonisation of the entire Jewish people."
The Labour Party has been mired in claims of "institutional anti-Semitism" and the Equalities and Human Rights Commission has launched an investigation into whether the party had “unlawfully discriminated” against people because of their religious beliefs.
Labour has insisted leader Jeremy Corbyn is a "militant opponent" of anti-semitism and "rooting it out of our party is an absolute priority".
Three former Labour members were arrested last month police investigating alleged anti-Semitism among Labour Party members. Scotland Yard said two men in their 50s and a woman in her 70s had been arrested during March on suspicion of publishing or distributing material likely to stir up racial hatred.
In his video, Gordon Brown added: "The message to the Jewish community should be clear and unequivocal: 'You will never walk alone and we will never walk on by on the other side. We must never tolerate intolerance.'
"Solidarity means standing up with those who are under attack and that is why I am joining as an affiliated member. I urge all my colleagues to do the same.
"Tackling anti-Semitism and racism and fighting for equality is not a diversion nor a distraction from our purpose as party - it is our purpose as a party. We will never allow evil to triumph over good and neither will the British people."