The Senate voted 33-29 late on Monday against holding the referendum prime minister Malcolm Turnbull’s conservative government had planned for 11 February.
Marriage equality opponents had supported the poll while most gay marriage advocates had opposed it, warning it would spark a divisive public debate.
Parliament should decide the issue during the current three-year term, said Ivan Hinton-Teoh, a spokesman for gay rights group just.equal.
He said: “Clearly in the last 24 hours we’ve had a very significant and historic step in the path to achieving marriage equality and that’s to get the campaign of marriage equality back on track to where it should’ve been.”
Mr Hinton-Teoh added he and other gay rights advocates would work with the minor Greens party to draft a marriage equality bill which would attract cross-party support.
“It’s just a matter of time. We know that we’re going to win this,” said Greens Senator Janet Rice, who was married to transgender Nobel Prize-winning climatologist Penny Whetton.
Christopher Pyne, a senior government minister who supports gay marriage, said the government would not make a quick decision on how to proceed now that the referendum, estimated to cost 170 million Australian dollars (£105 million), had been scrapped.
Mr Pyne accused opposition Labour Party leader Bill Shorten, who supports gay marriage, of refusing to support the public vote for political advantage.
“He couldn’t care less about same-sex marriage and he doesn’t care less about the many couples around Australia who’d like to have the same legal status as my wife and I enjoy,” he said.
“The sensible thing to do is let the dust settle on this issue and get on with the rest of our agenda,” Mr Pyne added.
Shelley Argent, spokeswoman for PFLAG Australia which represents parents of gay children, said the referendum had been designed to fail.
“The plebiscite if it had gone ahead would have been devastating. There was nothing positive about it,” Ms Argent said.
Opinion polls show most Australians support gay marriage, but public votes to change the constitution rarely change the status in Australia.