Goncalves buried the match-winner from 20 yards before he was ordered off after a tussle with Killie centre-back Kirk Broadfoot, who also saw red in what was afternoon to forget for the former Rangers man.
Hearts interim manager Jon Daly welcomed highly-rated young defender John Soutar back to the fold, making his first league appearance since rupturing his Achilies tendon in January.
Goncalves was also returning to the starting line-up after coming off the bench last weekend to hit a late consolation at Celtic Park, and he took just six minutes to make his mark.
After a cagey first few minutes, Cameroon international Arnaud Djoum found Goncalves through the middle. The towering Portuguese turned Broadfoot with incredible ease, before firing a fierce right-foot shot into the bottom corner from 20 yards.
The visitors picked Kilmarnock apart on the counter in the 20th minute but Goncalves, racing onto a perfectly weighted Kyle Lafferty through ball, was denied by a last-gasp tackle by Killie keeper Jamie MacDonald.
Frustrations grew among the home support - who saw their side lose against St Johnstone in the last minute last weekend - with Soutar and fellow Hearts centre-back Christophe Berra consistently untroubled Kilmarnock’s hopeful punts from the back.
This prompted Killie boss Lee McCulloch to mix things up at the break, sending on Jordan Jones for the ineffective Alex Burke.
The hosts showed a marked improvement after the restart, and should have levelled on the hour when Dominic Thomas was denied by an outstanding double-save from Hearts keeper Jack Hamilton.
Hearts almost doubled their lead immediately after, courtesy of more shambolic play at the back from Kilmarnock. This time, Thomas gifted the ball to Goncalves to break down the right, but Lafferty mishit the golden chance from six yards.
The match took a twist in the 67th minute when referee Stephen Finnie sent Broadfoot and Goncalves packing. It seemed fairly innocuous, with the pair clashing on the ground, and both left baffled to see their respective red cards.
Hearts looked most likely to find the net from here, with Lafferty firing wide from distance and MacDonald forced to react quickly to collect an impressive overhead kick from Don Cowie.
Neither side could find the net again, though, and despite some late pressure from the Rugby Park side - who host Celtic next - Hearts held out comfortably to hand Daly a precious three points.