The magnitude 6.8 quake was the largest among dozens that have rattled the island’s south-eastern coast since Saturday evening, when a magnitude 6.4 quake struck the same area.
Most of the damage appeared to be north of the epicentre, which Taiwan’s Central Weather Bureau said was in the town of Chishang at the relatively shallow depth of seven kilometres (four miles).
The three-storey building, which had a 7-11 convenience store on the ground floor and residences on the upper ones, collapsed in nearby Yuli town, the island’s Central News Agency said.
The 70-year-old owner of the building and his wife were rescued first, but it took longer to get to a 39-year-old woman and her five-year-old daughter.
A photo released by the Hualien city government showed the girl lying on a blanket and being handed down a metal ladder from the top of the debris by helmeted rescue workers in orange uniforms.
Photos showed the top two storeys sprawled across a narrow street and on to the other side, with electricity wires pulled down by the fallen building.
Police and firefighters rushed to a bridge collapse on a two-lane road in what appeared to be a rural part of Yuli where three people and one or more vehicles may have fallen off, according to media reports.
A landslide trapped nearly 400 tourists on a mountain in Yuli famous for the orange day lilies that blanket its slopes at this time of year.
A canopy on a platform at Dongli station in Fuli town, which is between Yuli and the quake’s epicentre at Chishang, hit a train, leaving three of the cars tilted at an angle, the agency said, citing the railway administration.
The shaking was felt at the north end of the island in the capital Taipei.
In Taoyuan city, west of Taipei and 210 kilometres (130 miles) north of the epicentre, a man was injured when a ceiling collapsed on the fifth floor of a sports centre.
The Japan Meteorological Agency issued a tsunami advisory for several southern Japanese islands near Taiwan, but later lifted it.