The annual Shetland Up-Helly-Aa festival celebrates the islands' Norse heritage through an intoxicating mix of primal fire and tribal loyalties

• Locals dressed as Vikings march through the streets of Lerwick

The only light is from the clock face of the turreted town hall; the spotlit flag flying on top – a raven on a scarlet ground – informs the local people, as if they did not know, that tonight is Up-Helly-Aa. Since 1873, the fire festival has been held on the last Tuesday in January. It is the highlight of Lerwick's year and surely Scotland's greatest, weirdest spectacle.

All is black. The crush of bodies on the roads and pavements is sensed not seen, as is the feeling of poised anticipation. Then, in the split-second of a camera flash, strange figures become visible. Flash! A giant puffin. Flash! A burly shepherd crammed into a skimpy dress.

Flash! A man wearing a Tommy Sheridan mask. A grandad in drag speaks urgently to his wife: "Quick, tak my photo so I kin pit it on Facebook." These are just four of Up-Helly-Aa's 959 guizers, split between 47 separate squads. The men in costume will spend the next hour or so processing through the town, most carrying a large flaming torch.

Suddenly, there is a huge bang as a firework is sent up to signal the start. Dozens of distress flares, hissing like angry cats, turn the scene an unearthly incandescent pink. Marshalls use these to light the torches, and in an instant the air is full of fire and paraffin reek.

The procession moves off, led by the Jarl Squad, 50 men dressed as Viking warriors. They pull behind them a nine-metre replica galley with a dragon's head prow, built in secret over the winter and revealed to the public for the first time that morning. Within the ship, wreathed in smoke, torchlight reflecting from the mirrored scales of his armoured breastplate, wearing a silver helmet plumed with great arcs of raven feathers, sits Up-Helly-Aa's most esteemed figure – the Guizer Jarl.

• A bus driver stops to take a picture of the Jarl Squad

In ordinary life he is John Hunter, a 37-year-old lifeguard at the local swimming pool, but Up-Helly-Aa has little to do with ordinary life, and so Hunter is effectively king for a day – hailed, honoured and cheered wherever he and his galley pass. Early that morning, the Jarl Squad had stopped briefly outside the home of the Goudie family. As a heavily pregnant young woman cheerily distributed drinks – "Who wants gin? Gin! Gin! Gin!" – her mother, Pearl Goudie, offered Hunter some food. Afterwards, as the Vikings advanced down the street, she regarded the leftover crust as if it were a holy relic: "The Jarl," she declared, "et that sandwich."

The Vikings have been touring Lerwick since dawn and will continue to do so until 8am the following day. But right here, right now – the fiery procession – is the crucial moment. The crowd roars the Guizer Jarl on, and he and his squad roar back, raising their axes in acknowledgement of the 5,000 spectators who have come out to celebrate both their Norse heritage and the community itself.

Hunter uses the word "kindred" to describe his feelings about Up-Helly-Aa and it's true that, despite the huge scale, the festival is a sort of family occasion, uniting old and young in a sense of belonging. Hunter, now Guizer Jarl, has memories of Up-Helly-Aa going back to early childhood. His late mother Kathleen loved the festival. She died before her time in 2002. Hunter has honoured her memory by having her portrait painted into The Bill, the ceremonial placard erected in Lerwick's Market Square on Up-Helly-Aa day.

Though Up-Helly-Aa is full of delicate personal resonances like those, it is undeniably a grand public event. Fire as far as the eye can see. Wind whips and stretches the flames. The heat from the torches is intense. To brass band tunes reminiscent of VE Day, the guizers march into the King George V park, spiralling like a fiery maelstrom around the Viking ship in the centre. "They'll burn the galley in a peerie minute," a mother tells her daughter, holding the restless infant on her shoulders. Peerie is the Shetland dialect word for little. A good few folk tonight have had a peerie dram, for instance. In the morning they will have a muckle headache.

Watching from his bedroom window, golden light flickering across his face, is 76-year-old Allan Anderson, a retired and widowed postman who was Guizer Jarl in 1971. He has two children and four grandchildren, and says being Jarl was the highlight of his life. "Up-Helly-Aa is far more than just an excuse for a drink, isn't it?" someone asks. "Oh my, yes," Anderson replies. "It is a brilliant excuse, though."

Although the Jarl Squad are the icons of Up-Helly-Aa, their bearded, snarling images beamed around the world, the majority of guizers dress in costumes which, though always eye-catching, are rather less magnificent. "A good mix of satire and vulgarity," is one guizer's summation. Often the disguises are topical, hence Tommy Sheridan, due for sentencing the following day. Sometimes understanding them requires a deep local knowledge. A guizer with a rubber quiff and inflatable guitar might be taken for Elvis if you didn't know the costume was a reference to Showaddywaddy, who performed at the local leisure centre many years ago.

The most common aspect of the guizing is cross-dressing. Not for nothing is Up-Helly-Aa known as Transvestite Tuesday. "I had a man in to try on a pair of tights and he had shaved his legs for the occasion," Inga Scott, owner of local fancy-dress shop The Stage Door, said earlier. "I'm the UK's biggest seller of extra-large fishnets."

"And," said Mandy, the shop assistant, "we've sold oot ah fake boobs."

The cross-dressing no doubt has something to do with the fact that women are not allowed to form a squad and take part in the Up-Helly-Aa procession. But Scott just laughed when asked whether she feels oppressed. "It doesn't really bother me. It's a tradition and you get used to it. Up-Helly-Aa is like a man's wedding day. He can be waiting years for it. It's probably more expensive than a wedding, actually."

Up-Helly-Aa was once a modest affair. Costumes were make-do and mend. But with the coming of North Sea oil, Shetland grew rich and the costumes became increasingly sophisticated. It wasn't unknown for the Jarl Squad to import bear and wolf skin for their capes.

This year's squad have settled for rabbit. But they have not skimped on the intricately carved helmets, shields and weapons. A Jarl Squad suit can cost over 1,500, or 900 for a child's version. It is an expensive business. Having been elected at a mass meeting, men know they are going to be Guizer Jarl 15 years in advance, and they spend that time saving. One former Jarl is said to have bankrupted himself.

It would be surprising if money is in John Hunter's thoughts as he surveys the scene inside the park through eyes red with smoke and emotion. He climbs down from the galley and a triumphant bugle sounds (this is such a well-loved noise that many locals have it as their mobile ringtone). On the final note, the guizers surrounding the galley hurl their torches on to its deck. The moment feels dangerous, vivid, wild. They sing The Norseman's Home then turn and leave, happy that the ritual has been observed for another year. Within a short time, the park has emptied. A constellation of sparks flies upward from the galley. It's a clear, cold night and the stars are out. The Plough hangs over Lerwick, bright and sharp like a Viking axe.

Though the aesthetics of Up-Helly-Aa are Norse, the festival was developed in the late Victorian period by a group of bright young socialists who wanted to give an aesthetic overhaul to the town's festive celebrations, which until that point had centred around hauling burning tar barrels through Lerwick's narrow streets in addition to such occasional pleasantries as firing a cannon full of dead cats.

The 1870s were years in which islanders were becoming increasingly interested in their Norse heritage. Shetland was colonised by Norsemen in the ninth century and didn't become part of Scotland until 1468. Ask Shetlanders whether they feel Scandinavian or Scottish and many reply thus: "Baith."

Shetland does seem a place where identity is in flux and the usual social and cultural divisions have been eroded. In Lerwick's masonic lodge, at lunchtime on the day before Up-Helly-Aa, young men in Celtic tops sank pints; sinking pots at the pool table, 30-year-old Ryan Wright bent over his cue, the sleeves on his Motorhead T-shirt riding up to reveal his name inked in Viking runes and a tattoo of a burning galley. The stereo played Scottish country dance music and Dixieland jazz. A jar of pickled eggs sat on a pub table, and round the table sat a group of men eating and drinking, enjoying a break.

These were the "torch boys" - the men charged with making the 1,153 torches for Up-Helly-Aa. Kay and Flossie, two middle-aged women dressed as nuns, had popped round to the lodge from the Happy Haddock with 30 fish suppers to sustain the hardworking crew.

There is a kid-on rivalry between the torch boys and the galley boys who build the ship. Really, there's a great deal of mutual respect. Up-Helly-Aa is run by volunteers. There is a huge amount to do. Planning for each festival begins in the preceding February. Work begins on the galley and torches in October.

Up-Helly-Aa enjoys a great deal of continuity, both in the ritual itself and in the people who run it. There are some highly esteemed men, tribe elders of a sort, who have performed the same role for many years.

Lowrie Shearer is 69, has been making torches for 44 years, and hopes to continue until the half-century. Jim Nicolson is 77 and has steered the galley through Lerwick's streets every year since 1963; he looks regal and mystical in his horned helmet and white beard. Roy Leask, 58, has been involved in building the galley since he was 15. He knew James Smith, a boatbuilder known as Boatie Jeemie, who established the template for the present galley in 1949, the first Up-Helly-Aa following the war. The skills Leask learned from the old men of his youth, he now passes on to a new generation of keen boys, and so it goes on.

Up-Helly-Aa is altogether remarkable, and some of its most remarkable sights can be seen in the halls – the buildings in which hostesses stage parties following the procession. Typically, food is provided but you bring your own drink. Bands play strip-the- willow and the like, breaking off as each Up-Helly-Aa squad arrives and performs their act.

In Bell's Brae Primary School, the walls of the assembly hall are lined with Jarl Squad shields going back to the 1920s. It is around midnight and the party will go on till breakfast. There are prim older ladies and gentlemen seated around the edge of the dancefloor. Among the young women it's all handbags and gladrags, Lambrini and cutty-sarks. A twentysomething's gold dress rides up to her waist as she is lifted on to the dancefloor by a man in a white bodystocking.

The atmosphere isn't horribly drunken, though, or especially sexual. It's quite tender in a way. Watching people of all ages dancing to Wild Mountain Thyme, it becomes clear that the true tradition of Up-Helly-Aa has nothing to do with Vikings but rather is to do with the generations of Shetlanders who have a clear sense of belonging to this place and have certain values in common – kindness, empathy and an enviable capacity for joy.

No sooner does that thought occur, though, than a squad turns up and performs – to rapturous applause – a dance routine to Fight For This Love, led by a plump 21-year-old man dressed as Cheryl Cole in peaked military cap and buttock-revealing leotard. It's a vision that, combined with heat, lack of sleep and neat whisky, can leave an outsider disorientated and confused. A good time, perhaps, to heed the advice John Hunter, the Guizer Jarl, had offered during an earlier conversation.

"Up-Helly-Aa's just Up-Helly-Aa," he said. "You don't ask any questions. You just get on with it."

• This article was first published in the Scotland on Sunday on January 30, 2011