Why are Scotland's magicians in crisis?

He points to one particular illusion and reads aloud the title: "To thrust a knife through your arm and cut half your nose asunder." He chuckles. "That's great, innit? You'd be lucky to see a 'nail-through-finger' these days."

The book is one of many thousands in Bruce's extraordinary library, known as Scotland's Magic Archive, the result of 40 years of collecting. He is 57 years old, a tall, distinguished gentleman in a white Panama hat.

Bruce's day job for decades has been playing double bass with the Royal Scottish National Orchestra, but magic is his ruling passion. In this, he has something in common with the 200 or so people in Scotland – most of them men, most of them getting on a bit – who call themselves magicians and spend a great deal of time and money in pursuit of an interest that often becomes an obsession.

You could walk past Bruce's home and think it derelict. Through tall grass, the ghost of a path leads to the front door. Inside, the floorboards lie bare and the walls are paperless and peeling. In one room, the ceiling is coming down and ivy has thrust its way through the windowpane. Once, a fox cub ventured in and set up home in a wardrobe.

It is safe to say that Bruce does not spend a great deal of his salary at B&Q. His money goes on the old, leather-bound tomes that line his rooms. Certain titles stand out from the shelves: Curiousities For The Ingenious; The Case For Spirit Photography; The History Of The Aztec Lilliputians. Finding these books, some of them very rare, is itself a kind of conjuring trick. Bruce makes them appear from history.

Discoverie Of Witchcraft set him back 1,600 at auction. He had expected it might go for tens of thousands and that he wouldn't be able to afford it. "I had thought about remortgaging the house. I wasn't eating and I grew a beard. It was awful." A wealthy friend arranged to loan him the money. Having secured Discoverie, he went across the road to the Art Club, toasted his success in champagne bought at the off license, and performed tricks from the book, though not, presumably, "To thrust a bodkin into your head and through your tongue". BYOB, even in Glasgow, doesn't stand for Bring Your Own Bodkin.

Bruce is happy to admit he is obsessed with magic. "I'll quite often sit up till four in the morning, just practising cards." The point of his archive, which he began by saving his school dinner money, is to allow Scottish magicians to research the history of conjuring and its development as a subculture. There is certainly a rich history of magic in Scotland, peopled by colourful characters.

Take, for instance, Dr Walford Bodie, an Aberdonian magician born in 1869, who had a Dali-esque moustache and a hat like a bus clippie. According to his headed notepaper, which Bruce owns, he was "the Earth's greatest entertainer and scientist ... cheered to the echo nightly". He had a routine in which he'd get an audience member on stage, put a potato in their mouth, pass electricity through their head, then present the cooked vegetable to the ecstatic audience. You don't get that with Derren Brown. Or Spudulike.

Bruce's fascination with magic goes back to his schooldays. A fellow pupil showed him a card trick and said he'd bought it at Tam Shepherd's Trick Shop on Queen Street. "So I trot off down there after school and the guy behind the counter said that if there was any trick I didn't understand, I should ask him and he would try to help."

The guy behind the counter was Roy Walton, one of the greatest card magicians in the world. He and the shop are still there, and their influence over Scottish magic cannot be overestimated. Tam Shepherd's is a hub where several generations of budding magicians have gathered to speak to Walton, seek advice and pay tribute to the modest man, now well into his seventies; Jerry Sadowitz is the best known alumnus. It's a dim, old-fashioned and cosy space with chansons on the stereo and plastic dog poo under glass; a refuge from a humdrum world.

That is the sort of escapology with which magic is less associated, but it is an important aspect of its appeal. "Magic is life-enhancing," says Bruce. "You take the audience out of themselves for a while. They're not worried about the recession, the mortgage, or being behind with the council tax."

Yet, for all that, magic appears to be experiencing a crisis. If you ask, Bruce would describe for you the 20,000 people who lined Edinburgh's Lothian Road in 1911 to watch the funeral procession of The Great Lafayette, who had died in a fire. Such a mass outpouring of grief and curiosity would never happen now. Magic just doesn't have that kind of popular appeal.

One problem is that there simply aren't enough small and medium-scale cabaret-style venues to nurture and sustain developing performers. There is nowhere in which to be bad until you are good. Most working magicians perform live for non-magicians only when they are booked for parties or corporate functions, which may pay the bills but doesn't foster creativity.

Britain's Got Talent, which on the face of it seems to promise a revival of the variety format, seems not to be having any effect either. There is a perception among magicians that Simon Cowell and Piers Morgan dislike live magic and therefore there is no point in auditioning.

"It's not something I would ever do," says Ian Kendall, a professional magician in Edinburgh. "But magic has a really bad image, mainly because most people have only ever seen it at a kid's party, or they've got this image of a socially inept guy forcing card trick after card trick on you. Dancing dogs get more respect. It's frustrating, because I've spent an average of five hours a day for 25 years on this."

Kendall is 42, a former flight instructor and IT consultant who now tries to make a living from magic alone. "What's the one thing a pizza can do that a magician can't? Feed a family of four," he jokes rather grimly.

Talking to magicians, you get the distinct sense that magic has been the most important thing in their lives. It's more than a job, more than a hobby. It's a calling. A Paul Daniels On The Road To Damascus moment.

For 30-year-old Kevin McMahon, the route was late and unique. In 2005, while working as a physics researcher at Heriot-Watt University, he appeared on Channel 4's Faking It. After four weeks of intensive training from mentors including Penn and Teller, he was able to fool a panel of magicians into believing he was a professional. Back home in Edinburgh, he quit his job and set about making magic his career.

Now he is a professional with up to five bookings on any given weekend. "Card tricks paid for my mortgage and my car. That's a nice thing to be able to say." But again, his work is restricted largely to weddings, birthday parties, corporate events and so on. "Magic often is done in venues which are not the best for getting the emotions across," he says.

Partly in response to this, he has founded the Edinburgh International Magic Festival, which begins next month. "This will give people a chance to see live magic in a proper theatre with proper sound and lighting. I hope it will lift the PR of magic and its profile in Scotland."

Given his initial boost from TV, McMahon could be forgiven for being something of an upstart. But in fact he is respectful of conjuring heritage and is on the committee of the Edinburgh Magic Circle. There are 11 of these local groups in Scotland and they are crucial to the continuation of the scene. Regular meetings are a safe space in which to discuss the workings of tricks and to honour traditions. "It's almost like church," says McMahon.

However, like church, memberships have been declining steadily, while the age of devotees increases.

This means, though, that local magic circles are knowledge banks, tremendous resources in which you can meet people with a vivid personal connection to a vanishing culture of live entertainment. Take Margaret MacLean, president of Paisley Magic Circle, which has been active for 72 years. MacLean is just one year younger than that. A former justice of the peace, she is proud to have been a pioneer. She was among the very first women, and certainly the first Scottish woman, to become a member of the famous Magic Circle in London.

For her, magic was a way of escaping the gender roles that so constrained a woman of her generation. She was a wife and mother who kept a home, but magic meant self-expression.

Growing up in Govan, she learned to play the accordion, and as a teenager performed in bandstands, church halls, town halls and theatres. One snowy night in 1958 she was on the bill at the Lyric Theatre in Sauchiehall Street when a Paisley magician called David Haggarty asked her to be his assistant. They performed together for the first time on an STV talent show. Their association and friendship continues to this day, surviving both the accidental cutting of her nose with a sword and the time her dress went on fire.

The world of variety theatre MacLean knew and loved is largely gone now, and few young people would consider, as she did, becoming involved in conjuring. One exception is Liam Black, a 14-year-old from Cowdenbeath whose enthusiasm for magic is extraordinary. Magic has preoccupied him for the last five years, ever since he saw a conjurer at a friend's birthday party. He will be competing against other young magicians at the Edinburgh International Magic Festival.

"I like the classic tricks from 100 years ago," he says. "Tricks that make people say, 'Wow!'."

Forget David Blaine, Liam's idols were mostly born in the 19th century. He is especially keen on Cardini, known as the Suave Deceiver. Like his hero, Liam wears white-tie and tails when he performs. He pores over footage of Cardini on YouTube.

The internet and DVDs are an important source of information for him. He has dyslexia, so doesn't read well, and would like to leave school as soon as possible and become a professional magician.

Standing in his sunny back garden, Liam pretending to cut off his own right hand with a fake blade. The neighbours are watching, horrified, the sound of playing children fills the air. The scene is modern and suburban, yet the illusion links back to Reginald Scot's Discoverie Of Witchcraft and that trick about thrusting a knife through your arm.

If magic does have a future in Scotland, it will be because remarkable people like Liam, now as in the 16th century, have an appetite to astonish and be astonished. "Mum," he says. "Run inside and get the bottle of fake blood. It's on the counter."

Edinburgh International Magic Festival (magicfest. co.uk), 7-11 July; see www.kevinmcmahonmagic. com; Ian Kendall is at the Edinburgh Fringe on 21 August, www.iankendall.com; www.scottish associationofmagicalsocieties.co.uk

This article was first published in Scotland on Sunday, June 27, 2010