Jake Arnott interview: The thrill of the novel

He was never very comfortable being labelled "crime writer" after his early novels, The Long Firm and He Kills Coppers, were highly praised for their accurate depiction of 1960s gangland culture in London's East End and adapted for television. His new novel, The Devil's Paintbrush, is a deliberate departure and does for occultist Aleister Crowley what his earlier work did for the Kray twins. But in his private life, too, Arnott has defied definition. Once a "gay writer", he was what it said on the jar until he fell in love with a woman. That really confused the labellers at the bottling plant because the woman he fell in love with was a lesbian, fellow writer Stephanie Theobald, who had hitherto been scathing about bisexuality. (Oh, never mind, a friend soothed her, just tell people you're proud to have been a hypocrite.)

For Arnott, it's actually not about hypocrisy or changing minds. "I think there is always a problem with labelling and identifying people by singular aspects of their character. I have always believed there is a continuing spectrum of sexuality. It's not a bipolar world. So I don't feel I have changed, I feel I have continued to have an identity that is about possibility and inclusiveness rather than something that is shut." But there's no doubt it was a surprise, despite a past bisexual period in his life. "To be honest with you, I never thought I would have a relationship with another woman. I can't imagine it with anyone else."

So why did he fall in love with her? There's a lovely long pause, more thoughtful than awkward, in which Arnott considers the question. He's sitting in an office at his publishing house, a civil man with dark eyes that reflect a mixture of kindness and anxiety. You can often tell the people for whom life has been just a little too kind for their own good. Not Arnott. When he says he was an anxious, nerdy little boy with an eye patch who was usually picked last for football teams, it explains a lot.

It's not that there is some obvious sign, a weeping scab from which blood still trickles at the edges. But the boy is still there somehow, in the politeness, almost deference, with which he engages in argument; in the keenness of his interest in things around him; in the internal focus of him rather than the external largesse that some successful people display. When I arrive an hour late because of a fog-delayed flight, I am embarrassed, inclined to throw myself at his feet in supplication – until the realisation hits that he has neither that big an ego nor that short a temper. "How awful for you," he says. "Is there anything I can get you?"

He is still musing gently. Good question. Why did he fall in love with Stephanie? "She's…" he begins, and pauses. It's hard to put into words and that frustrates him because words are his business. "I love her energy," he continues. "Physically, she…" He breaks off again. There's a Keats poem that she reminded him of. "Full beautiful – a faery child," he quotes. "That's how I felt about her." The line is taken from Keats's ballad La Belle Dame Sans Merci. "I met a lady in the meads/Full beautiful – a faery's child/Her hair was long, her foot was light/And her eyes were wild."You can see what he means in pictures of Theobald, with her elfin chin, clear eyes and abundance of untamed corkscrew curls. "There was something sort of fantastically glamorous in the proper sense of the word," he says. "Otherworldy. Tremendously intelligent and very emotionally bright as well. She has had a very wild and wonderful experience, as I have, in terms of emotions. We do connect really. I don't feel lonely with her. I try not to judge any relationships because you don't know how they are going to pan out, but with Steph there's constant stimulation and inspiration, and pleasure actually. She has a fantastic sense of what is pleasurable in life."

I wonder if, emotionally, it is very different to love a woman after loving a man. "The obvious answer would be that each person is very different anyway. I don't know how different women are to men but I think we're closer than we think. The notion of a continuum of sexuality, you could almost apply it to gender as well. To say there are just two genders doesn't cover it. I know lots of transgendered people."

If gender is sometimes a stereotype, so too is desire, Arnott thinks. "It's very easy for men to fetishise desire. It's certainly common among gay men that they will like a particular type of other man, and heterosexual men do that as well. They see their desire reflected back rather than seeing the actual person. You see this in representations of women all the time, that men aren't really interested in looking at women. All they're interested in is looking at what they want that image to be. By the end it can be a hall of mirrors with a distorted image because it has been reflected more than once."

Perhaps the biggest mental leap in moving from a homosexual to a heterosexual relationship is the very different life possibilities it forces you to think about. Children, for example. "I suppose so," he admits, then smiles. "Though, having said that, there were certain relationships I've had with men that were more likely to have children. Maybe if I could be the mother… It's not something you completely ignore but we've both, recently anyway, been engaged in having children in terms of books. That's who our children are."

When Arnott first encountered the work of Sigmund Freud, the father of psychoanalysis, Freud's theories on the Oedipus complex made perfect sense to him. "I was very much my mother's son. In a very Freudian way, I identified with my mother and felt distanced from my father. I over-idealised her." Inevitably, that led to disappointment. "The problem is they can never live up to that, and you feel slightly resentful. Mothers and sons, like fathers and daughters, can sometimes have this idealised polarity. It's funny now because I get on really well with both of them and I realise I had just as many issues with my mother."

It was Freud who developed theories not only about boys and their relationships with their mothers, but about the way humans use repression as a defence mechanism. The world of Arnott's 1960s crime fiction has its share of repressed homosexuals, including central gangster Harry Stark. These were different times. Arnott's homosexuals may have a measure of power or success, but at their core they are hunted figures, forced to live out the reality of their sexuality in a furtive rather than open way. It was never crime per se that fascinated Arnott in these novels. "I wasn't trying to depict an underworld. What interested me was how different layers of society connect. You can have an aristocrat and someone from the working classes occupying the same space. An academic, a journalist, people from show business…"

Arnott admits he is fascinated by masculinity, and his female characters are, with a few exceptions, less central than the men in his stories. "Masculinity is something I was very conscious of from an early age." He sensed that he wasn't measuring up? "Absolutely. I was very good in my teens at putting on a more butch persona than I really am. But, of course, that's what all boys do."

Throughout history, boys have sometimes turned to the army to assert their masculinity. In his new novel, Arnott turns to an earlier period, when he believes the army and sexual repression went hand in hand. Set in Paris at the turn of the century, The Devil's Paintbrush is the story of a real historical figure, Sir Hector MacDonald, a Scottish hero of the British Empire. Despite being decorated for his actions in Sudan and Afghanistan, MacDonald faced ruin in a homosexual scandal, fleeing to Paris before finally committing suicide. In real life, he had one chance meeting in the city with the infamous occultist Aleister Crowley, and Arnott uses this meeting as a springboard for a fictional tale that entwines the two figures closely together and charts the final days of MacDonald's life.

The idea of empire interests Arnott. "I do have particularly strong views about empire and the countries that actually made it possible. It's interesting the way the Ghurkas have been treated. It's a real hangover. The empire was built on the fact that it wasn't necessarily the English – who were getting a lot of the credit – who were doing all the work, and certainly not all the fighting. In the book, I do make reference to the fact that the Scots sometimes feel that they are slave soldiers to the English."

In MacDonald's case, some of his heroism was about trying to establish his masculinity in the face of his own homosexuality. "What interested me about this period was that this was happening on a huge scale, on an imperial level. You had people who wanted to prove themselves essentially because they weren't entirely happy with what they were inside. The empire was virtually created by this. I make the analogy with steam power. This was something the Victorians instinctively knew, that there was power in this, that if you put pressure on something you will get something back."

Was Arnott's own sexuality part of the problem with his father? "I don't think it was an issue for him particularly, but I don't think I could have talked to him. There were a lot of emotional things round my sexuality that were happening, and yes, there were times growing up when I completely repressed my sexuality. I was terrified of the idea that I might not be completely heterosexual. Me and my dad had quite a hard time with each other, and I think for obvious reasons fathers and sons can find it quite hard to get on. I identified so many of my problems with him that weren't his problems at all." He laughs. "I have to say, looking back he was a wonderful father, and I gave him far too much of a hard time."

His father was a management consultant, and Arnott acknowledges his upbringing was middle-class. Yet he followed none of the expected pathways of boys in his position and did very badly at secondary school. "I passed the 11 plus and realised it wasn't the most stupid kids who didn't go to grammar school and the brightest who did. It was arbitrary. I was aware at a very early age that a lot of it was to do with social class. The school was really engineered for 1960s work culture. There was a grammar school, a secondary modern and a technical high school as well, so you could pass the 11 plus and go either to grammar school or technical high. It was quite clear, aged 11, where the shop floor was, where the technicians were and where the middle-management was. And none of those things particularly appealed to me. It was clear the majority of us were going to end up in middle-management, and there's nothing wrong with that, but it's all a little bit fixed."

For a while he became the Angry Young Man who rejected everything of his comfortable and preordained life – his father, his education, his parents' Catholicism (he stopped believing when he was 13 and never found a reason to start again). What he really wanted was to forge his own path. He lived in squats with anarchists and worked in a series of jobs, from labourer to mortuary attendant – and jobs don't come more dead-end than that. "I didn't really believe there was very much that I could learn at school, which sounds ridiculous now. I was interested in reading and I was interested in educating myself, and that's sort of what I did really. I was quite lucky in my teens and early 20s, finding people and situations where there was quite a lot I could learn."

He has always thought of himself as a bit of an outsider. "I found it quite hard to fit in," he admits. "I think I have always found it quite hard to feel entirely part of a group." He doesn't know why, but he now realises that sense of self-sufficiency had its advantages. "As I get older, I realise it's one of the reasons I'm a writer. I did actually like spending time on my own, and that's the thing you have to do as a writer; the only thing really, to find time to spend alone. I had an imaginary world and could play with my own space. I had a very crowded upbringing, with seven in a semi-detached house, but I think one of my responses was to withdraw slightly. I was quiet and shy and liked the idea of writing and stories."

He wasn't published until he was 38, and once said it took him most of his life to pluck up the courage to write. Yet the way the story goes, The Long Firm was supposed to have been snapped up by an agent and sold within days for a big advance. "It's the old clich," he says. "You spend a lifetime becoming an overnight success. People love the idea of it, but I had been writing for a long time and thinking about it. The weird thing was that I had years of nothing happening and then it was sudden."

You might think that success brings security, but Arnott doesn't do things the easy way in life. "I think that's the death knell really, if you feel secure. You have to feel confident enough to work, but I don't ever want to feel secure. If it's not difficult, I know it's wrong – for me anyway."

In the past, Arnott has written critically of author Graham Greene, suggesting Catholicism is anti-intellectual and therefore, the inference goes, not really a 'proper' subject for a novel. (If you want a reasonable attempt at intellectual Catholicism, he argues, it would have to be Anthony Burgess rather than Greene.) Leaving aside the wrath of theologians who consider themselves very intellectual indeed, the whole argument suggests a respect for intellectualism that many who take a more traditional educational route than Arnott are often cavalier about. Who says novels always have to be intellectual?

The Devil's Paintbrush seems more self-consciously literary than The Long Firm, perhaps a deliberate attempt to move away from the crime writing label. "One of the advantages of people pigeonholing you is that you can try to avoid it," admits Arnott. "I have never wanted to be stuck in any particular period or genre." So is it important to him to have an intellectual strand to his writing? "I'd love to be able to jettison it. I do want to be driven by the emotional." In fact, he says, there is a need as a writer to embrace things that are illogical or absurd, or, like the occult, hidden.

Love brings surprises, Arnott says. Interesting to speculate, then, whether his writing has changed because of his relationship with a woman. In some ways, he says, it wasn't the fact that Theobald was a woman that gave him most pause for thought but the fact that she was a writer. "I thought that would be difficult. I never thought I would be able to have a relationship with another writer." In fact it has been a source of enormous inspiration. "She has raised my game. I am able to talk to her about stuff, and vice versa."

Is the relationship competitive? "I don't think so. I hope it never will be." Women, Arnott argues, make up the bulk of the publishing industry, yet female writers have a much harder time being taken seriously. He knows it. "To get respect and recognition, they have to work much harder. But I know she's a better writer than me. Sentence for sentence, she has a tremendous grasp of language that is slightly out of my reach. But that has made me want to do better. I find that encouraging. On paper, I might be a slightly more successful writer than she is but not on the real paper."

People have, on the whole, been supportive of their relationship. The editor of Gay Times phoned him recently to ask Arnott to write for him. "I had to say, 'Look, I've kind of done a Tom Robinson…' But he was so sweet and said it made no difference at all." In fact, it has been interesting watching where disapproval has come from. The problem with radical change, Arnott acknowledges, is that it doesn't just challenge you personally. "It can challenge everyone's status quo." Some people like things neatly labelled. But for Arnott, the whole joy of being alive is the possibility that life might just deliver something other than what it says on the jar. "I don't want to know what's going to happen," he insists. "I do want continuous surprise."

• The Devil's Paintbrush is published on Thursday (Sceptre, 15)