Interview: Kate Atkinson, author

That was back in 2004, when she first started writing about her policeman-turned-private investigator Jackson Brodie. She's written three more novels featuring him since then, and the latest - Started Early, Took My Dog - comes out next week. I'm in a Marchmont cafe to interview her about it over breakfast. Over the steamy shrieks of the cappuccino machine, I soon realise that this is going, for Jackson Brodie fans anyway, to be one of those "good news, bad news" conversations.

Bad news first. "I might not be writing about Jackson any more," she says. "Well, not for a while anyway." Having sworn me to secrecy - "because I've a track record of talking about books I never write; in Australia they think I'm about to write a book about Jane Austen. Something I said at some festival..." - she tells me about the three books she hopes to write before she next brings Jackson to life "maybe in Paris, maybe a noir book of Jackson in America". She sees my hopes rise. "But I'll be nearly 70 by then." As she looks as though she's just turned 50, my hopes visibly slump again.

"The fact is, I feel completely written out about Jackson. But I felt I had to bring him to the point where I could leave him for a while. I need some new structural device - I know readers say you should marry him off or something, but I need more than that: something you can hang a plot on."

Now if you haven't read any of Kate Atkinson's novels - and certainly if you have never read any of her Jackson Brodie ones - you're probably thinking that there's something a bit precious about this conversation in an Edinburgh cafe. Author decides to postpone writing about central character: so what?

Here's why it matters. In all of crime fiction - she hates that delineation, but let it stand for the moment - there's nobody writing as playfully, as imaginatively and as subtly as Atkinson does in her Jackson Brodie novels. Comparing her to even the best-selling British crime writers is like comparing a Scottish First Division team to the way Spain played in this year's World Cup.In theory, they're both playing the same game - there's a murder or two, invariably with all the clues pointing to the wrong suspect, and the goal is to resolve the case as stylishly as possible. In practice, there's no comparison: whereas the Scottish side might hoof the ball upfield more in hope than expectation, Spain play a controlled game of intricate midfield passing, in which the flair lies in the build-up, in imaginative running off the ball as much as the clinical finishing.

The point - and I'll drop the metaphor in a minute - is that a Kate Atkinson novel isn't bluntly direct but contains similarly skilfully controlled games within games. The posh word for this is metafiction, and the reason she can't just marry off Jackson to wife number three and shift him off to solve a case in, say, Tuscany is that this just wouldn't be enough. In Starting Early, for example, there's not only Jackson Brodie on the case but (unknown to him) another private detective with the suspiciously similar name of Brian Jackson. Their investigations run on parallel tracks. Brodie and another former detective bump into each other, each unaware that the other holds the key to the case they are both working on, one of them having kidnapped a child, the other having kidnapped a dog. One of the other characters is an elderly actress called Tilly who's got a job playing the mother of a TV detective in a series somewhat like Heartbeat in which Jackson's former partner Julia also makes a brief appearance. And when Jackson meets Julia, you're not completely sure that he has done, so often does he have imaginary conversations with her in his head. Similarly, Tilly isn't quite sure whether what she's seeing is real or imaginary, because that's what happens when dementia is starting to take hold of an actress's mind in which so much fiction has already metastasised into fact.

If you've read the rest of Atkinson's non-Jackson Brodie fiction, starting with her stunning debut, Behind the Scenes at the Museum, you can see how they mirror such playful inventiveness. But in those first three books, she was also creating an individuated form for each novel. By 2000, when she wrote her third one, Emotionally Weird, the effort was becoming too great. Effectively, she admits, she was writing herself into a metafictional corner. "I'd quite like to go back and break out of it," she says. "Certainly I had a really terrible time with Emotionally Weird. When I finished it, I thought, I can't write any more. That's why I wrote Not the End of the World (her wonderful 2002 collection of short stories] - because I knew that if I wanted to carry on being a writer, I had to find pleasure in writing again - and I love writing short stories. Every one of them was a pleasure, not a chore. Before then, I'd never really thought I was a writer - I was just sort of pushing my way in, but I wasn't the real thing.It wasn't that I lacked self-confidence, although it sounds like it - I'd never have written Behind the Scenes if that were the case. I just thought, 'I don't like writing enough to think I'm going to sit down and write a book and not have a nervous breakdown at the end of it.'"

That next book was Case Histories (2004), the first of the Jackson Brodie novels. Back then - and even two years later, when the second one came out - she'd have loathed them being called that. In her own mind, not only wasn't she a crime novelist, she absolutely, positively, definitely wasn't aiming to do a series about Jackson Brodie. However, never having written crime before, she instinctively grasped that the apparent limitations of the genre were actually liberating. "By becoming a crime writer," critic and novelist Patrick Ness astutely noted, "she has become a better literary writer than ever - funny, bracing, and delightfully prickly."

The great advantage of crime was that it offered an overarching framework she could use her imagination to subvert. And here's the second reason why her Jackson Brodie novels work so effectively: if they were just about playing games, they'd be interesting, but uninvolving. Add, however, her ability to write from inside her characters' minds - not just Jackson Brodie's, but all the other main players' too - and something intriguing happened. Her books became a refreshingly new, and entirely enjoyable, kind of novel.

Back in the bustling cafe, I get an insight into Atkinson's approach. I'd asked her why she bothered giving Brodie's backstory in her novels: so many frankly freakish things (a train crash, a huge inheritance, multiple murders) had happened to him in previous books, I argued, that merely hinting at such a complex past diminished his credibility. I knew she realised the risk: "The resume of his (Brodie's] life," she writes at one point in Started Early, "was more exciting than the extended version." So why bother including it in the first place?"

"But I quite like the irony in that. There's a knowingness there, because you're saying it's not really formulaic, whereas in fact it is. You've got your lone detective who is - God help us, I hate the phrase - a maverick, who is divorced, who has trouble with women but is still very much macho. And that's very much a stock figure.

"So what you're doing is taking stock elements and then playing around with them. I mean, he's not completely a stock figure, because he has a lot of mental energy and a lot preying on his mind. But I think the way not to write clichd characters is to inhabit them. If you write characters from the outside, you're doing that one great thing which all writers are told they mustn't - to tell, not show.Whereas if you write from the inside, you can avoid a lot of the clichs and home in on how people are feeling, how they see life.

"Mainstream crime is very end-driven: there's a plot that goes directly from A to B, and all the detective is doing is going about picking up clues. That's important, but it's not what these books are about. The interesting thing to me is character - I could have written a whole book about Tilly, for example - and character is unfashionable in crime. What a lot of crime fans want is plot, plot, plot, and character just gets in the way. So when you get something more rounded and interesting, it's inevitably not going to be mainstream."

I'm not sure about that last bit. On this side of the Atlantic at least, writing character-heavy crime hasn't been tried enough (US crime fiction is much better in this regard).And that, I suppose, is why - though I look forward to Atkinson's next non-Jackson Brodie novels - I'm saddened that I'll have to wait so long for the next one like this.

When, I'm thinking, will there be good news?

Well, in a sense, there already is. Firstly, there's the fact that Started Early, Took My Dog - from its flamboyant start in 1970s Leeds to its satisfyingly complex denouement there in the present - is every bit as witty, pacey, imaginative and mordantly observant as her readers have come to expect. Nobody writes this kind of novel better.

In Started Early, Took My Dog (the title is from an Emily Dickinson poem: Brodie turns out to be a fan) we are back in Atkinson's native Yorkshire. He'll find himself uncovering a case that reaches back 35 years, when the police were more corrupt and the cities fuller of soot and starlings than the sandblasted, post-industrial present. He'll untangle a nicely knotty case that will take him through so many parts of his and Atkinson's (and, as it happens, my own) remembered pasts - the abbeys, the coast, the heartlands of God's Own Country - that Yorkshire's tourist authorities ought to be throwing money at any film company wanting to give the novel an afterlife on the big or small screen.

Which leads me to a second bit of good news. Next month, filming of the first three Jackson Brodie novels will begin, with Edinburgh standing in for the Cambridge of the Case Histories and Jason Isaacs (Lucius Malfoy in the Harry Potter films) in the title role. Locations have already been scouted, scripts already written."It won't use Edinburgh as a character in the same way as, say, Ian Rankin uses Edinburgh," says Atkinson, "but I think it will be real enough - and certainly in the sense of the bourgeois Edinburgh I inhabit, along with a number of the characters from the books, like Gloria from One Good Turn." The six-part series, to be made by London-based Ruby Films (who also made the TV adaptation of Andrea Levy's Small Island and the film of Monica Ali's Brick Lane) will be shown on BBC 1 next spring.

Right now, though, says Atkinson, she needs a break from Brodie. "The first book was such a joy to write because I came to it absolutely fresh," she says. "It went so well that I carried on with a second novel, even though I had never intended to. With this one, although it helped that I could take him back to Yorkshire, the plotting was so complicated towards the end that it really made me realise that I needed a break. I know that I've got to come back fresh and I don't feel that at the moment. He's just got to develop offstage, and I need to go off and do something different. So the next time I see him might be in seven years' time, something like that." And then she offers me a tiny Emily Dickinson feather of hope. "But I might change my mind."

• Started Early, Took My Dog by Kate Atkinson is published on 19 August by Doubleday, priced 18.99. It is RNIB Scotland's latest Talking Book, also available in braille.