The power of love: After 30 years of provocation, American author James Ellroy reckons he's found redemption

FOR a man who has written some of the most propulsive, disturbing American fiction of the past two decades, James Ellroy, the self-described "mad dog of American letters," is in comparatively genial mood as he sprawls in the library of a west London hotel. His 6ft 3in frame unfolded across a sofa and coffee table, Ellroy speaks of his love of Britain and ambition to "move to somewhere on the British moors and get a couple of hounds who bay, ow-ow!"

Although his love of dogs - "God's creatures" - is well known, Ellroy the provocateur may be at work here. He can't seem to resist it. He tells me of how on a recent visit to Northern Ireland he told an audience a joke about a lion caught in flagrante with a zebra by the zebra's husband. It fell flat. "So I said, 'Look, this is a Catholic lion f***ing a Protestant zebra,' and they all broke up."

But after 30 years of provocation, Ellroy now claims to have found redemption. His latest autobiographical work, The Hilliker Curse: My Pursuit of Women, charts his obsessive search for love.

It ends with him meeting his new partner, Erika Schickel, the soulmate he had always been searching for. They met at a book fair in Los Angeles: there followed two years of mutual yearning, after which he believes they both "conjured" their eventual coming together last year.

But he says the memoir is not just about him, rather "the sacred conjunction of men and women, the romantic ethos I've always lived by" (because "autobiography only works if you address a bigger issue - otherwise it's just solipsism"). And he insists that this literary exorcism of his demons has worked, as he indeed predicted it would in its closing pages: "the dominant storyline of my life will dissolve on the last page I write here."

There is much to exorcise. Ellroy was born in Los Angeles in 1948; his world was ripped apart by his parents' divorce. When his mother, Jean Hilliker, reacted badly to the ten-year-old James saying he wanted to live with his father, he wished her dead.

Three and a half months later, in 1958, she was murdered. Her killer has never been found. In his autobiographical My Dark Places (1996) he sets out with a retired detective to find the killer, almost 40 years on, a task he now says he knew from the start was futile. It remains one of the most compulsive works of autobiography I have read.

Ellroy's matricidal death wish is the eponymous curse of the new book. The trauma of losing his mother shaped his life. It helped send him off the rails as a teenager, whence he descended into years of voyeurism, petty crime, drugs and alcohol.

Even when he cleaned himself up and began to write in the late 1970s, his search for a woman who was like his mother continued to dominate his life.

That pain and obsession has driven his prose too.He has been said to write "like a man possessed." Certainly both his bestselling LA quartet of noir crime novels set in the city in the 1950s, LA Confidential and three others, and his more recent Underworld USA trilogy, contain some of the most violent, challenging, yet relentlessly compelling fiction of recent years.

His style is boiled down to a frenzied staccato, diamond-hard, peppered with racial epithets and 1960s jive-talk, utterly unflinching. Consider the opening lines of the trilogy's second book, 2001's The Cold Six Thousand: "They sent him to Dallas to kill a nigger pimp named Wendell Durfee. He wasn't sure he could do it. The Casino Operators Council flew him. They supplied first-class fare. They tapped their slush fund. They greased him. They fed him six cold."

In his new memoir, he writes of how he aimed there to create "the novel as sensory assault," written in a style so driving and telegraphic that it "would force readers to inject the book at my own breathless rate." That is certainly how I have consumed most of his novels: like a cocaine rush, enjoyed slightly guiltily but hungrily, urgently.

But with last year's third and closing instalment of the Underworld USA trilogy, there came a change of style and perspective. Blood's a Rover begins in June 1968 after the assassinations of Martin Luther King and Bobby Kennedy and ends with the death of FBI director J Edgar Hoover in March 1972. It is still charged, breathless, dizzyingly plotted. It knocked Dan Brown off the top of bestseller lists. But it isn't quite as relentless or steely as its predecessors, American Tabloid and The Cold Six Thousand. For the first time in his work, several striking female characters take central roles.

The reason for the shift becomes clear in The Hilliker Curse. He wrote Blood's a Rover after the trauma of his second marriage breaking down.

He thought he had met the woman he had been searching for his whole life, Joan, a fiery, left-wing university professor. But he was too needy and intense - "I was dying for love, I'd never had a family." Ultimately she rejected him.

"I got the holy f***ing shit kicked out of me," he grimaces. "It's what was necessary to deflate my ego so I could start over again and write a different kind of book."

He thinks the result is his best novel to date. He is careful to stress the continuities with his other work: "My books are all Christian parables: bad men are drawn to darkness and light and there's always a level of redemption. They're also deeply romantic - bad men in love with strong women."

• The Hilliker Curse: My Pursuit of Women is published by Heinemann.