Woody Allen on his new film Magic in the Moonlight

Woody Allen with wife Soon-Yi Previn. Picture: Getty

In time, Allen will raise the curtain on this particular act, starring Emma Stone and Joaquin Phoenix. But for now, the trickster is keeping it hidden, emerging from his protective barriers only to talk about another film with Stone, Magic in the Moonlight.

Written and directed by Allen, this comedy tells the tale of a 1920s stage magician (Colin Firth) whose sober belief in the empirical world is sorely tested by an enticing younger woman (Stone) who claims to be a psychic medium.

Issues of artifice and uncertainty are pervasive in Allen’s work and life. Magic in the Moonlight is the latest of his films to exhibit his fascination with the early 20th century and to offer a philosophical arena where the forces of rationality and spirituality can battle it out, although it is no secret which side the author favours.

“I’m like Blanche DuBois,” says Allen, 78, sitting in a car park outside the cafe. “I hope in life that there’s a certain amount of magic. Unfortunately, there’s not enough. There are little, sporadic things one could think of as magical. But for the most part, it’s grim reality.”

Those remarks may simply reflect the comically crushing scepticism of Allen, who has considerable power to construct his own worlds, fictional and otherwise.

But the comments could also apply to the tumult of recent months, when he was confronted with past accusations he may have thought had vanished; and the reality that Magic in the Moonlight will test whether his audience has disappeared.

Magic has appeared in his movies in various guises, whether overtly, as in films such as Scoop (the 2006 movie in which he played a fumbling would-be Houdini), or obliquely, as in the 1985 comedy The Purple Rose of Cairo (in which a movie character steps off a screen into real life).

As an adolescent, Allen was obsessed with stage magic and taught himself card tricks and sleight-of-hand stunts, even performing them for small audiences. But, given what he calls “my inherent criminal personality,” Allen says, “I was interested in being a gambler, a card hustler, a dice hustler”.

Letty Aronson, Allen’s sister and longtime producer, says that her brother’s early interest in prestidigitation was a sign that he was “very, very observant of small things that happen in everyday life”.

As a film-maker, Allen has been a vocal disbeliever in a world beyond what is perceivable, as certain that there is no foundation for it as he is that such faith could never take root in him, although he holds a certain fascination with those who possess it.

“But if you’re the kind of person that finds it hard to deceive yourself – even though it’s seductive to believe the other thing – then you’re stuck with it,” he says. “The overwhelming amount of logic and evidence is that we’re all victims of a bad deal.”

The trials Firth’s character undergoes in Magic in the Moonlight – he yearns to find that Stone’s psychic is not a fake and even resorts to prayer in one desperate moment – might seem to indicate that Allen is more flexible about his beliefs than he is willing to acknowledge.

“Woody must at least understand that certainty is to be questioned,” Firth says. “Sometimes there are more paradoxes in good writing that are revealed about the person than they might consider.”

As further evidence, Firth cites a favourite line from Allen’s short story The Condemned, a pastiche of Camus existentialism: “Cloquet hated reality but realised it was still the only place to get a good steak.”

Firth says: “You can actually read a lot into that. You can escape and escape, but there are things in the world, hard facts and cause and effect, that you cannot deny.”

A streak of escapism might also account for Allen’s frequent return to pleasant depictions of the pre-Second World War decades, the setting not only of Magic but also of Radio Days, Bullets Over Broadway, Zelig and others.

Allen says this seeming preference is more about pragmatism – the most logical time when these stories would take place – and says he is not a nostalgic person.

“Nostalgia is a trap,” he adds. “It’s a pleasant, sticky substance, like honey, that you fall into.”

These fanciful renderings of bygone eras are surely more comfortable than the period this past winter, when Allen was publicly challenged by his adoptive daughter Dylan Farrow, who said in the New York Times and elsewhere that Allen sexually molested her when she was a child.

In the NYT in February, Allen responded, “Of course, I did not molest Dylan,” pointing out that a police investigation had cleared him at the time and adding that the piece would be his “final word on this entire matter”.

Allen dismisses the possibility that lingering outrage could affect the public’s interest in Magic in the Moonlight.

“No thoughts like that occur to me,” he says. “They only occur to you guys [the press].

“I don’t think anyone has ever not come to a film of mine that they thought they would enjoy,” he adds. “Nothing keeps them away if they think they’ll enjoy the film. And if they don’t think they’ll enjoy the film, nothing we can do ever brings them in.”

Professionally, Allen still enjoys wide latitude to make movies as he wishes. His last three directorial efforts, Midnight in Paris (which sold $151 million in tickets worldwide), To Rome With Love ($73 million) and Blue Jasmine ($97 million), are among his most lucrative movies ever, and Blue Jasmine earned its lead actress, Cate Blanchett, an Academy Award.

Allen says that his next four films, including the currently untitled one he is making with Stone and Phoenix, have financing, and he has the full support of Sony Pictures Classics, which has released his recent films, including Magic in the Moonlight.

Thoughts of winding down his career do not occur to Allen, who describes his work as if it were a mixture of fantasy and vacation.

“You go in, in the morning, and there are women like Scarlett Johansson and Emma Stone that you spend months with,” he says. “They’re charming, they’re beautiful, they’re gifted. The guys, like Colin, like Joaquin, are larger than life. They’re delightful.”

Should he simply one day run out of ideas, Allen suggests, he could still fall back on his old sleight-of-hand skills. “I could probably cheat some old ladies out of their pensions, if I had to,” he says.

• Magic in the Moonlight is on general release from 19 September.