His screenwriting Oscar for Good Will Hunting, shared with Matt Damon, seemed a distant memory lost in the fog of bad movie choices – Gigli, Jersey Girl, Surviving Christmas, Paycheck – and lurid tabloid stories about his relationship with Jennifer Lopez.
Trapped in the glare of a media spotlight that he believed was damaging him personally and professionally, Affleck tried to disappear.
"I didn't like what happened to me and what happened to my life. The way it felt," he says. "I also felt that if I don't do anything, almost like don't show up, don't do an interview, talk to anybody, almost didn't leave my house except to go to Starbucks, that some of that would abate. It was very destructive and I hated it."
When he resurfaced at the Venice Film Festival two years later, in 2006, as the star of Hollywoodland, Affleck was a different man. He was married to actress Jennifer Garner and had recently become a father for the first time, giving him a stability he had hitherto lacked. When the festival closed, he walked away with the Volpi Cup for best actor, for his performance as tragic Superman actor George Reeves.
This month he returned to Venice as the director and star of The Town, his first time behind the camera since his 2007 directorial debut, Gone Baby Gone. Based on Chuck Hogan's novel Prince Of Thieves, the film takes place in the Charlestown district of Boston where bank robbery is practically a family business, passed from generation to generation.
Affleck plays the leader of a gang who starts to rethink the life handed down to him by his jailed father (Chris Cooper) when he becomes involved in a relationship with a woman (Rebecca Hall) taken hostage during a heist.
The Town shares certain thematic similarities with Good Will Hunting and Gone Baby Gone. "Chief among them is the idea that where we grow up really has a lot to do with shaping what we become," he says, "and the difficulty in escaping that, and the tension between those things...
"It's not easy to change oneself, even if we know we're doing the wrong things."
Affleck appears to have spent a lot of time trying to break away from inherited attitudes to life and work. As a struggling actor, he took any work that came his way.
His "Communist old man", a janitor with a "working-class ethic", had "always kind of beat into my head, maybe more than I needed," Affleck says. "This thing like, 'Do what you got to do. Take a job. Make money.'"
Consequently, the Bostonian actor wasn't snobbish about his choices; he just took "whatever came down the pike and was going to put food on the table". His father was no help financially, although he does seem to have provided his son with a vivid life lesson.
"He was in the tank," Affleck says. "He'd been living on the streets for two years and ended up going and getting cleaned up, getting off drugs and alcohol, and got a house. So I had a sense from that that living life requires sacrifice, and you got to do what you got to do. So I did that."
Over the years, his feelings towards acting went from love to dislike. He says at first he felt he was doing something interesting, and then it became about "money and perception, and the idea that you had to do a certain kind of movie that just never ended up being all that good".
He told himself that acting didn't matter after all, that "it's just all bullshit. It's all just a bunch of actors running around in tights.
I felt ashamed that that's what I did, because I didn't think it was real or socially relevant. I feel differently now. I'm not ashamed of it. For better or worse it's become extremely culturally relevant." (A fact his younger brother, Casey, has pounced upon with his hoax Joaquin Phoenix documentary, I'm Still Here.)
Eventually Affleck reached a point where he felt he should just "sock away some money, make hay while the sun shines". He was his father's son alright, although looking back he says, "I don't think the Socialist janitor work ethic applies to being in Pearl Harbor. I don't know how to reconcile the two of those things."
By the time he decided to withdraw, his bank account was in good shape; he was miserable and unfulfilled, however. Reflecting the theme of his films, he says that he started to reassess his life and to consider "not what my father thinks I should do or what other people think I should do, or agents or any of this other stuff, but really trying to go, 'What do I really want to do? What kind of life do I want to have? What kind of person am I really? What do I care about?'" The process led him to Hollywoodland and, presumably, Gone Baby Gone.
It was clear from his first stint in the director's chair that Affleck was an actor's director, an impression reinforced by the strong performances from the likes of The Hurt Locker's Jeremy Renner, Mad Men's Jon Hamm, Gossip Girl's Blake Lively and Rebecca Hall in The Town. Affleck, who recently replaced Christian Bale on Terrence Malick's new project, praised his cast in Venice, saying that when you have actors of their calibre, you don't really need to direct.
Inevitably, he has grown in confidence as a director.
The first time out, on Gone Baby Gone, Affleck didn't even know if he would cross the finishing line or not. He discovered that directing is "a truly bi-polar experience," he says. "It's terrible and wonderful all in the same five minutes. You feel, 'Oh my God, this is the greatest thing that has ever been done. This is an amazing moment' and then two minutes later it's, 'This is a fraud. This is a horrible, horrible movie.' I talked to other directors and they'd just nod. I'd say, 'Do you feel this way? Is this how it is?' and they'd say, 'Yes, kid.'"
Tough as it can be, he wants to continue being a film-maker. "I love it. It's thrilling. I hope I get the opportunity to keep doing it," he says. At 38, he has plenty of time to hone his craft. And he is determined to make the most of his time.
Older and more mature, the things that in his twenties would make him think "I want to have this kind of life or this is what being famous means" no longer hold their allure for Affleck. "You see that they're hollow and now I don't really care. I want to have a life that makes me happy and I want to have a career I'm proud of. As I've grown into the perspective I trust that I will continue in that. But I will have the press around to remind me."
The Town is in cinemas from Friday
This article was first published in Scotland On Sunday, 19 September, 2010