The red-hot favourite had led 9-5 but blew that advantage as Selby came back to win 13-11, coming from a perilous position just as he did in the finals of the 2008 Welsh Open and this year's Masters, when O'Sullivan had victory in his sights on both occasions.
"You get to know your own engine. You're sitting there thinking that anything can happen but inside you know how it's going to pan out," said O'Sullivan.
The 34-year-old insisted he would not give the game up, even though he clearly did not enjoy his stay in Sheffield this year. "I love the game and it's been great to me. I'll always be involved in snooker because it's in my blood but I don't need it," he said.
O'Sullivan had been two frames from victory at 11-9 but his potting form stalled and Selby took full advantage to set up a semi-final clash with Graeme Dott, the 2006 champion.
Both semi-finals were due to begin today.
Selby had been still confident at 9-5 behind. "I just concentrate on my own game and my own game seems to be good enough to beat him at the moment," he said.
Dott had a similar win, beating Mark Allen 13-12 from 12-10 behind.
Glasgow's Dott, 32, should pose Selby plenty of problems after impressing so far.
The 2006 champion had dropped to 48th in the provisional rankings in late 2008 after a period of personal strife which saw him diagnosed with depression, but now stands provisionally 14th for next season and could climb higher.
Ali Carter won another final-frame thriller late last night, with the 2008 runner-up knocking out 2005 champion Shaun Murphy 13-12.
Steve Davis made his exit, losing 13-5 against Neil Robertson, who takes on Carter next.