In a speech that pinned the blame on his party rather than his own failings, Mr Johnson repeated how big his mandate was but admitted time was up.
He will remain as Prime Minister until a successor is in place, expected to be by the time of the Conservative Party conference in October.
Mr Johnson said it is “clearly now the will” of the parliamentary Conservative Party that there should be a new leader.
Announcing his resignation in Downing Street, he said: “It is clearly now the will of the parliamentary Conservative Party that there should be a new leader of that party and therefore a new prime minister.
“And I’ve agreed with Sir Graham Brady, the chairman of our backbench MPs, that the process of choosing that new leader should begin now and the timetable will be announced next week.
“And I’ve today appointed a Cabinet to serve, as I will, until a new leader is in place.”
The Prime Minister criticised the “herd instinct” in Westminster, and said he felt it was his “obligation” to continue to do what the Tories promised in 2019.
He said: “I want to say to the millions of people who voted for us in 2019, many of them voting Conservative for the first time, thank you for that incredible mandate, the biggest Conservative majority since 1987, the biggest share of the vote since 1979.
“And the reason I have fought so hard in the last few days to continue to deliver that mandate in person was not just because I wanted to do so, but because I felt it was my job, my duty, my obligation to you to continue to do what we promised in 2019.
“And of course, I’m immensely proud of the achievements of this Government.”
Some Tory MPs now want him gone as soon as today, thinking his behaviour means he is unfit to hold the office even a day longer.
A No 10 source said Mr Johnson spoke to Sir Graham Brady, chairman of the Conservative 1922 Committee, to inform him of his decision.
They said: “The Prime Minister has spoken to Graham Brady and agreed to stand down in time for a new leader to be in place by the conference in October.”
It comes after around a third of all MPs who held non-Cabinet ministerial positions at the start of the week resigned.
The Prime Minister then thanked the British public as he concluded his speech.
He said: “Above all, I want to thank you, the British public, for the immense privilege that you have given me and I want you to know that from now on until the new prime minister is in place, your interests will be served and the Government of the country will carry on.
“Being Prime Minister is an education in itself. I have travelled to every part of the United Kingdom and in addition to the beauty of our natural world I have found so many people possessed of such boundless British originality and so willing to tackle old problems in news ways that I know that even if things seem dark now, our future together is golden.
“Thank you all very much. Thank you.”
Labour leader Sir Keir Starmer said the decision was “good news”.
He continued: “He was always unfit for office. He has been responsible for lies, scandal and fraud on an industrial scale.
“And all those who have been complicit should be utterly ashamed.
“The Tory party have inflicted chaos upon the country during the worst cost-of-living crisis in decades. And they cannot now pretend they are the ones to sort it out.
“They have been in power for 12 years. The damage they have done is profound.
“Twelve years of economic stagnation. Twelve years of declining public services. Twelve years of empty promises.
“Enough is enough. We don’t need to change the Tory at the top – we need a proper change of government.
“We need a fresh start for Britain.”
Responding, Nicola Sturgeon claimed the decision would be welcomed across the country.
She said: “There will be a widespread sense of relief that the chaos of the last few days (indeed months) will come to an end, though notion of Boris Johnson staying on as PM until autumn seems far from ideal, and surely not sustainable?
"Boris Johnson was always manifestly unfit to be PM and the Tories should never have elected him leader or sustained him in office for as long as they have. But the problems run much deeper than one individual. The Westminster system is broken.
"My differences with Boris Johnson are many and profound. But leadership is difficult and brings with it many stresses and strains, and so on a personal level I wish him and his family well.”