The 80-year-old Swiss, Fifa president from 1998 to 2015, lost his appeal to the Court of Arbitration for Sport against his the suspension from football-related activity.
He is banned until October 2021, when he will be 85. He must also pay a fine of 50,000 Swiss francs (around £39,000).
Blatter and former Uefa president Michel Platini were punished in December 2015 over a £1.3 million “disloyal payment” made to Platini in 2011.
Both men have protested their innocence throughout, claiming the payment was made for consultancy work the Frenchman had carried out for Blatter between 1998 and 2002, and that they had a “gentleman’s agreement” on when the balance was to be settled.
Blatter described the CAS ruling as “incomprehensible”, but appears to have now drawn a line on his lifetime in the game.
“I take note of the verdict of the Court of Arbitration for Sport,” Blatter said in a statement.
“The way the case progressed, no other verdict could be expected.
“What I find incomprehensible is that the existence of the oral agreement between Fifa and Michel Platini is still steadfastly negated in spite of my testimony to the contrary and the testimony given by other witnesses. A Uefa protocol mentioning the agreement is also disregarded.
“I have experienced much in my 41 years in Fifa. I mostly learned that you can win in sport, but you can also lose.
“In this sense I have to accept this decision (my suspension as Fifa president), although it is difficult to follow it, because the principle of jurisdiction – culpability has to be proven by prosecution – was not applied.
“Nevertheless I look back with gratitude to all the years in which I was able to realise my ideals for football and serve Fifa.”
Fifa acknowledged the verdict in a one-line statement.
A Fifa spokesperson said: “Fifa has taken note of the decision rendered today by the Court of Arbitration for Sport (CAS).”
Blatter and Platini’s suspensions were reduced from eight to six years by a Fifa appeals committee in February, with their services to football a mitigating factor.
Platini took his case to CAS and Blatter followed suit. Platini in May had his suspension reduced by a further two years to four years, but Blatter’s full sanction remains in place.
CAS said Blatter had requested his ban be revoked, not reduced, although the three-person panel determined the sanction was proportionate.
The panel, which heard the case on 26 August, ruled the payment made to Platini in 2011 breached the Fifa code of ethics as an “undue gift” and had “no contractual basis”.
A CAS statement read: “The appeal of Joseph S Blatter has been dismissed.
“The panel found that the written employment contract established between Mr Platini and Fifa in 1999 voided any oral agreement concluded between Mr Blatter and Mr Platini in 1998 that the latter would be paid CHF 1 million per year for his work at Fifa.
“Accordingly, by approving a payment of CHF 2m to Mr Platini in 2011 for the balance of work carried out under the alleged oral agreement, Mr Blatter breached the Fifa code of ethics since the payment amounted to an undue gift as it had no contractual basis.
“The panel further found that Mr Blatter unlawfully awarded contributions to Mr Platini under the Fifa Executive Committee retirement scheme which also amounted to an undue gift.
“Turning to the sanction, the panel noted that Mr Blatter requested the annulment of the ban but did not request a reduced penalty.
“In any event, the panel determined that the sanction imposed was not disproportionate and therefore confirmed the appealed decision in full.”