Mr Raab has defied calls to quit over his widely-criticised decision, but said "of course with the benefit of hindsight I wouldn't have gone away".
"The stuff about me being lounging around on the beach all day is just nonsense," he told Sky News.
"The stuff about me paddleboarding, nonsense, the sea was actually closed, it was a red notice."
He also said the UK will use "every hour" left to evacuate people from Afghanistan as he declined to rule out British troops having to leave by the end of Friday.
US President Joe Biden rejected calls from Prime Minister Boris Johnson and other allies to delay his August 31 exit date, citing the heightened security risks to troops.
The move means the clock is quickly running down on the UK evacuation operation at Kabul airport, with British troops expected to have to leave ahead of their American counterparts.
Mr Raab said Britain is working "as fast as we can" to maximise the number of people who can flee, saying 2,000 have been evacuated in the previous 24 hours.
He declined to state when the last British flight will leave Kabul, amid suggestions that the UK operation will have to end as soon as Friday.
"The military planners are working through the limited time they need to draw down their personnel and equipment and so they will firm up those details," he told BBC Radio 4's Today programme when asked about that date.
"We will use every hour and day we've got to maximise that throughput to get as many of those residual cases out.
"We're going to keep going for every day and every hour that we've got left."
The Prime Minister had hoped to persuade Mr Biden to maintain his forces on the ground to allow the evacuation effort more time during an emergency meeting of G7 leaders on Tuesday.
But the US President said staying longer would raise the risk of attack by so-called Islamic State (IS) affiliates and straining a "tenuous" working relationship with the Taliban.
Having swept to power last week in the wake of America's major withdrawal of troops, the Taliban has warned evacuations "will not be allowed" after August 31.
The group suggested foreign forces remaining past the deadline would cross a "red line" that will "provoke a reaction".
Ministry of Defence figures put the number of people evacuated by the UK since August 13 at 9,226, but there are thousands feared to be remaining.
The MoD declined to comment on reports in the Guardian, which cite defence sources, that the evacuation could end within 24 to 36 hours.
The newspaper said that the US military requires two to three days to close its operations in Kabul, and that British troops aim to be at least 24 hours ahead of that, with the Americans providing security at their airport.
Mr Johnson said after the virtual summit that leaders had agreed the "number one condition" up to and after the deadline was that the Taliban must grant "safe passage for those who want to come out".