The Labour leader suggested the Conservatives were trying to distract from the cost-of-living crisis by making the accusation the two opposition parties were standing aside for each other in certain local election contests.
Mr Dowden has written to Sir Keir asking him to explain why Labour is standing fewer candidates in parts of southern England than it did in 2018.
He claimed the move was “far too substantial to be a mere coincidence”, as the Lib Dems were “returning the favour” in the North.
Asked about the claims, the Labour leader told Sky News: “I wouldn’t take anything Oliver Dowden says particularly seriously.
“The fact that he is spending his Sunday … attacking Labour … why doesn’t he say something about the cost-of-living crisis for heaven’s sake?
“There is no pact, everybody knows there is no pact. We will put a candidate up when there is a by-election, obviously after the Neil Parish resignation of yesterday.”
Sir Keir also claimed Labour was standing more candidates than ever in the local elections.
Sir Ed later told the BBC: “There isn’t a pact, there is not going to be a pact.
“In fact if you look at what we are doing in these local elections, we are fighting Labour in many areas, in Hull, in Sunderland, in Sheffield, in Haringey, in Southwark, I could go on.
“This is pretty desperate from the Conservatives. I am not surprised because lifelong Conservatives are switching away from them because they are really upset that we have a Prime Minister who is not decent to run our country.”
Meanwhile, Sir Keir said it was “very valuable” to have the endorsement of Labour’s former leader Sir Tony Blair.
Sir Tony has appeared in a campaign video for the Labour Party ahead of the local elections.
Sir Keir told Sky News: “Tony Blair won three elections and the Labour Party doesn’t win elections very often, so I think having the endorsement of Tony Blair is very important, very valuable and I think further demonstrates the amount of work we have done in the last two years, the change that we have made.
“I am very happy to talk to and with someone who has won three elections because I didn’t come into politics to be in opposition, I came into politics to win elections.”