"I mean, why hate a guy who's suffered through that?" Letterman said Wednesday.
The joke came in response to Romney saying on a leaked video that "Letterman hates me because I've been on Leno more than him." Romney made the comments in the same video in which he told wealthy donors that 47 percent of Americans were dependent on the government. It was recorded in May and released by Mother Jones on Monday.
Romney has appeared on Letterman's show three times. President Obama appeared on the show Tuesday, and criticized Romney for the "47 percent" remarks.
Also read: Obama Boosts Letterman Ratings
Letterman said he hoped Romney would return to his show before election day, "now more than ever."
"I'm telling you, if you think you're gonna get to the White House, you've got to spend time in this chair," he said, referring to the seat for his guests.
Romney also critiqued "The View" on the video, saying four of its five hosts were "sharp-tongued and not conservative." But on Wednesday, the lone conservative on the show, Elisabeth Hasselbeck, announced that Romney had agreed to return to "The View" in October.
Watch Letterman's full take on Romney's remarks: