There was no mention of any formal collaboration, but the pair “pledged to stay in touch, particularly if opportunities to work together on shared interests arise in the future," Politico quoted the White House as saying in a statement.
A photo posted to the White House Flickr account shows Romney and Obama, both in suits, shaking hands in the Oval Office following the lunch.
Romney arrived at the White House in a black SUV, walking in to the West Wing alone, and left after just over an hour, The Associated Press reported.
There was no formal agenda, but the president did want to discuss Romney's ideas for making government more efficient, White House spokesman Jay Carney told reporters.
"The president noted that Gov. Romney did a terrific job running the Olympics and that that skills set lends itself to ideas that could make the federal government work better, which is a passion of the president's," CNN quoted Carney as saying.
The meeting fulfills a promise Obama made in his victory speech on Election Night.
Earlier Thursday, Romney had breakfast with his running mate, Wisconsin Rep. Paul Ryan, at a hotel near the White House.
“I remain grateful to Governor Romney for the honor of joining his ticket this fall, and I cherish our friendship,” Ryan said in a statement released afterward. “I’m proud of the principles and ideas we advanced during the campaign and the commitment we share to expanding opportunity and promoting economic security for American families.”