Mitt Romney was caught on video making the comments to a closed-door fundraiser held by wealthy Republican donors in Florida in May, The New York Times reported.
The reception was secretly recorded and Romney's comments brought to light by Mother Jones.
"Of course individuals are going to take responsibility for their life, and my campaign is about helping people take more responsibility and becoming employed again — particularly those who don't have work," Romney said at the Segerstrom Center for the Arts in Costa Mesa, Calif.
He added that as a candidate for the White House, "my job is not to worry about those people. I'll never convince them they should take personal responsibility and care for their lives."
About 46 percent of Americans owed no federal income tax in 2011, according to the Associated Press.
The video shows Romney saying that for those who pay no income tax, "our message of low taxes doesn’t connect."
Democrats have said the comments proved that Romney had "disdainfully written off half the nation" in his quest to become President as the campaign enters its final 50 days.
However, at a news conference in LA on Monday, Romney conceded only that his comments were spoken "off the cuff" and not "elegantly stated."
"Of course, I want to help all Americans, all Americans, have a bright and prosperous future," Romney told reporters, the AP wrote.
While he did not dispute the authenticity of the hidden-camera footage, he called for the release of the full video to provide "context" to his comments.
He did not apologize when a reporter asked if he was concerned that he may have offended people.
The Times wrote that the video threatened to sidetrack Romney’s campaign as a time when he was trying to "restart his campaign with new ads and new messaging, in response to calls in his campaign and from outside for him to be more specific about how his policies would fix the nation’s economy and help the middle class."