Rep. Paul Ryan (R-Wis.) was on “The O’Reilly Factor” tonight to respond to backlash over remarks he made on inner city culture.

“We have got this tailspin of culture in our inner cities, in particular of men not working and just generations of men not even thinking about working or learning the value and the culture of work, and so there’s a real culture problem here that has to be dealt with,” Ryan said on “Bill Bennett’s Morning in America” radio show.

Rep. Barbara Lee (D-Calif.) called these remarks a “thinly veiled racial attack.”

“There was nothing racial whatsoever in what I said, and if you listen to the full context of all of my remarks, that’s pretty clear,” Ryan told Bill O’Reilly. He said we should get beyond throwing baseless charges at one another and have a real conversation about poverty in the United States.

Ryan said he called Lee to make it clear that race had nothing to do with his comments.

“She knows me well, and she knows that I don’t have a racist bone in my body,” he said.

When asked why Lee implied that he’s racist, Ryan said, “You’d have to ask Barbara that.”

Ryan said he isn’t mad at the California congresswoman for her response to what he said on the radio show.

“I’m a big boy. I understand that if you challenge the status quo, that if you get into these issues, sometimes you’ll be misinterpreted,” he said.

The Wisconsin congressman said he wants to solve the problem of poverty across the nation because government programs aren’t working.

“Why don’t we go listen to people in poverty? Why don’t we go visit with the poor and see what’s working, what’s not, and if you go do that, as I’ve been doing all year long, you will find tremendously impressive and inspiring stories about people successfully fighting poverty and getting themselves out of it,” Ryan said, stressing that we should be focusing “not on servicing poverty, but on solving poverty.”

Ryan also said the House has passed a bill to address skills gaps so that people can get the training they need in order to build a good career.

Watch his full interview above.

‘Cultural Problem Across the Board’: Will Defends Paul Ryan Amid Race Controversy

Obama ‘Basically Raised the White Flag’: Paul Ryan Blasts 2015 Spending Plan