The 40 percent mark is a new low for the president. Quinnipiac asked the same question in surveys in September and October.
As for his approval rating, voters disapprove 58 to 35 percent of the job Trump is doing, near his lowest score, a 61 to 33 percent disapproval in an August 2 poll.
On "Happening Now," pollster Chris Wilson said the numbers don't look good, but he called the fitness to serve question a "red herring."
"You've seen for a long time, most Republicans like the job Donald Trump is doing, most Democrats don't like the job he's doing. And you can ask any sort of question and you're going to get those same kind of numbers, whether it's asking fitness if or [asking] if you dislike Donald Trump as president," Wilson said.
He said these numbers are more representative of the "Twitter outrage" at Trump, as opposed to his actual fitness to serve.
Democratic strategist Scott Bolden disagreed and said the fitness and approval ratings numbers are important, and Republicans need Trump to improve both of them heading into elections in 2018 and 2020.
"The question is: How does the president turn it around?" Bolden said.
Wilson pointed out that when pollsters ask specifically about Trump's policies and remove his name from the question, there's a good indication that the American voters actually like what he's accomplishing.
"But when you start dealing with his persona, that's what drives the numbers down overall," Wilson said.
Watch more above.