Only 10 percent of people surveyed by the University of Michigan said they would have no concerns at all about riding in fully self-driving cars, while two thirds said they would be moderately or very concerned about it.
The research shows that while coverage of self-driving cars and the technology behind them has increased in the media, consumers are still unsure that the cars are safe.
The rest want the manual controls to stay.
Brandon Schoettle, project manager of the university's sustainable worldwide transportation program and lead author of the study, said it doesn't necessarily make sense for people to be more wary of fully autonomous vehicles than of semi-autonomous ones.
"We feel that there are safety issues involved with handing control of a partially self-driving vehicle back to the human driver, so the greater concern people have for completely self-driving vehicles is somewhat misplaced in our opinion," he said in an email message.
Schoettle said he expects attitudes will change as consumers are introduced to the technology and try it out.