Only 19.9 percent of respondents to the poll by Kyodo News agency said they approved of Naoto Kan's administration, a plunge of about 12 percentage points from a month ago, when Kan reshuffled his Cabinet.
Almost two-thirds said they disapproved of the government. Many cited a lack of leadership by Kan, who also heads the ruling Democratic Party of Japan, Kyodo said.
The results raise renewed questions about Kan's political longevity. A sharp erosion of voter confidence has been fatal for a series of recent prime ministers. Kan is Japan's fifth leader in less than four years.
Kyodo said the last time support for an administration fell under 20 percent was shortly before former Prime Minister Yukio Hatoyama resigned last June.
The rapid turnover has made it harder for the country to formulate and implement policies to tackle its many problems, including a lackluster economy, a rapidly aging population and a ballooning public debt.
The poll results also reflect a recent scandal within the ruling party.
The Democrats have been deeply divided over what to do about veteran lawmaker Ichiro Ozawa, who was indicted last week in a funding scandal. He rejected calls by Kan to resign from the party.
Party executives are expected to impose a punishment next week. But that could further deepen the intraparty rift as the government tries to pass a national budget and implement economic reforms against strong opposition.
Voters appear to be conflicted as well. Some 53 percent of respondents in the Kyodo survey said they wanted Ozawa to resign from parliament, but only about a quarter wanted him to leave the Democratic Party.
Ozawa remains a powerful political figure. He is credited with engineering the Democrats' rise to power two years ago and challenged Kan for the leadership post last year. He has denied he violated political funding laws by falsifying records relating to a 2004 Tokyo land deal.
Overall support for the Democrats in the poll slipped to 20.9 percent from 22.7 percent last month, Kyodo said. Approval for their main opposition, the Liberal Democrats, was slightly higher at 23.7 percent.
Kyodo said it surveyed 1,013 randomly selected eligible voters by telephone on Friday and Saturday. It did not give a margin of error. Polls of that size would generally have a margin of error of about 4 percentage points.