Olatunbosun Ugbogu, a 22-year-old Nigerian national serving in the U.S. Navy, was arrested on murder and robbery charges, a local police spokesman said.
He confessed to the murder and police plan to send him to prosecutors on Saturday for his indictment, the spokesman said.
Japanese officials reacted sharply to the arrest, which follows a furor over a series of criminal allegations against U.S. servicemen in Japan, where some 50,000 American troops are based.
The suspect, a crew member on the USS Cowpens, is accused of stabbing 61-year-old taxi driver Masaaki Takahashi on March 19 in Yokosuka, just south of Tokyo and the site of a large U.S. naval base. Police also accuse him of failing to pay the 19,560 yen (US$190) taxi fare.
The police spokesman, who spoke on condition of anonymity because of departmental rules, said the driver was stabbed in the shoulder with a kitchen knife, causing him to die from blood loss.
The suspect had been in U.S. custody since Navy authorities apprehended him in Tokyo on March 22 on an earlier desertion charge. He was handed over to Japanese authorities just before the arrest under a bilateral security pact, Japanese Foreign Ministry official Takashi Ariyoshi said.
Foreign Minister Masahiko Komura summoned U.S. Ambassador Thomas Schieffer on Thursday, expressing regret over the crime and urging U.S. officials to improve discipline of American servicemen and step up anti-crime measures.
Schieffer also met with Yokosuka Mayor Ryoichi Kabaya, who said he was "enraged" over the alleged crime by a U.S. sailor. "We demand the U.S. take concrete measures so that crimes like this are never repeated," he said.
U.S. Navy and Japanese authorities had questioned the sailor about the killing because a credit card in his name was allegedly found in the victim's car.
Japanese anger over the U.S. military presence has grown in recent months following an alleged rape in February of a 14-year-old girl by a U.S. Marine on the southern island of Okinawa, which sparked large protests there. Japanese prosecutors dropped charges against the Marine and released him after the girl withdrew her complaint, but the U.S. military is continuing its own investigation.
Other alcohol-related incidents have inflamed sentiment.
Earlier in Yokosuka, a Japanese court convicted a U.S. sailor of robbing and fatally beating a 56-year-old Japanese woman in 2006 and sentenced him to life in prison.
In an apparent move to quell the rising anger, the U.S. Navy imposed limits Wednesday on travel and alcohol consumption by personnel at the Yokosuka base.
The limits started immediately and were to last at least through Monday when they will be reviewed, said Cmdr. David Waterman, a Navy spokesman.