Mayor Iccho Ito was shot twice late Tuesday outside the city's main train station by a gangster reportedly disgruntled over damage to his car at a public works site, according to Nagasaki prefectural (state) police official Rumi Tsujimoto.
Kyodo News agency and national broadcaster NHK said Ito died of his wounds early Wednesday.
Footage from the southwestern city of Nagasaki showed several officers wrestling a man to the ground and pushing him into a police car, and an ambulance leaving the scene as police shouted for bystanders to get out of the way.
Tetsuya Shiroo, a senior member of Yamaguchi-gumi, Japan's largest organized crime group, was arrested on the spot for attempted murder, police said.
Shiroo has admitted to shooting Ito with a handgun with the intent to kill, Nagasaki chief investigator Kazuki Umebayashi said in a televised news conference.
One of the bullets reached the heart of the 61-year-old mayor, who went into cardiac arrest, according to hospital official Kenzo Kusano. He underwent emergency surgery and was on life support, but reports said he later died.
It was the second attack in recent years against a mayor of Nagasaki, which was destroyed by a U.S. atomic bomb in 1945 and whose leaders have actively campaigned against militarism and nuclear proliferation.
In 1990, former Nagasaki Mayor Hitoshi Motoshima was shot and seriously wounded after saying Japan's emperor, beloved by rightists, bore some responsibility for World War II.
Tuesday's attack appeared to involve a more trivial matter, however.
Shiroo reportedly clashed with Nagasaki city over a traffic accident in 2003, when his car was damaged after he drove into a hole in the ground at a public works construction site.
The gangster tried unsuccessfully to get compensation for the damage after his insurance policy refused to pay up, according to NHK.
Shiroo had also sent a letter to broadcaster TV Asahi to protest recent money scandals linked to Ito, including hidden accounts and public works contracts, according to Kyodo.
Ito, backed by the ruling Liberal Democratic Party, was campaigning for his fourth term. He was an active figure in the movement against nuclear proliferation, heading a coalition of Japanese cities calling for the elimination of nuclear weapons.
Peace activists expressed shock.
"Mayor Ito had a strong and boundless passion for peace," said Sunao Tsuboi, leader of a survivors' group based in Hiroshima, a city also flattened by an atomic bomb in 1945.
"The authorities must conduct a rigorous investigation and get to the bottom of this," Prime Minister Shinzo Abe said in a statement late Tuesday.
Commonly known as yakuza, Japan's organized crime groups are involved in real estate and construction kickback schemes, extortion, gambling, the sex industry, gunrunning and drug trafficking.
The yakuza are also behind most shootings in Japan, where handguns are strictly banned. Two-thirds of the country's 53 known shootings in 2006 were gang-related, according to the National Police Agency. Police estimate that there are about 84,500 gangsters across Japan.
The yakuza have had a long-standing political alliance with right-wing nationalists, though authorities gave no indication that Tuesday's attack was politically motivated.
Though attacks against politicians are uncommon in Japan, three have been killed since World War II, including socialist leader Inejiro Asanuma, who was stabbed to death by a right-wing activist at a rally in Tokyo.
Another opposition lawmaker was killed by a mentally unstable assailant in 1990, while a ruling party politician was fatally stabbed in 2002 in a dispute over political funds.
Last year, a right-wing extremist burned down the house of ruling party lawmaker Koichi Kato after the politician criticized a prime minister's pilgrimage to a controversial Tokyo war shrine.