The decision to take legal action against Australia's important trading partner underlines the government's "commitment to bring to an end Japan's program of so-called scientific whaling" in the southern seas, Environment Minister Peter Garrett and Attorney-General Robert McClelland said in a joint statement.
Japan gets around an international ban on commercial whaling by arguing that it harpoons hundreds of whales each year for scientific research.
Australia's announcement came after the International Whaling Commission chairman Cristian Maquieira on Thursday expressed optimism that nations gathering in Morocco next month can settle a long-running dispute over the hunting of whales.
But senior U.S. official Monica Medina signaled difficult negotiations ahead over a contentious IWC proposal that would effectively allow commercial whaling for the first time in 25 years, though under strict quotas set by the commission.
Australia said it had enjoyed support from other IWC members, including the European Union and Latin American countries who shared its goals and concerns.
"But to date, the response of the whaling countries has not been positive," the joint statement said.
"Recent statements by whaling countries in the commission have provided Australia with little cause for hope that our serious commitment to conservation of the world's whales will be reflected in any potential IWC compromise agreement," it added.
Japan, Norway and Iceland, which harpoon around 2,000 whales annually, argue that many species are abundant enough to continue hunting them. They are backed by around half the IWC's 88 member nations.
Australia has declared the southern seas a whale sanctuary and has long lobbied for an end to whaling there.
Prime Minister Kevin Rudd's center-left Labor Party was elected to government in 2007 on a promise to take Japan to court to end what Australia argues is commercial whaling in disguise.
But after winning government, Rudd announced that Australia would focus on diplomatic efforts to persuade Japan to end its annual whale hunt.
The announcement on Friday delivers on Labor's promise months from an election on a date to be announced late this year.