Commission members voted 29-23 against the proposal, which needed a three-fourths majority to pass. There were five abstentions, including countries that have often voted with Japan on other issues at the commission.
The 66-member commission, which regulates global whaling, banned commercial hunts in 1986, handing environmentalists a major victory in protecting the species near extinction after centuries of whaling.
Japan and other pro-whaling nations in the group knew they had virtually no chance of garnering the votes needed to overturn the moratorium at this year's annual meeting running through Friday.
But they had hoped to obtain a simple majority to back Japan's proposal for a nonbinding measure expressing support for limited commercial catches. That would have demonstrated that opinion among commission members has turned in favor of sanctioning commercial whale hunts.
Norway holds the world's only commercial whaling season in defiance of the ban. Japan, Norway and other nations which advocate what they call "sustainable use" of whales, this year are expected to kill more than 1,550 of the mammals.
On Monday, Japan said it would more than double its annual research cull of minke whales to as many as 935 from 440 this year, extending a researching whaling program begun in 1987. Critics call it commercial whaling in disguise.
Japan says it must kill whales to study them. It then sells the meat, which is allowed under commission rules. The United States criticized the decision to expand the research hunts, saying scientific advances make it unnecessary to kill whales to study them.
Japan maintains that whaling is a national tradition and that eating meat from mammals forms a vital part of its food culture. Japan claims whale stocks have sufficiently recovered since 1986 to allow the resumption of limited hunts.
Countries led by Australia and New Zealand reject that view. They advocate protecting whales and encouraging alternative ways of profiting from them, through tourism and whale-watching.
Japan's proposals on deleting the issue of whale sanctuaries from the meeting's agenda and introducing secret ballots were narrowly defeated Monday, but might have passed if several pro-whaling countries including the Pacific island nation of Nauru had arrived at the meeting on time.
The annual meeting went into a closed session for about 90 minutes Tuesday in what critics blasted as a delay tactic until more members who back the resumption of commercial whale hunts could arrive.
Members voted 28-20, with nine abstentions, in favor of the closed-door session as proposed by St. Lucia, one of several Caribbean countries allies with Japan in the pro-whaling camp.
Some participants evidently were frustrated that debate was being stalled; Brazil walked out of the closed meeting and accused pro-whaling countries of making long speeches on procedural issues.
"We just refuse to be taken hostage of rhetoric for unclear purposes," Brazil's representative Maria Teresa Mesquita Pessoa said.
By Kelly Olsen