Delegates passed the resolution by 25-20 votes at the commission's annual meeting in Berlin after a day of debate.
The decision highlighted a split between Japan, Norway and Iceland, which opposed the conservation measure, and the United States, Mexico, Britain, Australia and Germany, which sought increased protection for mammals once hunted to the brink of extinction.
The 31-page proposal, dubbed the Berlin Initiative, calls for forming a committee within the 50-nation IWC to work with wildlife groups and bolster efforts to protect the marine mammals.
Supporters say it would be the commission's most decisive action since it imposed a global ban on commercial whaling in 1986.
"This initiative marks a milestone in the history of the International Whaling Commission," Australian envoy Conall O'Connell said, adding that it would lead to "a strengthened commission with whale conservation at its heart."
Japanese officials earlier called for an end to the whaling ban and threatened to walk out of the meeting if the conservation measure passed.
It was not immediately clear if Japan would follow through with its threat.
Shiro Asano, the governor of Japan's Miyage prefecture, told the meeting that "commercial whaling should be resumed ... without further procrastination."
Norway's chief envoy, Odd Gunnar Skagestad, called the conservation proposal "inappropriate and unfortunate."
"We fear that this proposal will turn out to be yet another divisive issue which the commission could well do without," he said.
A number of Caribbean and African nations also opposed the increased efforts at preservation, which won the support of the United States, other European nations and South Africa.
"What we want is to strengthen the conservation agenda, no more," said German delegation leader Peter Bradhering. "It's not directed against anyone."
Japan and other pro-whaling nations have often found themselves at odds with the majority on the International Whaling Commission. Norway has ignored the global ban.
Japan says it will also ask the IWC in Berlin for a special quota on minke whales in its coastal waters.
The country kills hundreds of whales annually under an IWC exemption for limited "research" hunts. Tokyo says the hunts help gauge the impact of whale herds on fisheries stocks and provide data on their migration patterns and population trends.
Critics call the program commercial whaling in disguise because the meat from the slaughtered whales is sold in Japanese restaurants.
Iceland also is expected to seek permission to kill whales in the name of scientific research.
"The United States is seriously concerned that the safety net that has been provided by the commercial whaling moratorium is under threat," U.S. delegation head Rolland Schmitten told reporters. "Now a second country wants to exploit a loophole in the treaty."
"The United States wants lethal scientific whaling to stop," he said. "Whales do not have to be killed to be studied."
A new study unveiled before the IWC meeting suggests that accidental captures may be the biggest immediate threat to whales — even more than ship collisions and pollution.
More than 300,000 whales, dolphins and porpoises are believed to die unintentionally every year in fishermen's hauls, according to the study by American and Scottish biologists published by the World Wildlife Fund.
By Geir Moulson