The Center for Science in the Public Interest and the company confirmed the move Monday.
The CSPI told the company last month it should not use "all natural" if products contain alkalized cocoa, corn syrup, hydrogenated oil or other ingredients that are not natural.
Ben & Jerry's, a unit of consumer products giant Unilever, said it's not changing any recipes. It's just removing the label from all products.
The Food & Drug Administration has no formal definition for "natural." But it won't object to term as long as products do not contain added color, artificial flavors or synthetic substances.
The CSPI, based in Washington, said the government should define the term.
"The Food and Drug Administration could do consumers and food manufacturers a great service by actually defining when the word 'natural' can and cannot be used to characterize a given ingredient," CSPI Executive Director Michael F. Jacobson said in a statement.
Ben & Jerry's spokesman Sean Greenwood said the change would be phased in gradually across the Ben & Jerry's product line.