The religion isn't officially recognized in Jamaica, which means marriages can't be legally performed and places of worship aren't tax exempt. Rastas also say their traditional appearance and ritual use of marijuana has kept them from getting decent jobs.
"They put a wire fence around us so that when we reach for opportunities we get scraped," said Pato Solo, a 33-year-old attending the International Rastafari Conference, which begins Wednesday and is drawing followers from the Caribbean, Europe and Africa.
Fueled by anger over the colonial oppression of blacks, descendants of African slaves started the religion in Jamaica during the 1930s.
Its message of peaceful coexistence, marijuana use and African repatriation was popularized in the 1970s through reggae artists such as Bob Marley and Peter Tosh. An estimated 700,000 people practice the faith worldwide but the government of Jamaica can't say with certainty how many of its 2.6 million people are Rastas.
"We can't walk free, we can't talk free ... We're still under bondage," said Radcliffe Boyd, 33, who claims he had his wrist broken by police three years ago after being caught with marijuana.
He paid a small fine, as do most Rastas arrested for marijuana use. Penalties range from fines of less than $2 to six months in jail.
Rastafarians say they're blamed for crime in Jamaica and looked down upon for their use of marijuana, which followers believe brings them closer to God. Some sects believe their god is deceased Ethiopian Emperor Haile Selassie.
Few Rastas are employed in the formal sector in Jamaica. Officials say they don't discriminate, but admit they would rather hire someone who is clean-cut and sober.
Lansford Haughton, 51, said he was refused work at a post office: "They said, 'you have to groom yourself,"' he said. "I told them, 'This is my faith."' One of the religion's tenets is staying close to nature, which can mean not combing or cutting one's hair.
Police spokeswoman Ionie Ramsay-Nelson raised another question: "It's a safety issue. Sometimes we have to use firearms and if somebody takes a smoke they might just do whatever their mind tells them to and start shooting. It's a concern."
Rastas say using the drug doesn't affect their performance.
"Ganja makes me focus more, said Robert Gardner, 32, saying marijuana users are not "like crack addicts or alcoholics."
The treatment of Rastas isn't much better elsewhere in the Caribbean.
The British Virgin Islands in 1980 banned Rastas from setting foot in the territory. Several groups are trying to repeal the law. It's still on the books, though Rasta visitors are not turned away but complain of harassment at the airport.
In Grenada last year, prison officials told four Rastafarian inmates they would have to cut their dreadlocks to prevent disease and the smuggling of contraband. A court challenge is pending.
Discrimination hasn't stopped some from cashing in on the faith, whose colors of red, green and gold earn millions in sales of T-shirts, shoes and the crocheted tams into which they tuck their hair.
Television ads woo tourists with white-sand beaches and friendly Rastas to the rhythm of the famous Marley anthem "One Love."
"It's a mockery. They use us like a style, a fashion," said Solo, who plans to bring up his concerns at the conference.
One invited guest who won't be attending is Jamaican Prime Minister P.J. Patterson. He declined because of a scheduling conflict, said his spokesman Huntley Medley.
Howard Hamilton, Jamaica's public defender, said it's time for the country to fully recognize it's homegrown religion, including the use of marijuana as a sacrament.
"They have paid their dues," said Hamilton.
Politicians have promised to explore decriminalizing marijuana. Rastas aren't holding their breath, so to speak.
"America, England, Japan, that's where we get respect, not Jamaica," said Boyd, grinding bits of the lime-green herb. "Jamaicans don't want to know about their culture."