Calling themselves "Shinobi warriors," the men wore black SWAT-type vests and carried knives, throwing stars, swords, nunchucks and a bow and arrows.
Twenty-year-old Tadeusz Tertkiewicz and 19-year-old Jesse Trojaniak are charged with weapons possession. Tertkiewicz is also charged with harassment for a letter left for an ex-girlfriend.
After being arrested early Wednesday in a car on Route 46, the men said they were delivering warning letters to drug dealers and drug users urging them to stop their "impure" activities.
Their weapons, Detective Capt. Robert Rowan told The Bergen County Record, were to be a precaution in case they were confronted by the drug dealers.
The officers who arrested the men found the letters in five envelopes decorated with red Chinese designs, The Record reports.
Reports show the two men had already delivered one such letter to Tertkiewicz' 16-year-old ex-girlfriend in Clifton. Police contacted the teen and her mother and alerted them to the situation.
The letters warned drug users and drug dealers that the "Shinobi will stop your cruel and sadistic intentions with justified yet, merciful force."
In the letter, the two men accused drug dealers and users of having "committed sin of passing impurity" to others and that the "wind guides us to those of impure heart and intent."
"Their intentions may have been good, but we tell everyone that they shouldn't take the law into their own hands because it will cause more problems for everyone in the longrun," Rowan said.
Trojaniak's father tells The Associated Press his son is "not a maniac" and was only trying to help a friend.