Coast Guard crews recovered three people from the boat, one of whom is dead.
Coast Guard Petty Officer Chris McLaughlin said the 71-foot Lady Mary sank around 5 a.m.
Rescuers received a transmission from an emergency radio beacon around 7:30 a.m. and launched helicopters.
CBS Station KYW reports that a helicopter crew from Atlantic City found an unmanned life raft 75 miles off the coast of Cape May. Three crew members were found nearby and plucked from the waters.
"We spotted a wood plank and on the wood plank there was the red of a survival suit," said rescue swimmer Lake Dowhan.
Officials said two fishermen were unconscious, one without vitals, while a third was reportedly responsive in good condition. The survivor told rescuers that there were seven on board the boat when it sunk, and that all seven had donned survival suits before ending up in the water.
KYW said that several rescue helicopters and Coast Guard crews from around the region are still searching for the remaining four who were on board the ship.
It's not yet known what caused the boat to overturn.