The Public Committee Against Torture in Israel made the allegation in an 86-page report that examined six cases over the past year. It says Israel has put more emphasis on psychological torture since Israel's Supreme Court restricted the use of physical torture in a 1999 ruling.
In one case, the group said, Israeli agents convinced a suspect his wife had also been arrested and tortured, driving him to attempt suicide. In another, they detained a couple for an extended period, tortured them physically, and withheld information about their two young children to try to break them.
The Shin Bet internal security agency, which conducts the interrogations, denied the charges.
A parliamentary committee heard the group's findings in a special session Sunday.
A 1999 Israeli Supreme Court ruling sharply limited the Shin Bet's interrogation methods, essentially outlawing torture except when there was clear evidence the suspect had information that could save people from an attack in the making. The Supreme Court outlawed what the Shin Bet called "moderate physical pressure," such as exposure to extreme temperatures and tying up detainees in painful positions.
However, the Shin Bet has come under criticism for its new methods, which include shackling suspects in contorted positions and depriving them of sleep for long periods of time.
Yoav Loeff, a spokesman for the human rights group, said the Shin Bet has added using relatives as leverage during interrogations.
"They use family members to force people to confess and they cross all the red lines along the way," he said.
The Shin Bet said in a statement that it never detains relatives of suspects, or presents false information to its suspects in an attempt to elicit information.
"Terrorist investigations are conducted by the Shin Bet according to the Supreme Court ruling, under the restrictions of the law and the tight supervision of the Justice Ministry and the courts," a statement read.
"The information acquired in these investigations allows for foiling acts of terror, and many civilians in Israel owe their lives to these actions," it added.
The report Family Matters: Using Family Members to Pressure Detainees Under GSS Interrogation" is available online, in Hebrew; an English language version will be available soon.