The troops raided the compound because they suspected it harbored militants belonging to a suicide bombing network, and they were fired upon as they approached late Wednesday in Bati Kot district in Nangarhar province, said Maj. Chris Belcher, a spokesman for the U.S.-led coalition.
After the clash, a militant and two children were found dead inside the compound, Belcher said. A woman and another child were injured, he said.
The military has launched an investigation into the incident, he said.
"It is regrettable that the civilian lives were put in danger by the militants and our sincere condolences go to the families of the deceased and wounded," Belcher said.
A policeman was also wounded during the raid, said Ghafoor Khan, a spokesman for the provincial police chief.
Three other men from the house were detained by U.S. troops for questioning, Khan said.
U.S. and NATO officials say militants frequently use civilian homes as cover during clashes with foreign and Afghan troops, causing civilian casualties.
But civilian deaths incite resentment against U.S. forces and have sparked several anti-U.S. and anti-NATO demonstrations this year.
President Hamid Karzai has pleaded repeatedly with Western forces to do all they can to prevent such deaths.
Also Thursday, Taliban militants attacked a police checkpoint in Nad Ali district in southern Helmand province, killing five officers and wounding three others, said provincial police chief Mohammad Hussein Andiwal.
There were no reports of militant casualties, Andiwal said.
In western Farah province, six police officers were killed and two others wounded, and 14 Afghan army troops were missing after clashes with Taliban fighters on Wednesday, said Gov. Muhaidin Baluch.
A large number of Taliban have crossed into Farah from neighboring Helmand province and are still in control of Gulistan district, Baluch said.
Police have battled militants for three days in the area, and several guerrillas were killed, said Baryalai Khan, a spokesman for the provincial police chief.
Violence in Afghanistan this year is the deadliest since the 2001 U.S.-led invasion that toppled the Taliban militant movement from power in the country. More than 5,600 people have died this year due to insurgency-related violence, according to an Associated Press count based on figures from Afghan and Western officials.