Some experts say this shouldn't be too surprising.
"Unfortunately this type of surveillance is pervasive in homes and it gets to the main point that you should be able to watch your TV without your TV watching you," said Khaliah Barnes who studies privacy issues for the Electronic Privacy Information Center.
It's not just televisions gathering information in your home. A Federal Trade Commission report last month on "The Internet of Things" like smart locks or smart meters pointed to benefits, but also risks. It said if devices like TVs store sensitive information, 'unauthorized persons could exploit vulnerabilities to facilitate identity theft or fraud."
Shoppers who spoke to CBS News had mixed views on the technology.
"I would not want people to be able to get into my living room. To me the benefit of that service wouldn't outweigh the potential danger," said one shopper.
"I personally love them and I understand why they're collecting that data," said another. "Most of the reason for collecting the data is to improve recognition, improving what the features of the device can do."
The FTC recommended companies build security into their devices at the outset, rather than as an afterthought.
Barnes doesn't believe companies are doing that enough.
"Companies could go further by advocating for privacy enhancing techniques which would minimize or eliminate the collection of personal information," she said.
Samsung told us it takes consumer privacy very seriously and uses industry-standard security safeguards and practices, including data encryption, to secure consumers' personal information. The company says it doesn't keep that voice data, but also points out the voice recognition feature can be turned off by the TV's owner or they can disconnect the TV from the Wi-Fi network.