The answers to these questions matter, because in the United States, the Patriot Act is still in force and in effect, which enables law enforcement officials to, among other things, engage in warrantless wiretapping.
Unfortunately, there’s no way around the issue, except to just say no to smart TV’s. There’s simply no gain in paying a premium for the technology, then disabling it. In the end, it comes down to which you value more: the convenience of voice activation, or keeping your private conversations, private. For almost everyone, the answer to that question is a simple one.