Hitler used the path of least resistance, the undesirable. Remove those who are not respected or wanted by society and none will complain. As to the camps, most were in isolated areas until they had to build them near towns for resources and shipping. The answer is some knew and they didn't talk.