The trouble is the occasional nutter who goes overboard. The Muslim employee who demands that the office not close on December 25th because he wants to work that day. The person who asks HR to instruct all bearded employees to shave because 'beards are an aggressive statement of masculine identity and dominance' (I kid you not, this happened at one place I worked). Not to mention the horde of aggressive, proselytising, vegans!
These 'minorities within minorities' are seized on by right-wing commentators and media to tar entire communites with the same brush, and too often it works.