Because it is negatively affecting me and every other American, as long as bigots use it as the excuse for their own hatreds and prejudices. Every time political issues such as homosexual rights, stem cell research, or meaningful sex education in schools come up for debate, what arguments are trotted out against them?
Religious ones, obviously.
And in those situations, I fully encourage the use of logic and reason to point out the flaws in the arguments of the people that are against such things.
However, that's not really Religion affecting you, is it? The idea of Religion is much more than just belief in Christianity or Islam, or Buddhism. And while individual religions might impose commandments or self-righteous nonsense that affect our day to day lives, that doesn't mean that Religion doesn't server an important purpose in our culture, and it certainly doesn't mean that we can afford to get rid of it at present.
There are friends of mine that would be absolutely lost without religion. It's a character trait that I constantly raise an eyebrow at, but it's there nonetheless. They've developed a need for a personal relationship with their god, and taking that away would probably destroy them.
While Religion has done a lot of horrible things in the name of their god, you can't deny that it's done plenty of good, as well. I mean, we can all say that Science is probably the most important thing that we as a species have grown to embrace, yet it can also be argued that science has brought plenty of pain, suffering, and death along with the amazing medical and technological benefits. That doesn't mean we should just get rid of it, though.