Usually the first folks that call for banned speech fear the argument being made. It is up to the individual to use their intellect to discern what they hear., and if they want to hear it. Sadly, many people blame their own actions on what others have said, taking no responsibility for themselves or their own actions. Hitler's rhetoric; Obama's rhetoric; Osama's rhetoric; Stalin's rhetoric; did not make history, it was the people who acted on that rhetoric that did. In a civilized world, it comes down to personal responsibility. Rhetoric is easily ignored. If you feel intimidated by what someone utters, the best way to deal with it is to educate yourself about the issue at hand.
Be careful what speech you want banned... it might be yours next.