The issue is that the people who write the viruses and other exploits do so to affect the most number of computers possible which is why most go after PCs running IE.
Most of us here have web sites - check your stats. Unless it is based on a different technology or on a non-IE niche (graphic designers, Linux users etc..), I'm willing to bet the vast majority (at least 70%) of your traffic is a PC using Internet Explorer.
If a lot of people stop sing IE and say a large number go over to FF - if it's significant enough then these writers will start writing their malware for FF also - and the same goes for Mac.
Switching from IE will only delay the inevitable.