Want to disable this evil key completely? You can. There's no setting in Windows to do so, but all you need to do is run the disable caps lock script at JohnHaller.com. The script will update your Windows registry to completely disable the Caps Lock key, and it will never bother you again.
The site actually has several similar scripts; you can instead opt to turn the Caps Lock key into a Shift key, or, if you prefer, to disable both Caps Lock and the Windows key. Personally, though, I think that simply turning off Caps Lock is the sweet spot. I've never been so happy to be rid of something.
To use the script, just click it and let it run, then reboot your PC for the change to take effect.