Of course, we DO work with some other sources of nasty stuff. Consider all the critters we might own. Some urchins (and starfish!) are capable of exposing you to some unpleasant toxins. Cnidarians of various sorts are potentially dangerous--I have a potentially very painful coral (the consequence of impulse-buying a pretty critter without researching it first....

)
And even if you aren't into corals or such, consider the live rock you buy--quite possible you have some fireworms or other hitchikers. Some Zoanthids hitchiked on my live rock and were VERY happy in their new home, multiplying quickly. I've never confirmed the genus, but I'm 90% sure they're of the type that produces palytoxin, a very serious neurotoxin stored in their mucus (NOT in their stinging cells.) A coincidental accident removed most of them from my tank, happily. I just don't like having something that could hurt me badly in there.
Anyway...I agree, rubber gloves and careful hygeine are probably a very wise course of action. I use them all the time now. Of course, what I REALLY want is decent elbow-length disposable gloves, but I haven't found any yet...
rusty