Heh... I'm a couple steps ahead of you... They are called favicons, and they are a little bit of a pain in the ass to apply, but go
here or
here... oh, and
here.
This site makes the application a bit easier to comprehend but you'll still have to add code to each of your pages. Individual results may vary, I haven't tried these yet, but I've got my icon about ready. Oh, and it's a 15 or 16 pixel by 15 or 16 pixel square so limit your detail.
edit- Ok, so I'm done now and the hardest part by far is getting an image that looks good that small. Mine is a mix of AIG and it looks ok, but it took a bit of work to make it look right.