Mine grows really fast too, which is why I didn't cut it short before my surgery. I didn't want tiny hairs imbedded into my sling when I'd have to cut it all down again some 2 weeks later. And I kept it at a 4 this time, but was thinking about a #2. I decided not to go so drastic in the changing appearance this time.
ETA: And I just realized that I quoted the wrong post. Yup. I'm an idiot.
Although, my beard's kept at a #2. Like Brent, I get he 5 o'clock shadow sometime around noon, and just decided to grow it out with exception to my neck area. I shave that daily just to make it look like I'm trying for my professional appearance in the office.
Also, and this is just an added benefit, I recently read that it's much better for your face to not shave the beard daily. Obviously, you skip the razor burn and stuff like that, but there were other benefits that I don't remember. I only remembered that it meant that I rock and that all of my choices are the right ones.