Maybe that's what I'm doing wrong....cutting my nose hairs.
It all depends on the situation. You can't subject other people to the dust, so if you're drilling near anyone, it needs to be vacuumed.
Also, you don't have to carry a separate vacuum, every major player offers onboard extractors...