I don't ever take a non working vacation, but I get to get away several times a year. I'm in Indiana, and I normally go to Orlando for a week with the family (and ISA) on spring break, and then get to Michigan for a week in the summer, and try to go somewhere else a week or so with my wife in there, and then long weekends as often as possible. I'm normally on the computer or on the phone a couple hours each day while I'm gone, but I figure that's a pretty sweet trade off I can do back at the hotel at night, or at the pool while we're just hanging out, instead of being tied to the shop.
I recently went "calls and emails only" though. So far, it's been amazing, but you kind of have to choose rather you want your life under control, or if you want to grow your business. The hardest part was training people to not stop by just because it's convenient for them anymore. They always emailed me the stuff anyway, they just wanted to stop by first to tell me they were emailing me...once I could retrain their brains, it's been great...and most of the time now, they don't know rather I'm in town or not. With so many Merchant Members that make my one man shop run like I've got 2 employees, UPS can drop off a bunch of stuff, and I can still take care of the customer while I'm gone.
Like everyone else said, send them messages and let them know you're going to be gone. I do a lot of auction company work, and I just tell them a month before I'm leaving town that I'm going to be gone, and they make sure and have everything to me the week before I leave, so they can be setting it up while I'm gone. I realize I'm lucky, and not everyone has this luxury, but the worst that normally happens is someone will get jealous that they can't get away like I can, and my response to that is.....why would I take the risk of working for myself if I also didn't get the reward of being flexible with my time.
You just have to make yourself do it. You make time for what's important to you. Everyone does. It's just different for everyone. If you have clients that won't accept that for an answer, you have to decide if you want to be a slave to them forever, or if you want to change your life and put your nuts on the chopping block and hope they understand.