It sounds to me like they're leaving 4-5 minutes early to take a smoke break, before going on their normal break. We had some people like that in my last job. All the non smokers got pissed because management allowed it, and if a non smoker was sitting down at the break table 5 minutes before break, they got in shit.
If that is the case... Tell them their break is 15 minutes, not 15 + 5 for a smoke. Otherwise you're not smokers are going to start doing it too, and your 15 minute break turns into 20 for everyone. And while 10 minutes a day extra might seem like nothing, thats an extra hour of unworked time per week.
It also depends on the employees. My current company we have a 30 minute paid lunch, and no breaks, paid or otherwise. When we're slow... We can sit around, or just stand around and chat for a bit. So long as everyone is done all their work, the powers that be don't mind. Everyone here usually groups up and works on cleaning, or doing machine maintenance, or some other menial task though.
So.. it all depends on your work culture, if it's causing a problem, or you forsee it to start causing one. But I'd say if theyre leaving 5 minutes early everyday, and not just once in awhile when there's nothing to do, it'd be a problem and you should put a stop to it.