Hmm.. v-carving in 3mm. That creates problems all it's own. Unless the acrylic is textured in some way there still shouldn't be any issue. It's possible that the heat from the bit is warping the acrylic on deeper plunges, and that may be affecting the immediate area, but other than that I have no suggestions. Flat sheet is flat sheet - if yours isn't to spec and flat, why are you trying to rout it? Send it back or try another sheet. There are no programs or techniques that will allow the bit to follow the uneven contours of a sheet. You either need to check your sheets at the beginning with a straightedge or deal with the problems of trying to rout material so thin.
That's some mighty cheap *** acrylic you got there if the variation in thickness varies up to .5mm. That's .019". That's way outside most manufacturer tolerances. Personally, I'd return that crap and get decent acrylic.
My guess is your table is not flat or the material is not laying flat on the table.