So after years of struggling with accuracy on rock walls with irregular surfaces, we tried something today and it worked like a charm.
Before we started drilling with the masonry bit and hammer drill, my partner went along and used a carbide tipped ceramic tile bit instead of a punch to give me a small pilot hole. Barely an 1/8th deep was all it took to give a really good head start on where to place the masonry bit.
None of that bit jumping all over the place and the arrow shaped point on the ceramic tile bit was perfect to create a really accurate, pinpoint of a hole to start drilling into.