Film can stretch 'noticeably' over larger panels. As you have seen, even a small % stretch, adds up to a significant stretch over 4'.
If you feel you can't avoid stretch the way you're doing it, maybe you will have to try it wet till you figure out what is wrong with your current technique.
Also, your design allows for zero tolerance error, which just makes it worse on you. Even offset printers (whose equipment is far more accurate than our inkjet stuff), need a 2mm bleed on business cards. This means, that even on precise equipment and expensive German engineered machinery, a 2mm tolerance is required for good results.
Lining up lines in text is insanely difficult with the way you're doing it.
When I have been stuck, I have always at least tried to match from the centre, then trim accordingly the top and bottom of everything to match it up. If you go from the centre, your offset is shared top to bottom instead of having a heavy bias towards the top or bottom only.