What bit are you using?
1/4" doesn't tell us anything. 1/4" downcut, upcut, 1 flute, 2 flute, etc. Who makes it, what's the model #?
Are you cutting aluminum with a router bit or an end mill? Router bits have a lot more clearance angle on them than end mills, made for cutting aluminum. If I were cutting 1/2" thick material, I'd probably not be using a router bit unless it was all I had. They will work, but an end mill will work better for that application.
It could certainly be a bad collet. Runout with a carbide tool, cutting into metal is a VERY bad thing. Carbide doesn't like to be shocked and uneven cutting loads on the tool caused by a cutter that's running out can break the tool almost the instant it hits the work.
Still need more info. How are you holding the work down? How rigid is the setup? Remember, it's a router, not a milling machine.