Most agencies are sleazy? It might be more true that most sign shops do not cover their butt good enough....
What did your contract say about cancelation and kill fee? How does your contract cover your butt? If it has no cancellation/kill fee, why not? Cover your butt.
Most agencies use the GAG and AIGA standard form of agreement... translation, they cover their butt. For an explanation go here and read up on how design firms (or what most of you people call "agencies")
http://www.aiga.org/resources/content/3/5/9/7/documents/aiga_9standard_agreement_07.pdf
I don't think you need a 25 page contract, but you need to protect yourself from nimrods who are window shopping.
You should be reimbursed for your time you put into it (unless you volunteered for free) and a restocking fee if there was a restocking fee imposed on you. If it was not in the contract, you might have issues, but I would still impose a restocking fee anyways since that material has to sit around till you use it on another job(s).
Next time you do business with a design firm or "agency" have a contract that has this covered. Make sure you know who get billed beforehand... NEVER EVER EVER give wholesale to a design firm... they "usually" are a pain in the butt when it comes to color so you add a line item for color matching and add some amount for the "pain" it can be for working with this type of client.
When doing work for a few design firms, it was common to have 2 prices.. the regular price and the Mouse company price because you knew that the Mouse was going to complicate the process so much, that you were going to lose your butt on the job if it went slight bad. Same goes for working with a design firm.