I agree that something is missing from the equation, but if it is how you said it is, the first thing is to hire an attorney. If you paid the guy and he hasn't done what you paid him for the only way to get your money back is with a suit against him or the bond, if he has one. Most states on the East don't require them.
I don't fully agree with the "payment once finals are done" concept seing as how 90% of what we do is installations for other national companies and we group the inspections together. You should in future have a protocol for paperwork, and require certain documents before full and final payments are made. We submit photos, front and back of the wall, sign lit, copies of the permit and receipts, as well as a client sign off sheet to verify the work was done to the end users satisfaction with every project.
Going to the board that licenses' this company is definately and option, but I found here in Florida that they do little to nothing about helping.
Good luck.