From what I have seen, the nitro colors people work with all seem to be either clear over dyes, or solid non-metallic colors. There are toners you can get from stewmac, and a few guys, CB and Jack I think, recommend a brand called mohawk.
If you really want to get into any sort of crazy metallics, you are going to have to look into automotive finishes. For the most part they are going to be poly products, and a lot of guys here poo poo the poly. Not me, I used it on my MIJ Strat rebuild and couldn't be happier. One thing I will say though, if you do want a custom-sorta color and you go the automotive finish route, be prepared to pay for it. You only need a pint, but ends up being a lot more than just a pint. For instance, the color I used was a PPG product. I had to get a pint of base color, a pint of transparent mid coat, a pint of clear, a quart of reducer, and a container of hardener for the clear. When it was all said and done, I spent about $175 bucks just on the paint!