No worries. The glue is more of a filler than anything else. It makes things tight and secure, but it doesn't really behave as an adhesive because you're talking about a metal to wood joint. So, the frets will sometimes move as the neck acclimates to its environment. Expansion, contraction, stress, etc.
I go through a lotta necks here, and they're never perfect to start with. If somebody is checking to see if they are, they'll find problems even on better parts like Warmoth sells. The problems are usually small, but they're there. So, you grind 'em back a bit, recrown them, dress & polish them and call it a love story. Chances are they'll never give you any problem later, outside of any wear you inflict yourself.