One of the most notorious is Cocobolo. Even if you're not allergic to it initially, you will most likely develop a reaction after working with it over time. But that's just if you're actually cutting, sanding, generating dust, etc. (which is why Warmoth doesn't bother with it.) But I have zero problems with a Cocobolo fretboard on my current #1, and I have all sorts of allergies. And the neck is actual Padouk, unfinished, no problem. I actually haven't heard of problems with Padouk, and only very limited cases with Pau Ferro.
But if you're just playing the neck and not sanding the wood, I think you'll have no problem.