The EASY way
**** from www.frets.com ***
If the nut is too low, the string will bump into the first fret when it is played and it will "buzz," even if the rest of the instrument is properly set up and working well.
If the nut is too high, playing at the first fret will be seriously impaired. I frequently see guitar on which it is impossible to bar a first position "F" chord because the nut is way too high. This is extremely common because most new instruments only receive a cursory "set up" at the factory. We hope (sometimes in vain) that the more expensive instruments won't have this problem.
Is there an ideal height? In theory, yes. The ideal height would be just about the same as a fret's height in the same position as the nut. In practice, I've found that there are some musicians whose playing attack is so vigorous that they require the nut to be higher to avoid the open string buzz.
Can I check my own nut action?
Easy! Here's how. . .
With the guitar, banjo, mandolin or any other fretted instrument tuned to pitch:
First, grab the neck and mash down on the string just in front of the nut.
Pretend your thumb is a capo, and just squeeze down. Don't give it the King Kong grip, though. The idea is to avoid having the string elevated artificially, as it is when first put on. A string that has never been fretted strongly at the first fret tends to stand just a little high. I'm only talking about the stiffer bass strings because they seem to have minds of their own.
This is just a preliminary exercise, and is only a minor precaution.
Here's the real stuff:
Press the string in question down between the second and third frets:
The string is at full tension and it strikes a straight line from the nut to the second fret. There's no need for a straightedge here, the string is straight!
Get down and look CLOSELY under the string above the first fret. It pays to be nearsighted. For the unfortunate non-myopic, magnifying reading glasses may be necessary.
If the nut is at its ideal height, you should see a very tiny space between the string and the first fret:
How much space? Well, it varies with need, but generally, I like to see just enough space to tell there is any at all. I'm talking the thickness of a hair! Notice the string in this picture, IT IS CLOSE. This guitar plays well for the average musician.
Here's the first string:
If there IS a gap that's too small to see, you'll hear a little click as the string hits the frets. Cute trick, eh?
Bear in mind that nothing is absolute when it comes to instrument set up. It is, after all, a matter of adjusting instruments to the player's needs. Action at the nut is one of the more easily described and yet more overlooked aspects of set up.
**********************
Remember, you want a break angle, then a bit more of an angle (relief) toward the posts. The break angles should all be the same. The releif angles can vary with the angle of the string to the post. Anyone wanting to learn how to set up a nut should saunter on over to www.frets.com and read the several good pages on the subject.