Scrape it off, or you may get false readings from your various measuring/checking tools. It doesn't need to be on the frets anyway; it's for the wood. You don't need to work too hard at it, though. The levelling file(s) will take the top off, and the crowning/dressing files will get much of the sides. By the time you're to the polishing stage, you should be in good shape. Highly recommend you tape off the fretboard, though. It'll only eat up 15 or 20 minutes of your life, and it'll quite likely save you a lot more work later.
For example...
Some things you can see, others you can't. For instance, you can't see all the scratch marks perpendicular to the grain that would have ended up on the fretboard's face, because there's tape on there to protect it. You can see where the polishing wheel discolored the tape on a few frets up from the 12th. That probably wouldn't have affected this fretboard much because it's ebony, but on a finished maple 'board, it would have been cause for much profanity <grin>