NEVER use compressed air on a speaker. Never ever ever ever ever. The chance of blowing stuff into the coil gap, or tearing a cone or separating a cone or spider... way too great.
Damp anything - not good. The cone can fall apart.
Best way I found - using a brand new, clipped paintbrush. Take the brush and shorten its bristles to about an inch long (just get them shorter, stiffer, length may vary on bristle). That will take off anything that is safe to remove.
If that dont do it, it needs not be done