If its just the finish that's too thick to let it fit, then yeah, I'd say sand it lightly, to make enough room.
Just be sure to keep it round and not get oval shaped hole, 'cause then, it'll wobble.
The 'brushed' or grooved part is to hold it in more securely.
The bushings should go in up to the flanged (flattened overhang) part.