FM 5.0b54 – this seems to be a repeatable error 
Select an image and apply an Oval Mask. On my system, it is automatically defaulting to a Softness of 2. Not sure why, since there is no place to set a default value for that. In any case, I dragged the slider back to a Softness of 0 and the image now appeared with a “stippling” effect (kind of like “noise” was applied to the image using a Photoshop filter). Setting the Softness to 1 eliminated the effect. Setting it back to 0 reintroduced it.
Rectangular masks don’t seem to have a problem that I can tell.
There is an obvious bug here – if it’s reproducible by someone else.