It is a fungus digesting the organic matter in the soil underneath the plant, and it is, if anything, beneficial. See the bark chips in the soil? They are what the fungus is breaking down. The white stuff you saw under the surface is mycelia. The part on the top is the reproductive material.
It really only eats the undecomposed material, rendering it into something the plant can access. If the look bothers you, it would be fine to remove the patch, but it won't harm the plant. As stormy notes, this type of fungus usually only appears when the soil surface doesn't get much time to dry out. This constant damp is hard on a potted plant. Make sure the top 1-2" of soil dry out in between waterings.
But from what I can see, the rosemary looks fairly healthy. I also see a little sprout (weed?) coming out, usually a sign that the surface soil isn't drying out.