You should start liking the mosses. Very nice and relaxing. And it make your garden have an "alpine look".
You may remove it (strong brush), or just turning them, but they will return quickly, unless you try to fight humidity: more sand and stones, less mulch, and maybe you should remove some trees (so having more sun). In such case you may have a more "rocky look" (dry).
Ev, you can plant some runner, which will cover the rocks (e.g. periwinkle), or you may plant some Sedum/Sempervivum/Saxifraga on the rock: they will look better, and they will take some humidity.
I fear it is a lost battle, if you doesn't take a lot of resources regularly.