Moss is an opportunist. Spores are everywhere. The moss on the ground is not making the moss on the tree. Moisture (we must be looking at the north side of your tree, yes)? Is all moss needs.
I would scrape the moss off the tree...at least in the Y's between branches and trunk. Otherwise, I would not mess with the moss on the ground. Perhaps a thin layer (1") mulch of finely ground bark. Your neighbor is wrong that moss is 'catching'...if there are bare spots with constant moisture there will be moss. On trees, soil, bare spots in the lawn, north sides of homes...moss is not a problem, moisture is.
The base of the trunk looks healthy. I'd be more worried about moss thickly covering the base at the soil level as that will girdle and kill the entire tree. Looks just fine. If and when you cover with mulch make sure it is thin and does not touch that bark of the trunk. Any bark, rock, soil, moss, weeds up on your trunk can kill your tree...because they keep moisture on the bark, rotting it and next would be damaging the thin vascular system just below the bark.