I don't want to kill it, but I want to keep it from leaving my mulch bed and wandering into my lawn. If I spray the edges of the mass, will it only kill a small amount, or will it kill the whole mass?
According to the advertising for this product, yes it will kill the whole plant back to the root. In practice, it doesn't, usually, but the way to control your Vinca that's best for the plant, your garden and the environment generally is to give it a good haircut after its flowered by pulling up all the long strands of growth and cutting them off at about 2 inches high. If you haven't done this in the past, you've probably got new plants growing from where the long runners have rooted, so dig these out where they are getting too close to the lawn.
Cutting it back yearly after flowering also creates neater, bushier plants which are much more attractive than a random patch of Vinca with bare bits in between long runners.