I found out elsewhere that they seem to be crane fly larvae, which fits the images google spits out.
This makes sense, crane flies are fairly common where I live (close to the Rhine), and I remember spotting one in my apartment a few months back, despite fly screen on the windows. That's probably the one responsible for these larvae, or maybe it just came ...
The rosemary leaves won't cause any real harm if left on the soil (assuming the plant is in the ground) but they won't do much good either in terms of being a fertilizer. If there are enough of them, they might act as a mulch layer - they can take quite a long time to break down completely. Probably best to add them to a compost heap instead.
You should smell it when you rub or bruise it, so nothing seems to be wrong with your plant. When it was in the bag and it smelled when you open it, it was probably rubbed by the bag.
There was recently a similar question answered by Bamboo, where the OP asked the same for lavender.