Well, there are some things you can do that are "Organic," and safe for the environment. #1 you are already doing.
Handpick caterpillars from plants as you find them. You may need to patrol your garden at night to find slugs and some caterpillars. Relocate the caterpillars to a different area, or kill them by dropping them into a bucket of soapy water. Or feed them to your birds.
Spray the foliage of plants with neem oil. Neem oil controls caterpillars and other pests, including whiteflies, leafminers, and thrips. It may also kill beneficial insects such as bees and earthworms, however just don't spray you plant when they ar are blooming, and you can avoid much of this. To make a longer-lasting mixture purchase Neem Oil and Agricultural Oil to mix with water. You can also, spray with peppermint, lavender, or other essential oils, but for your problem, this would get expensive.
Dust plant foliage with Bacillus thuringiensis, or Bt, powder. Bt is a natural bacteria that kills caterpillars, but it does not harm most other beneficial insects. One of my favorites.
Invite birds and predatory insects into your garden. Hang a bird feeder to attract birds to the area. Ducks and chickens also enjoy eating caterpillars. If your garden lacks predatory insects, such as praying mantises, you can usually purchase a box of them from a garden center. My favorite since I started feeding the birds is a troupe of sparrows that have taken up residence in my Boxwood Hedge, and there is no insect damage anymore. Also, introduce habitat for toads and frogs on your property, in the long run, you pest population will dwindle.
Plant herbs and flowers around your garden to repel caterpillars. Many insects dislike the scent of lavender, mugwort, sage and peppermint. I like to plant garlic cloves among my plants, which drives away pests.
Finally, predators and co-planting are the long term solutions. For now, I would spray and use the bacilli solution. Wasps are a great solution for caterpillars; one big wasp nest will eat a couple of pounds of caterpillars a week if you can stand them. Several plants weirdly attract wasps; my favorite is Sea Holly (Eryngium maritimum). Plant it around your border and see what happens. Remember to put up with some damage, but if this is an invasive species best to eradicate it.