When I saw your photos I immediately said "slugs". Or snails, depending on where you live. The roundedness (if that's a word) of the bite marks is the clue; that, and indications that the culprit nibbles mainly from the outside of the leaves, eats in irregularly shaped patches, stops nibbling after a short time, and doesn't skeletonize the leaves. Since you're asking here, I'm also assuming that you never see what's doing the eating, which also fits slugs and many snails, since they tend to feed at night or on cloudy/rainy days. Have you noticed shiny trails on the mulch, leaves, ground or any surrounding concrete in the morning?
The only thing that causes me to question my ID is that we're talking about a potentially large shrub here, one with presumably one or just a few trunks, but with a large amount of damage. Unless the slugs or snails are hiding out in the leaves, or the damage has been going on for a fairly long time, the damage seems greater than your average slug/snail infestation.
One way to confirm it's slugs is to set the lip of a shallow container even with the top of the soil/mulch near the trunk of the buttonwood, and then fill it with beer. Slugs like to party, so if they're around, you'll find them in the beer.