We have small, dark brown bugs on the underside of our mint plant (in a window box on our balcony). I noticed today these bugs on the underside of our leaves.
Looks like a black bean aphid . I usually make a solution of soft soap and water, around 7 parts water 1 part soap. You can also add some paprika or something rather spicy to it, but that's not completely necessary.
Simpler solution would be to spray them out of the plant with cold water ensuring that they are not falling on other plants or in the vicinity of the boxes.
Ladybugs are natural predators to the aphids, you might attract some and have them around your boxes. You can find more info and treatments here