I don't know that it will be possible to provide any absolute answer. There are lots of remote possibilities and unknowns. The main concern is heavy metals. If the wood was pressure-treated before it was painted, metals are in the wood now and can leach out. Putting pressure treated wood in soil is a good way to poison the soil.
If you feel quite sure the wood was raw when you painted it, there is much less concern. Paint can - will, rather - flake off over time and end up in your soil. Unless the paint is very old and contains lead, however, the paint itself is unlikely to be toxic. Latex paint, the most common now, is more like bits of plastic that end up breaking up in the soil - bit of a mess, difficult to remove since they break so easily, not very harmful either. In the end you're going to have to decide how to hedge your bets.