That would be the smallest pot I would use for those plants. I only re-pot once (from the seed tray). Of course re-potting disturbs them and sets them back a bit. Also the roots can get tangled (hence the disturbance).
Over the last 2-3 years, I've come to the realization that peppers do much better in raised beds or normal beds, rather than pots. I have had reasonable size adult plants in pots that size or larger (pepperocini come to mind, although this year's are slow growing), but they tend to be the exception.
Ornamental peppers seem to do fine in pots as well, but ornamentals are generally smaller plants to start with.
Of course pots might be the only option, in which case I think you have to live with smaller plants and less fruit.