My plum tree is about 10 years old and has fruited for a few years. It produces loads of fruit, loses a lot before ripening, but they remain the size of a very large olive and I don't know how to encourage them to get bigger. I don't feed very often, so that might be the problem. Whilst the fruit are growing, the tree also gets quite a few new branches, which I try and trim back in order to encourage the good stuff to the plums. Do plums take a while to grow to size? It's a self fertilizing victoria plum. The wasps got them all last year, but once ripe they are quite tasty.
It doesn't sound as if you practice thinning out the fruit - plums particularly have a tendency to form a lot of fruitlets, and even after June drop has occurred, there are often too many fruits left behind which then don't mature properly, remaining small. Guidance on thinning in this link https://www.rhs.org.uk/advice/profile?pid=340.
Also ensure the tree is kept well supplied with water if the weather is particularly dry once the fruitlets have formed.