Our garden in the North of England was full of these when we moved in. It grows a lot vertically, never seems to flower, is deciduous, and has leathery green leaves.
Been trying to work out what it is for a while with no luck.
Looks to be some kind of fruit tree that's been 'let go'!
The leaves are somewhat plum like and the trunk appears to be a bit cherry-ish - is the bark at all shiny?
There being lots of them in the garden, plus your lovely wall, would tie in with them possibly being part of an orchard, or maybe they've self seeded. As to the lack of flower/fruit, I would imagine it's been grown from a grafted root stock, very common in modern fruit growing. The root stock has died, leaving the infertile suckers to take over.