I had a similar problem on a newly-planted apple tree. It wasn't fruiting, but the aphid infestation was severely stunting new growth. Although aphids don't infest fruiting plant parts, aphid attacks will weaken the plant as a whole, so a severe infestation will most likely lessen fruit production in some way.
The most affective, 'organic' mode of control for the aphids and the ants (at least for me) was to use a combination of ladybugs (larvae or adult) to attack the aphids in combination with a sticky barrier (like fruit tree grease or a glue band) around the trunk of the tree to stop the ants.
By the way, I found that the glue band works far better than the gloopy gel-type barriers that are available. Glue bands are a product specially made for wrapping around fruit trees to keep climbing pests off. They're a strip of plastic(?) with a very sticky adhesive on one side. You wrap this around the tree, adhesive side out, and tie it on with twine to keep critters from sneaking under it. Here's what it looks like:
If your tree is not too large, you could also spray the aphids with safe insecticidal soap or even just a strong jet of water to knock them off.