I have three mature citrus trees. One orange, one tangelo, and one really productive lemon tree. These trees produce a ton of leaves as you might imagine. Will composting so many citrus leaves "taint" my compost or will it not matter all that much? What about composting the fruit itself? I want to use everything I can. The rest of the material is mostly vegetables from food prep and compostable paper products.
The general rule is to compost everything you can from as many disparate sources as possible, avoiding anything which appears to be affected by pests or diseases which would be propagated back to make an existing situation worse. You don't mention any existing problems so your goal is probably to add into your composting more different sources, grass clippings, bagged earth or manure and so on to increase the variety of materials in the compost heap, then manage the compost to allow it to reach really efficient hot internal temperatures.
Depending on your composting setup, whole fruit could increase the appeal of your heap to rats, raccoons, or whatever critter is near you. They can be very clever and persistent when there is a big supply of sugar at stake.
If you do end up composting a lot of whole fruit, you might want to get litmus paper or a pH meter to check whether the result is too acidic. The leaves shouldn't be a problem for this either.