This delightful plant has been planted too deeply in the original pot. Roots only escape to find water. They die when there is little air and too much water. They rot. It needs to be transplanted into sterile potting soil until you've found a place to transplant into the garden soil.
For now, replant this guy in a pot 2 inches larger. Looks like it is in a 10" pot? Transplant into a 12" pot with sterilized potting soil and a bit of Osmocote 14-14-14 Extended release fertilizer.
I can see that that trunk is below the soil line. This will kill your plant eventually. Make sure the trunk is completely out of the soil, mulch, rock whatever. The roots right below the line of trunk that needs to be exposed to the air and the root crown...have to be below the surface of soil. The trunk if below the soil will rot and you will lose this little tree.
This little tree will be able to last in a pot until you decide on a great place on your property to plant it in a forever home. Sterilized potting soil only while in its pot. Do not amend the soil when planting in your garden. Dig a hole only as deep as the root ball, no deeper or it will sink and the bark of the trunk will be compromised.
Where do you live? What zone are you in? I would transplant into a slightly larger pot, use potting soil only until you know where you want to plant it in your yard. Use a basic basic fertilizer (Osmocote 14-14-14) go by the directions except half the applications per year. Do not amend the soil in the garden for this tree. Check the pH level of your soil by getting a soil test.
Removing flowers and fruit allow that energy to be redistributed to the entire plant. That is a good thing.