They were on a large tree in the mountains of North Carolina.
I believe this is the paw paw tree or Asimina triloba.
- It does grow in North Carolina
- the leaves come to a sharp point as they do in the picture
- it grows in a variety of habitats up to 40' (13 M) tall but usually not more than 25' (8-9 M) tall
- it has large clusters of fruit which start green and ripen to a yellow colour
- bark is described as "Smooth, brown, splotched with wart-like lenticels, often with light gray patches." The patches can be seen on the smaller branches
- alternate simple leaves
Positive identification would be crushing a leaf and confirming a green pepper odor. This answer has more details on growing and the difficulties of harvesting the fruit.
I can tell you with confidence that this is Yellow Buckeye (Aesculus flava)
Pawpaw was a good thought, but you're right, the fruit texture and shape isn't right. Pawpaws are very smooth are more elongated.
If you look at the picture closely, those leaves are palmately compound. Pawpaw is a simple leaf and it is rarely found in high elevation areas of NC. In fact, I rarely find pawpaw wild in the NC mountains in general.
Yellow Buckeye is commonly found in high elevations, can become a large tree (I have found some about 6' dbh), and would have had fruit that size when you asked this question. They are found in rich coves, or mesic slopes and are an indicator of rich soil.
If you had pulled a fruit off the tree you could have opened it and found where it gets the name "buckeye". The seed looks like the eye of a buck. You're supposed to keep one in your pocket for good luck, you know?
I'm going to take a wild guess. Did you take this picture in Great Smoky National Park?