I had the same pale yellow fungi in my pride of Barbados pot , but probably 5 X larger . I pulled the first 2 and threw them into the back yard ( wife did not like the look of them). The weather turned warm and I moved the pot outside, and it grew a couple more which I left. I just looked and there is one more about the size of photo in my pot. As a wild mushroom picker ,it never occurred to me to wear gloves. Very unlikely that the mushrooms have anything to do with the cactus. Supplemental : The mushroom looks like Lemon Yellow Lepiota , Lepiota lutea, poisonous. Common in compost, in greenhouses and house plants ( north America).