Anecdotally, I've noticed that my allergies seem worse after working with it, even when using those brands specifically formulated for indoor containers.
(Also, I tried to tag this "allergies," and it wouldn't let me because of lack of reputation.)
Potting mixes are not all the same. I am horribly allergic to bark dust, bark in any form. The usual ingredients to potting soil/medium/mixes are: Perlite, Vermiculite, peat moss, worm casings, coco fiber. Yours might have a bark dust incorporated.
Send the list of ingredients. Don't handle the mix without latex gloves, keep the top moistened with a spray of water, not too wet! Just enough that the ambient air movements aren't able to stir up the dust.
Your potting soil just might have cedar or pine finely finely ground into dust as part of its formulation. The other ingredients are less likely to be a problem.
Potting soil is the only medium that should ever be in pots for potted plants. Send that list of ingredients. All ingredients. Do you have other allergies? Strawberries, grass pollen, animals? Send that info as well. What are your reactions?
I think your potting soil has finely ground bark as part of the ingredients...that is easily fixed. Also send a picture of your plant/plants. New potting soil without bark on top of the potting medium with bark should all that will be necessary to ease any interactions between you and bark, if that is the problem.