If you will be living In the same space as the pots then use pumice.
Perlite is 'glass popcorn', its benefits are noticeable in industrial scale agriculture.
In residential settings it is advised not to use perlite especially if you are living in the same room , as it difficult to contain and may get crushed when movingthe pots moving around or more likely to float to the top diring watering and eventually fall out.
Perlite dust is not healthy for humans.
Whilst perlite is safe when damp, when it is dry it is very light making it susceptible to being airborne.
The dry glass dust is so fine, that it should only be used in a well ventilated area whilst wearing a dust mask!
You should only use perlite if you know how to contain it in the pot.
Which involves layering your soil, I use coco coir/sand at the top and bottom with the compost and perlite in the middle, the aim is preventing the perlite from falling out of the pot.
bottom watering also keeps it from floating to the top!
Both perlite and pumice are used as they aid oxygen levels in the soil by providing good drainage and actually releasing oxygen when they are wet eg. during watering of the pots.
Perlite is lightweight and that's one of the reasons it's used industrially.
It's not just about the chunks.