No, it's not too late to 'unplant' it, it won't yet have started to put out roots from the pot shaped rootball. The best way when planting a tree is to prepare the ground beforehand by digging it over in as wide an area as possible and adding humus rich material (garden compost, composted animal manure, whatever you can get hold of), then letting it settle for about 10 days, then digging your hole and actually planting, at which point, you won't be including anything in the planting hole other than fertilizer, and then only if it's spring or early summer.
Given you've got the tree already, and haven't prepared the ground, the best thing to do is unplant it, put it back in the pot in which it came, remove the potting compost and new topsoil as far as possible, dig the area over again, preferably bigger than twice the size of the rootball, this time including something decent as mentioned above rather than potting compost, walk heavily on the area on your heels to compress it and get out air pockets, then dig your hole and plant your tree. Rake up the area you've walked on around the tree so there's loose soil on the top, not solid, compressed ground. If you haven't any garden compost or humus rich materials available, you have time to go and get some before you dig up and replant the tree, it'll be fine as it is for a few days, you'll still be able to do it.