I agree with michelle...are you adding ANY soil to your mix? It sounds as if it is all compost or organic matter! Plants HAVE TO HAVE SOIL! And I gotta say NO WAY DO YOU USE GARDEN SOIL in pots/bags. Garden soil is a very large ecosystem. It is not perfectly mixed but in the larger picture it is better able to provide balance. There is NO WAY to shovel up a little chunk of this ecosystem and ensure balance for potted/bagged plants. ALWAYS use sterilized soil as a basis for pots and bags.
Making raised beds in your garden would be the best thing, Morgan! You can add compost as you double dig but after that you should never have to double dig again unless it is late fall or early early spring. Sounds like you are doing TOO MUCH OF A GOOD THING and a heck of a lot more work than is necessary or prudent.
I HATE vermiculite. Perhaps it is just the little white spots but organic matter makes ANY SOIL better and wonderful. In fact DECOMPOSED ORGANIC matter is the ONLY way to improve ANY SOIL. Somewhere on this site I have gone at lengths to explain. The only time I will ever use peat moss is to add acidifying organic material to soil. I love the texture and ease of peat moss but it also SHEDS water if it isn't mixed in well and certainly your amounts are way too much.
Have to add that there is this...'hippie' type guy I kinda knew in Seattle, WA. He made his own mix and was selling it to clients/followers/non-gardeners for $100+ per yard. Lots of pea gravel 's involved...for drainage I guess. What a waste of money!! He got rich 'making the best ever soil' as well as using his clients money to 'design' landscapes... but this is not correct management of soil! Out of all the soils I've had to manage, heavy clay did as well as yummy silt loam. I hate this sandy, porous pumice I am struggling to grow stuff in!! But all three; I use raised beds, I mix DECOMPOSED organic matter (compost) when I first double dig and fluff and create my raised beds with trenches along the bottom of these beds to direct excess water. I cover and keep covered with compost all year. Organisms eat this stuff and go back into the profile to POOP IT OUT. The decomposed compost feeds the organisms in the soil and their activity and poop support plants as well as making ANY type soil wonderful soil. DECOMPOSED is the important word here! Kitchen scraps, bark, rock...all are considered mulch. Compost anything that is organic. Compost should mean DECOMPOSED organic matter!!
Gotta stay with what you have for soil and learn the proper management for your type of soil. The ONLY proper way to improve garden soil is adding DECOMPOSED organic matter on top of your beds if they are already established. I've said this before but made a big impression for me as a gardener...if you've got clay and add gravel, gypsum, lime, water and agitate...what is produced? CONCRETE.
Get a soil test, take a mason jar and put a few inches of your soil in the jar and fill with water. Shake, shake shake and allow to settle. This will show you percentages of the different soils that make up your natural soil. Heavier sands will be on the bottom, silt/loam in the middle and clay on the very top. Organic matter will be floating in the water. I don't remember the percentages for labeling your soil but that is out there on the internet. Add to this a laboratory soil test...and you'll be able to make decisions about your soil/plants/garden/management practices that make sense for your soil...and much more chance of success!! You are definitely a gardener, Morgan! This information will FEED you!! Grins...