Nothing should be added to soil. It would be nice if your soil had been made into a 'raised' bed without using wood or concrete on the sides.
Guess what concrete is made from? Clay, aggregate such as your stones, sand, gypsum, lime when it needs more heat and water and rotation! Rotation is what shouldn't happen to clay soils ever. Clay are tiny particles of rock and they are flat. Their tiny flat surfaces are highly electrostatic and water and rotation will make these particles stick together and literally make brick, concrete.
This is a picture of my first year garden in Zone 3 with caliche clay. Raised beds, decomposed organic matter on top of the surface, trenches to direct the water. This is the soil they make bricks from. Total clay on top of shale and sandstone. No additions of anything to the clay, no grit, no gravel, no gypsum...nada. pH was boosted just by the decomposed (the magic word) organic material. Every year the soil gets better. Up to a certain point then one needs to quit adding decomposed organic material or at least hold off for a year. Too much of a good thing can be a very very bad thing.

You soil looks great. Yummy. Soil by the way rarely has 'nutrients' for plants. Rather 'chemistry' for plants with which to use for photosynthesis where plants make their OWN nutrients, food.
Your biggest problem right now is that soil up against that brick. Pull all soil, gravels, mulch away from your foundation. Try oh try to make some sort of a swale a soft ditch (6' wide 1' deep) to collect water and make sure all water flows AWAY from your foundation. I am not kidding.
The only way to improve ANY soil is by adding decomposed mulch to the surface. The soil organisms need this stuff for energy and they are the ones who mix the organic matter into your soil perfectly. No rock, no gravel, no sand, no gypsum, no peat moss...NADA. This is a picture of how all our beds for plants; ornamental and vegetable should be made. My opinion of course, grins! This no till and no fertilizer propaganda needs to be dumped as gardening techniques.
Note the trenches at the bottom. These beds will never again be 'tilled' or 'rototilled'...I just clean out the trenches once a year and I am ready to plant in the spring.
I most certainly add fertilizer, sparingly and cautiously! And always I cover the surfaces with decomposed organic matter (purchased bales) for the winter. I did an awful lot more of surface covering with DOM during the first year but there comes a point where too much of a good thing becomes bad. Notice the difference in soil from the paths?
I use these once double dug beds for all kinds of soil to include Caliche Clay. Pure clay that turns to snot when it gets wet. I dearly miss my clay soils. All soil is great soil. One just needs to learn how to manage the type of soil they have once they know what type they have. Have you done the mason jar and water thing?
Update: I just have to say, once again, that we humans can not nor should we try to make 'better' soil by mixing anything into the soil. Double digging to make the original bed is the most work we humans should do...well, adding decomposed organic matter to the surface of the soil! Anyone who thinks we humans know best and adds what they think is for better drainage or whatever is just uninformed. What I advocate is the simplest, most sure way to grow any plants. Gee, hate to be so bold, but until someone educates me that there are better ways to grow plants, and sadly that has not happened, the basics are real and necessary.
Adding gravels, pea gravels, sand, gypsum...to our soils and mixing it by hand is off the charts unnatural. The extremes of gardening, are being promoted. Learn about soils, soil life, how in nature soils are made better. What plants need and if anyone says 'plant food' you need to go back and learn why it should not be called plant food! When humans eat more food than we need we get fat and unhealthy. Plants make their own food. Fertilizer, mulch, compost is NOT FOOD for plants. Adding gravels, sand, rototilling to surface level soils is a waste of energy and not at all conducive for plant health.
