Clay is a type of soil that is composed of many very very tiny particles, and its nature is to retain water (poor water drainage, causing waterlogging and root rot) when wet, and to form large (hard to break up) clods when dry (hard for plants to expand new roots into). Some plants grow in clay soil just fine, but those plants are resistant to the root rot caused by clay's lack of drainage. Clay is normally rich in plant nutrients, but its tendency to compact and poor drainage cause many plants problems.
The quickest and easiest method to create an area for plants that do poorly in clay is to create a raised bed of non-clay soil on top of clay. In the case of the OP, he could remove the clay soil used to fill his holes and replace it with a non-clay soil filler.
A somewhat less quick and easy method is to convert existing clay soil to a non-clay form of soil. The disadvantage of this method is that non-raised garden beds in areas that are mostly clay soil is that water in the growing area is unable to easily escape through the surrounding clay. The solution to this lack of drainage involves digging drainage ditches/paths (and then possibly filling them with gravel and/or drainage pipes) through the surrounding clay to allow any water to escape. If your growing area is on a slope or hill, the slope may allow water to escape without digging drain paths.
To quickly convert clay to a less clay like soil, you may use one or both of two methods: 1) Mix in any amount of organic matter, such as compost, into the clay or 2) Mix sufficient amounts of sand into the clay (at least as much sand as clay by volume). In both of these cases what you are trying to do is "uncompact" the clay, or in other words, allow air pockets to exist in the clay soil. If you use the "add sand" method, mixing insufficient amounts of sand (less than 50% by volume) will actually make the situation worse, since the soil will become heavier (sand is heavier than clay, making the soil compact faster). Adding insufficient amounts of organic matter, on the other hand, may not help 'fix' the clay sufficiently, but will not make soil compacting worse.
The air pockets you create in the soil will do two things: allow growing roots to penetrate the soil more quickly (allowing faster plant growth), and allow water to drain through the soil faster, removing the tendency of clay to cause waterlogging and root-rot. Ploughed or rototilling clay soil (or clay soil with insufficient added sand) will quickly revert to a solid clump (without air pockets) when wet, while organic matter or sufficient amounts of sand mixed with clay resist compaction of the needed air pockets.
When selecting the type of organic material to add to your clay, remember that un-composted (raw or green) organic matter such as wood chips, lawn clippings, or fresh manure cause changes in the soil chemistry and pH that may create problems for growing plants (see the rules for judging soil type below).
The slowest but easiest way to convert clay to a less clay-like soil is to encourage the growth of any plant that is willing to grow in the clay soil, and then occasionally rototill or plough under those plants into the clay soil. Those roots of those clay loving plants break up the clay and add organic matter when rototilled into it, so this method can be considered a slower method of adding compost to the clay. When using this method, adding a layer of an inch or two of sand or gravel on top of the clay will allow plants that cannot normally grow in clay to thrive. This is because some or most of their root system is above the clay soil, but the tips of the roots can still pull nutrients and water out of the clay.
Rules for judging the soil type you have:
1) Physical composition: How easily the soil becomes compacted. Affects soil water retention, water drainage, and ability of roots to easily penetrate the soil. Clay is at one end of this scale (few air pockets, roots penetrate with difficulty, retains water, and drains poorly), sand is at the other end (high water drainage, lots of air pockets, roots penetrate easily), and loam (ideal soil) is in the middle. different plants are happiest with different soil types; some like sand, some like clay, but most are grow best in loam.
2) Nutrient levels: Adding organics (mulch) or fertilizer to soil tend to also add plant nutrients to that soil. There are exceptions to this; for example wood chips reduce nitrogen when breaking down, which is why they can be used as a ground cover to inhibit weed growth. Almost all plants require Nitrogen, Potassium, and phosphorus, but there are dozens of other micronutrients plants also need from the soil in lesser amounts. Organics also tend to improve water drainage (they retain water in sandy soil and allow better drainage in clay soil). Although few plants have problems with too many nutrients in soil, adding the 'wrong' organics or fertilizer can cause pH problems (see below). Lots of organics in soil that have broken down completely tend to create the 'black' soil that you often see as potting soil.
3) pH of soil: this is the acidity or alkalinity of the soil. Adding raw manure or green mulch (grass clippings, leaves, etc.) will cause soil to become more acid; the process of organic breakdown creates acid. Adding lime or limestone to soil will cause it to become more alkaline (reducing acid as the lime breaks down). Any plant has a "preferred" pH level that it really likes best and adding lime or organics to adjust the pH allows you to tailor soil acidity to match the plant you want to encourage. Adding raw or green organics that have not broken down (composted or aged) sufficiently will tend to 'acid burn' plants until that raw or green organic has broken down.