The clay itself isn't going to break down due to decomposition because it is made up primarily of inorganic material - minerals and such.
If you plan on doing the lazy compost method, where you pile organic material up and forget about it, then it won't much matter if there's clay other than the fact that you'll have a lot of clay in that pile. Eventually, the organic material will break down, though you may find that weed seeds prosper in that environment because it is unlikely to heat up enough to kill them off. That's a big mess to deal with. Not worth the long-term hassle in my opinion.
If you plan on do the actively-involved compost method, where you turn the pile and ensure it is moist and encourage aerobic decomposition, then if there's a lot of clay, you may grow to hate turning the pile and it will likely revert back to the "lazy compost" pile that doesn't heat up much. And there you are with a pile of clay and eventually some rich material veins running through it (and likely weeds).
On our farm, I've thrown clay clods along with weeds and turf into a pile along with horse manure and it has heated up just fine but I'd still hate to turn that by hand. I pick it up with the tractor bucket and toss it that way (it's easy when you can just scoop it up and dump it with machinery) but in my compost bins that are made of pallets I avoid putting clay in it if I can because I actively turn these with my pitchfork and as much as I love a farm workout, clay is (as you said) heavy and I lose my gusto for the turning.
My recommendation is to remove as much of the clay as you can from the pile. It'll break down much faster that way. It will be a pain to deal with it once rather than a pain to deal with multiple times.
With regard to the worms - if it heats up considerably (which is what you want), they will move away from that heat down into the cooler areas. I'm assuming your bin is on the ground and isn't one of those tumbler types. If it is a tumbler, the worms will die if it happens to heat up enough and they have nowhere to go.