You'll be fine Kavi! 3' deep beds? What did you use for their structure? Bigger question is; are your beds connected to the original soil? Did you break up the surface of this soil before back filling your raised beds?
What soil did you use to backfill? What have you added?
Worms need decomposed! organic matter to eat and derive energy. You provide that and you will have earthworms next year. Zone 4 is not that death defying for the macro organisms in soil. more about earthworms and dormancy and eggs
earthworms in winter
When there is no decomposed organic material (decomposed is the operative word) earthworms and all other macro and micro organism populations have ways to go dormant until there is decomposed organic matter to eat and/or proper temperatures to wake up.
Did you purchase your worms? I have a problem where I live now in Central Oregon. Haven't seen a single worm. Zone 1B. Pumice soil. After 5 years improving this soil not a single earthworm. Have never known a garden without worms. But on the other hand, I've got soil spiders, ants, millipedes, mycorrhizae fungus...and plants do well.
I lived and had incredible gardens Zone 3 to 5. Full of worms, also clay soil which I love. I miss my clay soils. They don't travel that far down into the soil either to survive...one thing about raised beds is that you are essentially making your garden into 'pots'. Anytime plants are in pots or raised beds with sides made of structural elements (lumber, concrete) the roots are more subjected to cold. Roots, after deciduous leaves are very susceptible to cold. Potted plants are likely to have roots killed during the winter. Are you going to plant perennials?
I wouldn't worry about my worms. I would worry more about my perennial plants (rhubarb, blueberries, raspberries, asparagus, horseradish, fennel bulbs, strawberries, etc.) My asparagus, strawberries blueberries, rhubarb survived these last winters. This very last winter killed my thyme, oregano and mint?! I always have raised beds but I do not use structural elements. Not one single worm ever! Weird.