We live in California City, CA. High dessert with your exact same problem. It is driving me crazy. When we first bought the house, five years ago, the large lot was a blank slate.
We had 11 palms, etc. Brought in, put in drip irrigation to about 10 dessert pines and lots of other xeriscape plants. Did a focal point of dry stream bed with large river rock and had 1/2 to 1" granite gravel spread on the dirt along with areas of crushed brick for interest.
It looked nice for about a year, until we had a huge dirt storm like they have in phoenix. It almost covered the front yard with fine dirt. In some areas you could hardly see the gravel or brick. Our neighbor (who was born here and does nothing to their yard) said it doesn't take the dessert long to reclaim itself, so true.
I'm glad we didn't let the landscaper (who we didn't hire) talk us into the expense of putting down plastic or landscape cloth. All the weeds we have are from the dirt that is blown in consisting of tumbleweeds, rabbit bush etc. Seeds. They germinate whenwe have any rain in the spring.
All this was a huge mistake and if we could start again, we'd just have a small lawn to mow and water and do some decomposed granite that could just be blown off, not the nightmare of gravel.
We, too, blow off the gravel on a windy day. I have washed more gravel by shoveling it into a wire mesh wagon and spraying it with a hose and dumping it back on. Back breaking work, way more than mowing a lawn.
Too, it never looks very good, not like a freshly mowed lawn. It almost always has some dirt that constantly blows in on the gravel. People do give us lots of compliments on our oasis low maintenance yard, ha!!
Hope I have discouraged some poor new home owner from making the mistake of putting gravel on a yard in a windy, dusty dessert.
Don't know if I helped a lot with the original question, except to identify with it, lol. If a landscaper could come up with a solution to all of this and offer that service, we'd be the first to hire him.