Once I had some wasps building a nest in a place that I could not tolerate. Before I knocked it down, I rolled up some newspaper and got it smoldering. I held it near the nest in order to 'smoke them out'. I was expecting a general panic of wasps but to my surprise, they calmly 'marched' out of the nest in an orderly line and up the roof. I then knocked the nest into a metal can/bucket and set it on fire.
I never saw them again. I assume they moved on to a new location. It was a pretty new nest, and I am not sure they would behave the same way if it was an established nest. But, if you want to avoid spraying poison on your property, you might want to try this first. Wait until dusk when they have returned to the nest for the night. Create a lot of smoke and blow or fan it over the nest. If you are lucky, this will trigger their instinct to flee the nest. Be careful that you don't set the nest or other things around the nest on fire at this point. For future reference, it's easier to do this before knocking the nest down due to the fact that smoke and heat rise.
Then, put the nest in a something fire-proof like a coffee can or an outdoor fire pit and burn it. You might need some add some paper, kindling and/or other fuel to fully destroy it. Once the nest is gone there will be no reason for them to come back to that spot, but it is possible they could start another nest in the area.