I would look first for sun. It's the hardest thing to fix. If you've got an otherwise good site that is shaded by trees on your property, then you could cut them down. You can't, however, change a northeast-facing slope into a southwest-facing slope.
Drainage can be fixed (in all but the most severe cases), but it does take some effort; if you have two sunny locations to choose from, pick the one with good drainage.
If the property has a steep slope to it, consider this as a factor. You could think of it as part of drainage, but gardening on a slope can be challenging.
I think you'll have to fix your soil quality to be ideal for a vegetable garden just about anywhere you place it, and this is relatively easy to fix so it would be my last consideration, so long as you have at least something to work with. (Though I suppose if you've got a spot with incredible soil but poor drainage you might decide to put the effort into drainage instead of soil-building.)
- Access to water. Don't put the garden 500' away from the house where you can't easily reach with a hose. Lugging water back and forth won't be much fun.
- Proximity to house. You'll pay more attention to it if you don't have to hike to get there. You're also more likely to harvest frequently, which means you'll get more out of it.
- Possibility for expansion (if applicable). If you ever think you'll grow the garden (you said "start out", so I'm assuming this may be a possibility), make sure there's room for it. E.g. if you expand to 40x40, will the expansion area be shaded? Will it have good drainage? (E.g. are you siting the 40x20 in between two big puddly areas?)
- Protection. Will the garden site be protected from heavy winds? E.g. if your prevailing winds (especially storms) are from the west, it would be beneficial to have trees or a building not far from the west side of the garden (but not shading it). Even some large-ish shrubs can cut the wind.
My garden gets all day sunshine in the summer, has protection from northwest wind by the house, south wind by tall pine trees (only one of which has fallen into the garden during severe winds...), but
has had poor drainage. I used to get terrible puddling in summer thundertorms. I've fixed it by adding organic matter to improve the soil's water-holding capacity so it soaks in instead of puddling and by giving the water a way to get out of the garden without flooding the beds. I've also added "capture" areas above the garden that can absorb a ton of water before I have problems. It's about 50' from the back door of the house, and I can reach the entire garden area with 150' of hose. It's also easy to go out and cut some lettuce and pull a carrot when I want a quick salad for lunch.