I'm sure I've heard before that the time to harvest potatoes is when the flowers fall off. Is that right or is it more complicated than that? For example, does it differ based on climate or variety, etc.?
Every time I've grown potatoes, there's been a day when the plants as a whole just all collapse. Distressed, I've dug up to see what's going on, and ended up with perfectly nice potatoes. If that doesn't happen to you, then do it before a frost, because potatoes aren't one of those vegetables that is nicer for frost exposure.
(Actually I grow them in a stack of tires, so "dug" isn't quite the right word, but "knocked over" doesn't make sense without the context.)
Given the size difference between "new potatoes" and "ordinary potatoes" in the store, and the historical role of potatoes as an everyday staple, I'm pretty sure there's a huge range of acceptable harvest times. The only downside of too early is that you might have got more by waiting. So don't do all the plants at once until you know you're done for the year - either because the plants have died or because there's a frost coming.
If you harvest when the flowers fall you will get a more tender potato, better for frying and boiling. If you wait until the tops are dried, you get the best storage potatoes, better for mashing and baking. Harvesting over time is easier on you and gives a steady supply.
Also, there is nothing wrong with frost on the vines at maturity, because the vines are useless to the plant at that time. The vines are going dormant like most other perennials do near the hardest time of year. Frost Might make the vines dry out, but you don't need to worry unless you are going to get a hard freeze. You will want to harvest The rest of your potatoes before the hard freezes start.
Depending on where you live you can harvest potatoes anytime. When I lived in zone 5 for example, I'd leave the potatoes in their (raised) beds all winter. I had silt-loam soil and lived near the ocean. It was such a treat to go out and cram my hands into the beds to pull out potatoes in the middle of winter; large or small...by spring I'd pulled them all inside to keep cool in shredded newspaper. Buried in the soil protects potatoes an awful lot. In zone 4 or lower, I'd get them pulled up and nestled in shredded newspaper or sawdust. They'll be fine in a dark garage or root cellar or cool basement stored in this manner for the entire winter.
Potatoes were one of the first things to 'die back' in the vegetable garden. There were white flowers (white potatoes), pink flowers (reds) and blue flowers (blue potatoes). But they started dying back before anything else.
Once the plants start dying the potatoes won't be getting any larger. I've always grabbed new potatoes whenever I wanted and there were always the huge ones. Which I don't think are as good as the 'new' potatoes. Potatoes are one of the easiest vegetables to grow and harvest!
Just one caveat...don't eat any 'green' potato. They are the ones that are sort of exposed on the top of the soil and get sunlight. When you see potatoes exposed grab a handful of soil and cover them. They are starting to produce chlorophyll and as their genus proves, becoming toxic.
Potatoes are one of the best most forgiving vegetables to grow!