I'm sure I've heard before that the time to harvest potatoes is when the flowers fall off. Is that right or is it more complicated than that? For example, does it differ based on climate or variety, etc.?
Every time I've grown potatoes, there's been a day when the plants as a whole just all collapse. Distressed, I've dug up to see what's going on, and ended up with perfectly nice potatoes. If that doesn't happen to you, then do it before a frost, because potatoes aren't one of those vegetables that is nicer for frost exposure.
(Actually I grow them in a stack of tires, so "dug" isn't quite the right word, but "knocked over" doesn't make sense without the context.)
Given the size difference between "new potatoes" and "ordinary potatoes" in the store, and the historical role of potatoes as an everyday staple, I'm pretty sure there's a huge range of acceptable harvest times. The only downside of too early is that you might have got more by waiting. So don't do all the plants at once until you know you're done for the year - either because the plants have died or because there's a frost coming.
If you harvest when the flowers fall you will get a more tender potato, better for frying and boiling. If you wait until the tops are dried, you get the best storage potatoes, better for mashing and baking. Harvesting over time is easier on you and gives a steady supply.
Also, there is nothing wrong with frost on the vines at maturity, because the vines are useless to the plant at that time. The vines are going dormant like most other perennials do near the hardest time of year. Frost Might make the vines dry out, but you don't need to worry unless you are going to get a hard freeze. You will want to harvest The rest of your potatoes before the hard freezes start.