It is difficult to come up with something that has a strong effect on peppers and not tomatoes. Don't expect your fruit to be as big as store-bought bell peppers. As with most fruit & veg, most varieties simply do not get that big. (Other types like anaheims, jalapenos, etc. are more like the store varieties in size).
Peppers love the sun. Yes they need water but don't water log them. I give them lots of water but it is >100F outside as I type this (they dry out quickly). As for nutrients they want a well balanced fertilizer - comparable to tomatoes. I've just looked in Rosalind Creasy's "Edible Pepper Garden" and she says "well balanced" is the key. They particularly need calcium and phosphorus, but too much nitrogen will result in lots of foliage and premature pod drop (i.e. don't go overboard with the nitrogen).
Are there any particular symptoms such as yellow patterns on the leaves? That could be a symptom of Tobacco Mosaic Virus (TMV); or a particular kind of wilt/insect. The best solution for TMV is to keep all smokers away from your plants.
Most ailments that Creasy lists (eg. the various wilts and fungi) also affect tomatoes. Could you have a nematode problem?
If you think you do have a fungal or nematode problem (other than TMV), you could try growing them in a large pot. Peppers in pots generally don't do as well but you should get fruit, but make sure you fill it with store-bought compost and nothing from your garden. That should isolate them from anything that might be in your soil. You could sterilize the compost for good measure if you think you're being sold dodgy merchandise.
On average I find the plants generally do better than seeds, although there's less variety available. They have a head start and you don't have the random chance of germination.
Where are you getting your seed? My supplier must have well over a hundred different pepper varieties so "I've tried almost every variety of seed" is an exaggeration :-) Could it be a bad supplier? Less likely with a specialist nursery, but more than likely with a grocers, and possible with the Lowes/Home Depot type place.
One final thought: Seedlings dieing within an inch or two could be "damping off", although usually it occurs before the first proper leaves come out. This is caused by fungi and oomycetes in the soil which thrive in dark, humid conditions. The solution is to use sterilized starting soil; start the seeds in a bright, well-ventilated place; and allow the soil surface to dry between waterings. Also delay fertilizing (actually I usually only fertilize once, after they've been transplanted).