If you get the seeds from the same place all the time, it could be the variety, though that usually wouldn't explain the lack of "fruits", which is more likely temperature or nutrition. We grow most everything in raised beds, and I am always adding compost. It's like an earthworm factory in there.
They need hot. Hot, hot, hot.
We always start them inside, then transplant as soon as practical. We have a relatively short season, so they are usually planted through holes in black plastic, increases the soil temp. We lived in Oklahoma City for years, you just stuck them in the ground and got peppers. Now in Eastern Washington, much cooler, and thus harder to grow. Even in a good year it we don't get the yield we got in the South. If that is an issue, you could always make a plastic tent with some pvc, or maybe some old windows and get temps higher.
Could be too much nitrogen - or not enough phosphorus / potassium, they get a little leafy with not many peppers if the nitrogen is too high. If you have the space, or some containers, you could try putting them in a different soil, see if the result is better.
I am spraying liquid kelp on the the leaves this year, and I have some peppers already starting up, my little experiment with foliar-feeding. The sprayed ones do seem to be doing better, both tomatoes and peppers.
Should be able to find a guide to tell you how much to fertilize with the size of your garden - over fertilizing could keep them from setting fruit.