... only because of the weight factor. They're heavy enough to move around when they have lightweight potting mix. Because I have to have areas clear for leaf cleanup and then snow plowing in the winter, I have to move all of the containers out of the way every fall and put them back in the spring, so I have no choice. Plus all of the tropicals that go in and out with the seasons. If they could stay in one spot, I'd probably do what you do.
Actually, I did a little experiment a few years ago with some impatiens in window box planters on my back patio area. The township was installing new pedestrian paths, and they brought in "topsoil" to repair the lawn damage. It was the worst, nasty sand and gravel, yellow brown crap. Just for the heck of it, I filled a couple of my window boxes with that, the rest with ProMix which is my usual brand of potting media. I saw very little difference in growth, since I used Osmocote and did a constant liquid feed as well. The ones in the sand did dry out faster, but the plants grew fine. Basically, the soil is just there to hold them down if you are feeding in this manner, it's almost like doing hydroponics.