Now, I don't want this post to become a philosophical discourse on fate or teleology. My digression here was to help me assert that D&D, purely as a game, doesn't promote or encourage a story. Story is an optional extra added on top of the game, either by ex post facto pattern-finding (which is, generally, the old school sense of story) or by imposing it on the game beforehand (which is, generally, the newer approach). I think it's important to make this clear, because many gamers assume that all RPGs, by their very nature, including D&D, are "story games." Indeed, many D&D players, primarily those who entered the hobby in the post-Dragonlance era, accept this assumption without question and may have even been drawn into the hobby because they were attracted to the conception of an RPG as "fantasy novel where you're the hero," which is how the hobby was promoted throughout the 80s and how it's largely been designed since the 90s.
With that behind us, let's return to the question at hand: what is the thematic core of D&D, which is to say, what is its organizing principle? The game may not, of itself, tell stories, but, given that gamers almost always find stories in their play, what kinds of stories does D&D support natively? This is where my regular invocation of pulp fantasy rears its head again. I think it's easy to get hung up over the specific elements of this pulp fantasy story versus that one and argue, as many do, that D&D doesn't "model" pulp fantasy very well, because this or that element either doesn't exist in the game or is in fact prohibited by the rules as written. If that's the way you judge D&D, then, yes, I agree that it doesn't do a very good job of being a pulp fantasy game.
I'll return to the modeling issue later, because it's very relevant. For now, though, it's important to realize that there are common elements that undergird all pulp fantasy stories and it's these elements that D&D picked up and built a game around. They're the underlying assumptions that, taken as a whole, (largely) explain why D&D is the way it is and why it has an affinity for certain types of "stories." As I read pulp fantasy, the assumptions D&D takes from it are the following:
- The protagonists are "rogues," by which I mean outsiders generally of low station (though not necessarily birth) who live on the margins of society.
- Said society is generally corrupt, or at least venal.
- Consequently, the protagonists generally pursue personal betterment (whether monetary, secret knowledge, position, etc.) rather than more "noble" goals.
- Despite this, the protagonists sometimes achieve noble, or at least broadly beneficial, goals in the course of their pursuit of personal betterment.
- The world is generally humanocentric, with non-humans relegated to the margins, which is why the protagonists often interact with them.
- Magic is (at best) unreliable and (at worst) downright dangerous (if not morally dubious).
With the exception of the last two entries in my bulleted list, there's a strong affinity between the pulp fantasy story and the picaresque, which is probably no accident. The picaresque is a clear antecedent of "adventure stories" of all sorts and many pulp writers latched on to the Picaro archetype as an ideal vehicle for telling lurid, sensationalistic tales set in far-away lands. I contend that it's here that we find the thematic core of D&D and that the game was written on the assumption that most characters would come from this mold. I see, for example, few alternative explanations for why characters improve in D&D through the accumulation of wealth.
My feeling is that one's level of dissatisfaction with D&D is closely related to one's dissatisfaction with picaresque stories. If your preference is for something more "epic" than a bunch of rogues -- possibly with hearts of gold -- on the make, then you're likely to see D&D as lacking in some way. And many gamers have from the very beginning. Eventually, whether by nature or nurture I can't say, the vast majority of fantasy gamers wanted something more out of fantasy than Picaro in a wizard's hat, which is why we saw the growth and popularity of things like Dragonlance and many of the myriad campaign settings TSR published during the 2e era. But I contend that, in most cases, D&D is simply a poor fit for these settings, because its thematic core evokes the picaresque rather than the epic. To do the latter, one must change D&D in various ways -- and so its publishers have, either by modifying it on a campaign-by-campaign basis (as was commoner in the past) or by modifying it permanently (as has been done in recent years).
I hope there's something coherent in the above, as I'm still working out some things in my head and may well not have been clear. So, to summarize, in case I was indeed opaque: D&D is a game founded on pulp fantasy, which is a modern development of the picaresque. The game was designed on the assumption that the typical PC would thus be rogueish and possibly venal rather than nakedly heroic. Its rules, while not necessarily good at emulating every particular example of pulp fantasy, are built to support this assumption. While D&D is flexible enough to do other types of fantasy, the farther one gets from pulp fantasy/picaresque roots, the more "broken" the game is likely to seem. To this I'll add that the history of post-Gygax D&D has largely been one of trying to "fix" this seeming brokenness in various ways, which has led us to where we are today -- a game divorced from its roots and of limited appeal to people such as myself who prefer its original one.