In pre-Greyhawk OD&D, dwarves were limited to 6th level as fighting men, halflings were limited to 4th as the same, and elves could progress no higher than 4th level as fighters and 8th level as magic-users. Supplement I loosened things up a bit if a demihuman character had exceptional ability scores. Dwarves with 18 Strength, for example, could reach 8th level as fighters, while Elves with 18 Intelligence could reach 9th level as magic-users. In general, though, the trend was to limit demihumans to 9th level and below (with thieves being a notable exception to this rule).
It's my belief that level limits were intended to serve two purposes. First, since demihumans got a number of abilities that humans lacked (infravision, improved saving throws, etc.), level limits were an early attempt at "balancing" their advantages against humans' lack of same. Now, as I think I've made clear over many months, I'm not a big fan of the balance über Alles school of game design and find it almost always makes a RPG less fun rather than more fun. Moreover, OD&D already possesses a means of balancing the varying abilities of characters -- variable XP charts. If one believes that the abilities of elves, dwarves, and halflings are good enough that they ought to exact some kind of "penalty," why not use XP as a means of representing this?
This is the approach that Moldvay Basic opted for in its particular interpretation of race-as-class. Elves and dwarves -- though not halflings -- require more XP to gain a particular level than do their human counterparts. It's a decent solution, I think, although the treatment of halflings undermines the logic behind it somewhat. Halflings get a number of advantages, most notably excellent saves, and yet their XP charts are identical to those of human fighters. Their main drawback is that their advancement is capped at 8th level, the lowest level limit of all the demihuman races. The halfling thus throws a wrench in the notion that there was any kind of consistent philosophy behind the demihuman XP charts.
The second purpose behind level limits, I believe, was genre emulation. Most of the pulp fantasies that influenced Gygax and Arneson didn't include lots -- or any -- non-humans as major protagonists. Non-humans were rare, exotic beings, far more likely to be talked about but never seen. The idea of a largely demihuman adventuring party (Tolkien's works to the contrary) was likely seem as peculiar. By making demihumans more limited in their progression, I suspect it was hoped that they'd prove less attractive and thus less numerous. The limit of halflings to 4th level, for example, has always struck me as the authors' way of saying, "Well, sure, you could play one of those hobbit guys, if you really want, but they're not really cut out for adventuring."
As I've played OD&D more extensively, I've come to some conclusions about all of this. Firstly, I like race-as-class a great deal and see it as a good way not only to keep non-humans strongly archetypal but also as a way to more implement the variable XP idea that Moldvay didn't seem to have followed through with. Secondly, I don't think level limits are a good way to discourage people from playing elves or dwarves or whatever. If the referee wants to limit the number of demihumans in his campaign, then he should simply do that: limit them. That's what I've done in my home Dwimmermount campaign. Dordagdonar, for example, is the only elf the characters have ever encountered. Even in a large city like Adamas, elves are exceedingly rare and he often raises eyebrows when he meets people who've heard of the existence of elves but never had the occasion to actually meet one.
Of course, Dordagdonar's uniqueness cuts both ways. Elves in my game are generally reclusive and stay out of worldly affairs. For whatever reason, Dordagdonar is different; he breaks a lot of the stereotypes about elven behavior, spending all his time with "ephemerals," as he does. Consequently, I don't plan to limit his level. Like his human companions, he can continue to progress indefinitely, although I do use a different XP chart to represent his abilities. Mind you, in OD&D, level advancement is slow and I doubt the characters in the campaign will ever see much beyond 9th or 10th level anyway, so it's largely a moot point.
In the end, I can't say level limits do a lot for me. I don't hate them or think them an abomination against "good" game design, but I also don't see their elimination as necessarily problematic. Far worse in my opinion is the ubiquity of demihumans, which destroys a lot of the flavor I associate with OD&D's literary influences. Having a party consisting largely of, say, elves bugs me more than the possibility that the single elf allowed in a campaign might one day reach higher than 8th level. But then I'm weird that way.