I know this is going to be somewhat unpolished; it's a starting point, not a final declaration.
So, recently, I've started saying "More Pratchett than Tolkien" to describe my approach to D&D (and other RPGs, but it seems to be more an issue with those who prefer older versions of D&D). But what do I see that as meaning? I've got a few points on this; mostly what I perceive as being the core parts of a "Tolkien" mindset of game worlds, versus a "Pratchett" mindset. Note that I am not ascribing these opinions specifically to JRR Tolkien, but rather than Lord of the Rings is frequently invoked as defense of them. After this, I'll avoid referring to this as "Tolkien" or words derived from that; my statement is pithy, better for a signature than discussion.
1) Racism as a necessary component
One of the primary objections to playing a "humanoid" character is that the character will face insurmountable racism; the 1e DMG says "So unless the player desires a character which will lurk alone somewhere and be hunted by adventurers", before going on to explain that one might want to play a gold dragon, because it can look human, but that no gold dragon would want to do this. The games will often include something like a "Basic Acceptability of Racial Types" table; 1e has one in the PH to cover humans and demi-humans, and the DMG has one to cover most types of humanoids in their interactions with each other; Hackmaster includes it in the PH. With this, it is argued, the game wouldn't be "realistic".
Leaving aside the conception of a world which must, necessarily, include often violent racism, it also shows a lack of imagination, or consideration of the nature of the world presented in D&D and similar games... while many different species live in the world, Good and Evil are real and quantifiable things, and there are gods who can communicate relatively freely with their worshipers on the Prime Material (if they don't live there themselves). A view where violent, unexamined racism can leave one still aligned as "Good" is one that is alien to any understanding of the word "Good" that I have, much less the definition of "Good" as laid out in the 1e DMG. "Basically stated, the tenets of good are human rights, or in the case of AD&D, creature rights. Each creature is entitled to life, relative freedom, and the prospect of happiness. Cruelty and suffering are undesirable." (p 23) How is this compatible with the idea that Good people can freely kill non-combatants of "evil" races?
A particularly evident argument of this is seen in half-orcs (and, in Hackmaster, half-hobgoblins). The 1e PH states that "some one-tenth of orc-human mongrels ore sufficiently non-orcish to pass for human." (p 17). Hackmaster has "Note that even where hobgoblins are acceptable, half-hobgoblins are mistreated and disliked by the locals. Half-hobgoblins share a special bond with half-orcs,alongside whom they are often persecuted as beastly monsters." (HM PH, 31) Despite both being specifically being called out as being non-sterile hybrids, the only consideration is that all half-humanoids will "will always have an human mother and have been born and raised in human lands" and be "The offspring of questionable parentage and violent couplings" (HM PH, 31); none will be, for example, the offspring of a mommy sil-karg and a daddy sil-karg who love each other very much. Hackmaster mentions that "The city of Prompeldia has developed a sizable sil-karg quarter that is threatening to become large enough to count as its own culture." (HM PH, 31). But all will be "The offspring of questionable parentage and violent couplings."
A Pratchett-style view of the game does not pretend that racism does not exist, but rather disputes the idea that it is insurmountable. The dwarves and trolls of Ankh-Morpork do begin with hatred, but they reach an uneasy peace when forced into proximity by the city. Sam Vimes, like many in the city, is riddled with prejudices about non-humans and the undead, but he overcomes them as he deals with them as people. Some of this is inspired by Carrot, the human raised by dwarves, who still considers himself a dwarf in many respects... but also respects all people, learning their names and treating them as citizens, not as trolls or gargoyles.
2) Race as unbending identity
"Humans with pointy ears"
is frequently invoked when you have a non-human character who is
perceived to act unlike their stereotype, or one of the acceptable range
of stereotypes. Are your dwarves "not dwarven enough?" Does he like
wine instead of ale? Does he shave his beard? Is he a dagger wielding
tunnel rat, instead of a chain-mail clad warrior with an axe? You're
playing a short human, not a REAL dwarf.
This also tends to get
invoked when dealing with traditionally hostile non-humans. An elf or
dwarf might violate their racial alignment, but a good orc? Unheard of.
It's argued that it is impossible for any of the "humanoids" to be other
than their Monster Manual entries, despite that never really being
written anywhere, and several counter-examples, especially from 2nd
edition (such as the Monster Mythology deity who prizes his good bugbear
converts). Gary Gygax, revered by many old school gamers, even stated "The old addage [sic] about nits making lice
applies", referring to noted proponent of genocide John Chivington's
statements about Native Americans. As I have argued before, Dungeons and Dragons owes a lot to Westerns,
and the attitude towards humanoids in a game tends to mirror the
attitude towards Native Americans in contemporary Westerns; in Chato's
Land (1973), you have a half-Indian protagonist, while Dances with
Wolves (1990) presents the Native Americans as the sympathetic
characters.
One objection to having the possibility of non-evil humanoids is the idea that a DM may use these as "gotchas"... "You killed all of these orcs but they were secretly good so now you're all evil!". But this argument fails, in my opinion, as one could also use that as an argument against DMs... "We found a vorpal sword in a crypt guarded by a skeleton with no legs!"
A more Pratchett-style game recognizes that the
stereotypes exist, and may be the norm, but they are not the only way
characters of that race can be played. Pratchett's dwarves are uniformly
male-presenting, with beards and axes and iron boots, with one's actual
sex being so concealed that the beginning of a dwarven courtship is
"find out if the person is the correct gender." In Feet of Clay,
however, we're introduced to Cheri Littlebottom, a dwarf woman who
doesn't want that. She wants dresses and jewelry, lipstick and high
heels welded to her iron boots. It is noted to be unusual. Other dwarves
react with disgust... but it also doesn't make her impossible to exist
as a character. Other examples include trolls who join the watch,
vampires in the Überwald League of Temperance who abstain from
intelligent blood, and golems who buy their freedom to become their own
selves.
3) Only certain races are proper.
Some of this ties into the distinctions drawn between humans, demi-humans (elves, dwarves, gnomes, and halflings), semi-humans (half-elves, half-orcs, half-hobgoblins), and humanoids (orcs, goblins, gnolls, etc.). In the older game mindset, players should only be humans, demi-humans, and semi-humans; anything else is too weird or outlandish. For this, I will set aside the humanoids; I feel the objections to them are largely addressed above. This is addressed to what are often seen as the more exotic races... dragonborn, warforged, tabaxi, and the like.
The older mindset tends to describe these are unnecessary. I've even had someone describe the non-evil drow, minotaurs, and other uncommon races described as "played out", as if the dwarves with beards and axes griping at the elves of the forests isn't itself cliche at this point. The small stable of relatively human races is seen as a bonus... provided they don't go outside "appropriate" for their species, that they're human-like is enough. Orcs and humanoids are rejected as above, but there's also examples like Lizardfolk (who have always been Neutral, if cannibals) who are part of the world, but considered too out there for PCs.
The Pratchett-inspired game eschews this; while golems are unusual in Ankh-Morpork, and certainly powerful (more powerful than you might want in a low-level game), Warforged are a similar idea, but rendered playable. Tabaxi represent a different kind of character than "short and resistant to magic in some way". Dragonborn evoke the namesake of the game, again providing a different sort of character to play. That these exist don't invalidate the options to play elves, dwarves, and halflings. A Pratchett-inspired game will have a more cosmopolitan approach to races, both in terms of racism (as above), and in terms of options.