Now part of this depends on what system you’re using and if it even has any meaningful support for non-lethal defeats without houseruling, but there’s a number of things that can be done regardless of rule set. Communication is important, for example. None of this stuff will have much impact at all if the players aren’t aware that you’re trying to avoid the “lose combat = bad end” habits that I suspect are an artifact of the wargaming roots of tabletop RPGs. If they expect enemies to take no prisoners, they’ll respond accordingly and enemies would naturally escalate in reaction.
Another important requirement is some sort of social contract, an understanding established with the players in advance how it will work out, since that isn’t really part of the typical mindset alongside “don’t split the party” and other advice. In this regard I’ve been inspired a lot by Night’s Black Agents (by Pelgrane Press). In Night’s Black Agents, handling a captured player character is part of the genre emulation of the system (it’s a game about secret agents after all), but it’s simply stated outright that if a character is captured, then A) They will get something out of it, and B) They will get a chance to escape.
In Night’s Black Agents, the thing the PC gets out of it is usually some valuable piece of information, because the GUMSHOE system is built around investigative games and as such, the main reward currency is clues rather than experience points or loot. In a system like D&D, a similar structure might be to ensure whatever encampment or dungeon the characters are held in after capture is also housing something of value, a relatively minor plot coupon of some sort (like a key to a secret entrance to the enemy keep), or another captive who, if sprung along with the captive PCs, can provide some kind of favor or reward. In addition, there should probably be an experience reward for escaping that doesn’t require subduing the entire enemy camp, so you don’t risk ending up in a perpetual loop of adventurers trying to defeat every single guard in a group that already defeated them once and being re-subdued and re-captured.
Ensuring a chance for escape is also important because without it, players will simply treat being captured as a particularly slow and unpleasant cause of death, and that will sort of derail the whole reason for trying this approach. The actual particulars of this might be a little complicated though, especially for groups accustomed to viewing every problem as a battle map, as well in no small part because players (reasonably) are leery of being put in situations that limit the options available to their characters and therefore limit player agency (i.e. railroading). It does, however, offer potential for interesting and different approaches to play, where players might have to scheme and bluff and sleight-of-hand their way to freedom (and the chests containing all their confiscated gear). The important thing is to make sure that there are a variety of possible approaches to escaping and that the party is never left completely stumped due to failing the perception roll to notice the button for the secret door, for example. If a clue is necessary for escape, give it to the players no matter what. Maybe reward a higher roll with additional information, but make sure they get the minimum necessary information somehow.
Another part of ensuring escape (and to establish the benefits of being able to use enemies that are potentially too big for the party to take, which having enemies who want to capture rather than kill can provide) is establishing that when dealing with these sorts of situations, it might be okay to split the party. If the drow have knocked out half the PCs and are too much for the remaining party members, it might be better to encourage the still-conscious heroes to escape and find a way to mount a rescue. Use caution here, though, because you don’t want this to produce a situation where half of your players have nothing to do the whole time. Make sure the players don’t feel punished and try to give as much spotlight time to those plotting to break out as those preparing a rescue. But if you can maintain this balance it provides for a much greater variety of possible escape routes.
Now, onto the practical matters. How do these antagonists go about capturing heroes in the first place? Well the ease of this depends a lot on the game system, and some of them don’t support it very well, which I suspect is part of where the tendency to avoid the subject entirely comes from.
For D&D 5e, it’s actually quite easy to implement in a lot of cases. The Rules As Written already have it taken care of for melee attacks, as any attacker who drops a target to 0 hp can choose to knock the target unconscious instead of sending them into the land of death saves. Many spells could reasonably be ruled by the GM to utilize the same rule (Vicious Mockery or Sacred Flame for instance), and it would not be difficult to add weighted, blunt arrows and crossbow bolts (like those historically used for hunting small game, or for a more dungeon-punk over the top approach for tinkerers, unfolding blunt-impact projectiles like those used by Stun Gun Milly in the anime Trigun) that deal bludgeoning damage and provide the same option for non-lethal takedowns. Plus there are always nets, snares, and non-spiked pit traps.
In Pathfinder or D&D 3.x, nonlethal damage doesn’t come up much (in my experience) except with regenerating foes, in part because the way the rules were originally written were somewhat obfuscating and people just avoided it and potentially developed misconceptions. I know my home group spent many a year neither knowing nor caring to know how it really worked. But the way it actually works out essentially means non-lethal blows delivered at any point in the combat (rather than just the defeating blow) creates a buffer zone where the enemy will be knocked out if they’re damaged into that range but not beyond it into dying. It’s confusing to track until you get the hang of it, but the way it’s supposed to work means that nonlethal damage at least still contributes on a 1-to-1 basis to defeating the enemy—which is not especially clearly stated in the rulebook.
Fantasy Craft—a system that is obscure, but close to my heart—handles knocking out enemies a bit differently. Nearly all blunt weapons deal subdual damage by default, but any weapon (barring special rules) can be switched from lethal to subdual or vice versa with something vaguely akin to a called shot. Now the interesting part is that for your rank and file enemies, the mooks, the goons, the nameless squads of hostile intent, lethal and subdual damage (as well as stress damage) all pour into the same damage track, and they simply are knocked out if they are dropped by subdual damage (or faint in the rare case of being dropped by stress damage). For PCs and their important sidekicks, as well as characters of greater story importance (the henchmen—your Oddjobs, Jawses, Wints and Kidds; and villains—the Trevelyans, Dr. Nos, etc), accumulated subdual damage causes higher and higher save DCs to beat on each subsequent instance of subdual damage or else they gain increasing levels of the Fatigued condition, and after 5 failed saves, they are rendered unconscious. This does a number of things:
First, it means that enemies with the sort of weapons that would intuitively be useful for knockouts will naturally convey to the players that they won’t die even if they get clobbered since it’s baked into the gear description rather than relying on a conscious decision not to kill.
Second, it provides the GM with assurance that the Big Bad won’t go down in a single hit from a sneak attack with a sap, and protects players from the same, but it means characters might be rather difficult to actually subdue under normal circumstances.
All in all, Fantasy Craft is a crunchier system with rules designed for effect in play rather than wargame simulationism, so it’s trickier to use for this purpose reliably, compared to 5e’s “Nah, he’s just gonna knock you out” approach, but it can also be rewarding for a player who wants to develop some extra system mastery and make a character designed around non-lethal takedowns.
Beyond these, there are other approaches, for example that of Alabaster by Scott Gearin, in which the system’s hit point equivalent is not a measure of injury but rather a meter below which a character is Defeated, and Defeated is assumed to be non-lethal by default. Another example is Shadowrun which has parallel health tracks for physical damage and stun.
All in all, there are a lot of ways to ensure characters can be captured rather than left bleeding out on the floor, either within RAW or with tweaks, but the most important part is the social contract between the players and the GM, so everyone buys in with the same understanding of how the game will be working.
