This has recently become salient in my world, again.
Internal candidates raise all kinds of issues – information asymmetry, historical baggage, sense of entitlement, and the like. There's a really basic awfulness in saying 'no' to somebody internal.
Saying 'no' to anybody is unpleasant, but it isn't so bad with people you're unlikely ever to see again. You can be vague and relatively quick, and emerge soon enough with both parties' dignity intact. (That doesn't always happen, of course, but it should.) Once the deed is done, it's done, and you can get on with your work.
But with internal candidates – especially those who had a fairly realistic shot, and who harbor a sense, fairly or not, that the job is owed them – it's much harder.
On a basic level, you know you'll see them again, repeatedly. In fact, you'll have to continue to work with them, and hope to get their best from them.
Depending on the candidate, there may also be a disappointed cheering section of friends and allies, some of whom will hold the choice against you. (I've also seen people bad-mouth an internal candidate behind closed doors before the selection is made, then rally to hir side afterwards. So it goes.)
If the choice comes as a surprise (at least to the candidate), you may get the barrage of 'why' questions. Depending on the real answers, these can be easy or hard to address. If it's a relatively bright line credential, you're home free (“your doctorate isn't finished yet”). The disappointment will still be there, but it's really not personal. But if the reason is more of a judgment call, it can be tough to put too much out there without forever poisoning the well.
From this side of the desk, how someone handles rejection tells me a lot. I've seen candidates express disappointment in the moment, but then (maybe after a weekend) get right back to doing what they do well. These are the ones who sow doubt about the selection. More commonly, I've seen candidates intimate that darker forces are at work, that there will be political hell to pay, or that the rejection is 'typical' of (fill in the blank). In these cases, I'm quickly reassured that I made the right call. If the response is selfish, then I have a sense of what I would have been in for.
Having been through this several times now, I'm increasingly convinced that there's a common expectations gap. A non-trivial number of people think of internal postings as spoils, or as rewards for having paid dues, or as something for which you take a number. In other words, they think of them as entitlements, and react to rejection as they'd react to theft: something was taken from them that was rightfully theirs.
Simply put, I don't buy it. Nobody is 'owed' any more than fair consideration. Positions aren't rewards. They're bundles of tasks that the institution needs done as well as they can be, within existing constraints. Sometimes the best candidate is also the next one 'in line,' and that's great. But sometimes not. If a newer hire has a skill set better suited to the job at hand, and has managed to inspire confidence that s/he gets stuff done, then forcing hir to 'take a number' is wasting a valuable resource. It's running the institution for the employees, rather than for the students. It's a category mistake.
Turning down the 'next' candidate carries risks. Flight risk is an obvious one; I've seen heirs apparent get passed over, howl in righteous outrage, and announce a new job elsewhere a few months later. I don't blame them for that; what one college needs at a given moment may be very different from what another one needs. If what you have to offer is out of sync with what your college needs, then finding a place that needs what you have makes a lot of sense. But I wouldn't make a wrong choice just to avoid flight risk; if the fit is wrong, it's wrong. And I have faith – rightly or wrongly – that the world's talent pool is big enough that nobody is really irreplaceable.
There's also the constant churn of rumor, for which difficult choices provide tempting fodder. All I'll say to that is that I've found over the years that there's really no appeasing the gossips. Give some what they want, and you'll magically create others. Worse, you'll embolden the ones you appeased, so they'll crank up the pressure later. There comes a point at which you just have to accept a certain amount of rumormongering as a cost of doing your job. With rare exceptions, these folks have exactly as much power as you give them. Shrug them off.
None of which makes it easier to have that conversation, knowing that the disappointed candidate will still be here next week, and the week after that, and the week after that. So it goes.