Tukukkk

In a recent game at my table, the cleric wanted to use his shield to pin a ghoul against the wall in order to stop it hitting people and make it easier to hit.

The player was thinking of it like a grapple or pin, but those mechanics didn't make sense -- the grapple only stops a creature, but doesn't really inhibit its ability to attack or make it easier to hit; and the pin is only available with a feat the player doesn't have.

But it was a good story beat that I didn't want to refuse, so in the moment I just mechanically called for a shove to knock the ghoul prone, and in the narrative he had the ghoul pinned against the wall with his shield instead of actually prone. It wasn't exactly what we wanted, but it was the closest thing I could come up with.

Is there a better way to mechanically represent crushing your enemy against a surface?

Prone was an excellent choice

You said that the cleric wanted "to stop [the ghoul from] hitting people and make it easier to hit." The prone condition has the following two applicable features:

• The creature has disadvantage on attack rolls.


• An attack roll against the creature has advantage
  if the attacker is within 5 feet of the creature. Otherwise, the attack roll has disadvantage.

It might seem like the "prone" condition doesn't apply because the ghoul is not splayed out on the ground. But keep in mind that "prone" does not always mean knocked down: it's a spectrum of conditions that make movement awkward and inconvenience a creature at short range (while making it harder to hit at a distance). As an example, consider the following rules on Flying Movement (PHB, p. 191):

If a flying creature is knocked prone, has its speed reduced to 0, or is otherwise deprived of the ability to move, the creature falls unless it has the ability to hover or it is being held aloft by magic, such as by the fly spell.

This text indicates that a flying creature can be "prone" while still (supernaturally) flying, which wouldn't make much sense if "prone" only had its traditional meaning.

Jeremy Crawford has described the various ways that the Shove action can be interpreted as something other than a "shove" (like "a throw, slamming into the other person, and so on.") Similarly, "prone" condition could cover a number of conditions that are analogous without actually requiring the creature to be lying down on the ground.

There are other options

If the Cleric was trying ensure the ghoul was not only inconvenienced but also trapped, he could have tried to both Shove the ghoul prone and Grapple it. For example, if he had the Shield Master Feat, he could use the Attack action to grapple the ghoul, and then a bonus action to Shove the ghoul into the wall with your permission that this would mechanically make it prone. This would stop the ghoul from simply using half its movement remove the prone condition.

But if the cleric didn't have a Feat or special ability that would permit him to render the ghoul restrained via an Action, I'd be hesitant to let him do so. You'd essentially be giving the cleric the ability to do with one attack what other characters can only do with several (or a Feat). If the cleric had only one attack available (as most classes of cleric do), I think that your decision to declare the ghoul to be "prone" was completely appropriate.

The condition the cleric was trying to describe was a combination between restrained and grappled:

Restrained:

• A restrained creature’s speed becomes 0, and it can’t benefit from any
  bonus to its speed.


• Attack rolls against the creature have advantage,
  and the creature’s attack rolls have disadvantage.


• The creature has
  disadvantage on Dexterity saving throws.

Grappled:

• A grappled creature’s speed becomes 0, and it can’t benefit from any
  bonus to its speed.


• The condition ends if the grappler is incapacitated (see the condition).


• The condition also ends if an effect removes the grappled creature from the reach of
  the grappler or grappling effect, such as when a creature is hurled away by the
  thunderwave spell.

Using official rules, you can take an action to grapple an opponent:

When you want to grab a creature or wrestle with it, you can use the
Attack action to make a special melee attack, a grapple. If you’re
able to make multiple attacks with the Attack action, this attack
replaces one of them. (...) Using at least one free hand, you try to seize the target
by making a grapple check instead of an attack roll: a Strength
(Athletics) check contested by the target’s Strength (Athletics) or
Dexterity (Acrobatics) check

Officially, shoving or grappling is probably what you want to if you are sticking to official rules.

If you were to homebrew something, I would suggest that the character could make a Athletics vs. Athletics/Acrobatics check to try to pin them against the wall, and then grant disadvantage on attack rolls for the ghoul, as it is difficult for it to attack when pinned, and then give attacks targeting it disadvantage, as you are blocking it. Its speed would also become zero.

This is semi-balanced, as it is like the grappled condition, with some advantages and some disadvantages.