The term comes from coats of arms.

In heraldry, en abyme names the centre point of a shield — abyme being an archaic spelling of abîme, abyss. And a shield that bears, at that centre, a smaller shield is described as being en abyme. When that inner shield repeats the outer one, the design contains its own portrait, which in principle contains its own portrait, which in principle contains its own.

André Gide borrowed that vocabulary for literature in an 1893 journal entry, and the borrowing is exact: a work containing a reduced model of itself.

The distinction people miss is the one that makes the term worth having. A story within a story is not mise en abyme. That is a frame narrative — nesting, and as old as storytelling. Mise en abyme requires the inner work to resemble the outer one: to mirror its structure, its situation, its themes, its problem.

A play performed inside a play, which restages the crime the outer play is about. A painting hanging in a room that shows that room. A novel whose protagonist is writing a novel that is going badly in exactly the way this one is.

So nesting is structural. Mise en abyme is commentary — the work turning briefly to look at itself, and letting you see it do so.

What it produces in a reader is a very specific and slightly uncomfortable effect: you stop being inside the work and become aware of the work as a made thing. Which is the opposite of immersion, and is why it is used sparingly. A story that reflects itself has admitted, mid-sentence, that it is a story.

And the heraldic image carries one implication the literary term keeps. The abyss in abyme is not decoration. A shield containing itself containing itself does not stop — it recedes, at full speed, into a point. The device is named for the direction it points, which is down, and forever.