tl;dr, if you use damaging skills in one map for too long, you'll eventually reach a point where your FPS goes to pomegranate and you crash out.

I believe the issue to be with the method the display uses to hide multiple lines of damage. Instead of unloading old damage lines, they're just interrupted when a new damage line is called.
Since a replaced damage line doesn't finish its animation naturally, it never unloads from memory.

That said, the memory is freed up by changing maps in-game; I assume because map change unloads the bugged damage numbers like it would unload normal ones.

Steps to reproduce:
1. Enable the "Blade Effects" damage effect option
2. Continually use any damaging skill from any class in any map (skills that deal multiple lines of damage work best for reaching the critical point)
3. Eventually your client drops FPS drastically and crashes out

Kanna's Soul Shear skill is a very quick way to encounter the bug, since it can deal 15 lines of damage to up to 15 targets each time you use it.

Already posted this issue to the official MS bug report board here.

Edit: Look at all the damage lines Soul Shear spits out in "Cascade" display:
Now imagine all but maybe one line per target never leaving memory.