Just a short and untested idea that's probably been hashed before, but:
Instead of keeping a separate record of what spells that are memorized, make them take up inventory space/weight equal to their level.
So if you're tracking inventory space in a grid system: level 1 spells takes up one space, level 2 spells takes up two, and so on. Or if you're counting weight, make their level equal to some logical unit like pounds.
And to end this post, here's another picture of two wizards arguing: