Since the mod scripts remain in your save file after removal, to my knowledge, you shouldn't install mods for entirely trivial reasons. Mods which only replace texture or audio files don't require an ESP, but I'm unsure if it's possible to replace any game variable (e.g. the amount of XP gained, damage done/taken, health values, buff duration etc.) without using the said extension.