Using the enboost helps greatly for not reaching the limit however it is not enough to keep Skyrim stable especially not if you run a lot of mods. Skyrim favours Nvidia cards more then the AMD ones and I also noticed this on my 3GB AMD Radeon HD7950 OC card.
Unfortunately I had the same issues that OP mentions.