How large is your category list?
Honestly, I've never encountered this problem before and there is no "work around" as you can tell. I can't imagine why you would be seeing such a load from what is essentially a cahced listing of your categories and associated data - to not cache it would mean having to run complex queries on every page load which would likely result in even worse performance.
That you are simply not loading all the data to begin with won't get you very far.
Please do not PM me for support or sales questions. Thank you for your understanding.