Can you give me more info re the loop point limitation? Even if that were true, a loop point is the time before the part of the menu that repeats, which can be longer.
Your problem, in any event, is the 1 Gig limit for menus.
I wonder if you can reduce the size of the menu renders by changing the default transcode settings. But those cannot be changed in an existing project, only set new, and for DVD, all you can do is reduce the max bitrate to 6. So I'm not optimistic about that.
You can try to transcode the menus outside Encore. Encore will re-transcode any menu that has anything other than highlights added or if it is not to spec. I don't know if Encore will re-transcode simply because of a bitrate setting.