From what I can tell, the maximum number of menu buttons (per page) is 18 in HD/BluRay or 36 for DVD.
There is also a limit of 1GB for all menus.
I replaced my motion menu graphics with a still image, reduced the poster frame buttons to text only and still have corrupted buttons in BluRay.
I'm beginning to think that problems like this are why Adobe dropped Encore from the cloud.