I can understand the frustration here - getting supplied files to the correct specs can be an unbelievably frustrating experience.
It should be perfectly permissible to use both HD & SD on one disc in Blu-ray. The difficulty, as you have discovered - is always the file type needs to be correct. The best place to begin is to know what your spec legal options are (so we can discover if the issue is in Encore (in which case we need to find an alternate solution) or in what you are trying to attempt. If you are going to be doing a lot of this, I can warn you in advance you will end up collecting some very eclectic software along the way as utilities seem to help no end. You have an issue right there though, as Mac are just not Blu friendly. Worry about this later and I only mention it as you earlier dismissed the TMPG tool as a suggested workaround for being too expensive which at $100 is - for what it is capable of - incredibly cheap.
I know I bought it just to attempt one file format conversion, which it did a pretty good job of that time although did let me down on a completely different project but I ramble on and digress, as I am prone to doing. Sorry.
So - back to the specs. Blu-ray has several legal frame rates & sizes as follows:
i/. 1920x1080 (aka"Full HD") allowing frame rates of 29.97i, 25i, 24p and 23.976p
ii/. 1440x1080 allowing frame rates of 29.97i, 25i, 24p and 23.96p
iii/. 1280x720 allowing frame rates of 59.94p, 50p, 24p and 23.976p
iv/. 720x576 allowing a frame rate of 25i
v/. 720x480 allowing a frame rate of 29.97i
Better still, you also have 3 different codecs available to you
1 - H264/AVC as .264, .avc & .bsf (the latter may not be in Encore)
2 - VC-1 (as .wmv - this is the old Windows Media Video 9 type)
3 - MPEG-2 - not (and this is important) configured for DVD though and available as .m2v, .mp2s & .mpg.
The first thing to take note of I think is that the SD resolution needs to be interlaced - you cannot use 23.976 or 24p, and you most definitely cannot use 25p or 29.97p - everything in SD must be interlaced so all footage in SD NTSC must therefore be at 29.97, ideally upper field first but if source was all BFF then it is vitally important to not change this without checking the results on an old style glass monitor, as LCD panels will not reveal interlacing artefacts and if you get those in the final encode it will ruin your day.
I am banging on about this at length because unless I am reading you incorrectly, in post 7 above you detail a file that is still giving issues, and it is an NTSC SD piece at 23.976 progressive scan - this is just not Blu-ray legal.
The best rule of thumb is to think of it this way -
A - Progressive Scan is always 23.976/24fps.
B - Anything in old style PAL or NTSC resolutions will be interlaced.
Yes, there is an exception but it is so rarely seen it can by & large get ignored.
Your biggest problem is the wide array of source file resolutions - it looks like some were MPEG-1 or half size MPEG-2 (there were some odd things done in early days of DVD) and the easiest way out is to plan the assets out in advance - it seems to me a certain amount of re-encoding may well be necessary but this will depend on your original source. Are you still having trouble? I ask as I am late to this party
NB - I do find it very strange that Stan was able to import MPEG-2 DVD encoded files for use in a BD project with no issues for re-transcoding (first reply above). I would take a wager that it would get re-transcoded during disc build, or may even trigger a fail although I could be wrong.