In Encore bluray (so far), there are no final observations/solutions! Yah, I know what you mean. Get it out the door.
Have you tried tell Encore to use AME as the transcoder? This would test whether it is something unrelated to the settings (Encore passes them to AME).
I'm not convinced you found the right setting in Encore, but you tried the right kinds of things.
In any event, many users would agree that transcoding before bringing into Encore is the right/best way. That is certainly your result.