I have not used tsMuxer, but do understand its function. And I agree that it is relevant to showing that the PS3 handles your material in a bluray structure.
As you note, Encore has often been shown to prefer demuxed assets. And yes, from your description, it sounds likely that Encore did not re-transcode the file. But something is going on.
I would still do the experiment with the new project. Just export a short (5 minute) m4v/wav from PR.