The setting you used has a bitrate minimum of 1.5 (very low) and a target of 5. A bitrate calculator looking at program length of 105 minutes suggests a target of 5450. Since Encore had less on the disk when you used automatic, it was likely using a target in that range.
If you still have the transcodes Encore did on automatic, examine one with Gspot or Mediainfo and see what bitrate was used.
I would try minimum of 2.5 or 3; target of 5.4; maximum is a debate in terms of compatibility, bit try 8 for now. Will it fit with the other material? I don't know, but it may give you a better idea what is happening to the quality.
