I have what may be the same problem. Current iMac 32GB ram with Mavericks, Encore CS 5 vrs 5.1.0.342
Exported a 10second video to Encore and Encore reports that the video is too large for the media (25GN BD-R Blu-ray disk). 10GB used for video and 1952GB used for BD-ROM content.
Bug or incompatibility between Mavericks and Encore CS 5.1??