The files will not take any more space that they would when doing the transcode at build time. When an asset is not to specifications, Encore transcodes it/re-transcodes it to a folder inside the Encore project directories (under [Encore Project]/Sources/Transcodes), whether the transcode occurs during the build or using Transcode Now.
You may receive errors due to low disk space, and they may not be correctly identified as due to disk space. And yes, Encore uses much more space than the size of the Bluray to build it.
mp3's cause a variety of problems for DVDs/BDs. They may work just fine too, and it is often difficult to tell the difference.