Encore may set the layer break correctly if it finds a good place, but there are numerous complaints about it being unreliable. In your case, it is probably not able to find a good break point.
The workarounds I find here in the forum are all Windows/ImgBurn.
You may need to change your chapter points and/or file sizes/lengths, so Encore can find a point easier.
Also, use DVD+R, not DVD-R. According to Neil Wilkes, -R is more exacting on the break point.
Please post a screenshot of your project panel where we can see the file sizes and transcode status.