So...here's the final obersvations/solutions...
Increased the minimum bitrate to 3. Changed the target at 5. tried it at both 23 and 29 frame rate. Tried at at both the interlaced Encore had as default and as none. Looked like crap every time - piano strings were ragged as in the above picture.
Looked at the m2v file with media info. Said overall bit rate was 5.
I even imported as an asset the m2v I ripped from the original "looking good' DVD, which did not like to be imported and needed to be transcoded (and thus looked crappy after all was done)
So this morning tried encoding the footage using Adobe Media Encoder. Used the 5Mbps 'automatic' preset. Worked like a dream. Took no time, imported beautifully into Encore (I just had to re-do Chapter markers and re-link to the menus) and the finished product not only fit the DVD (4.66GB) but looked beautiful. No artifacts. No jagged piano strings.
So...it looks like for some reason or another, sometimes Encore does not do a great job encoding when one changes the presets. It may be due my using a computer that needs an upgrade (hurry up Apple). It may not like my processor or graphics card. Or it may be something buggy in the program. But my takeaway is that if it defaults to a lower (and thus lossy) encode, leaving 500MB or more free, I'm going to be encoding externally from now on...and save myself a lot of time.