I'm thinking about trying a Myst game, which one should a newcomer start with?

RealMyst for sure.
First, the Masterpiece Edtion has several problems to run because of old Quicktime version (I have it on Steam and I never managed to play it).
Second, RealMyst has an additional Age you can't find on the normal version.
They are the same game, but RealMyst is the right choice to play Myst today.

Definitely RealMyst. Looks quite nice, runs better, and the full 3D makes movement easier.

http://www.gog.com/forum/myst_series/real_myst_or_myst_masterpiece_edition

I prefer the original, realmyst has a lot of issues

I'll go with Real Myst then, but what is the order of the games here on GOG? And, counting the games available, is there one missing from the catalog?

Myst/realMyst/Myst ME
Riven
*Exile
*Revelation
URU
EOA

*not available as digital download anywhere

I'm not in charge of your wallet but if you think you'll like Myst, the bundle price of $10 is pretty amazing - at least you'll have both the original and RealMyst depending on which one works for your computer. Anyway, just a suggestion ;-)

Just bear in mind, if you choose to continue, that there's no realRiven...

This. I wouldn't recommend RealMyst simply because it sets you up for a jarring start to Riven (the best game in the series IMO) because of the step backwards in technology. The 'additional age' is like two rooms. It doesn't really add anything.

I'd recommend Masterpiece edition because by the time you finish it you'll either realise the series isn't for you or you'll be used to the 'slideshow' style and will be able to be immersed in Riven. Ah well, I guess you already bought RealMyst? I hope you enjoy it and aren't put off Riven by the step backwards in tech.