Fraps not working.

I recommend using OBS for recording or even ShadowPlay, if you have NVIDIA GPU (Then again, OBS can do the same job as ShadowPlay).

FRAPS hasn't been updated for like 4 years.

Or you could give ShareX a try!

Fraps only works with specific renderers, from what I recall. I think GOG sets its releases to "overlay" by default. To change it you have to open one of the conf files in the game's folder, it should be called something like dosboxDukeNukem3D.conf. Look for "output" and replace whatever it is set to it with opengl or openglnb. I *think* that should work (although I'm not 100% certain Fraps supports OpenGL, to be honest).

Anyway, for capturing video from DOSBox I actually recommend using its built-in recording feature. Note that it does not capture MIDI music, though (it will capture AdLib music, though). And it actually captures the raw video data of the game in its original resolution, it will be up to you to upscale it in a video editor to HD if you need that (which you certainly do if you want to upload the material to YouTube).