This is proving a little more complex than I had hoped. It appears my first attempt was on the right track. No way around mucking with trigonometry I guess. I'll start with rotation first this time around. The idea here is that while standing at the origin, I rotate the world around the origin so I'm looking in the right direction, and then translate to the correct spot.
I need to keep track of three angles, rotation about x, y and z axes. Let's see. Aha, here's one problem. I've been trying to make two rotations, one after the other, on each three axes. However, it appears that when using glRotate, the vector you give is relative to the absolute coordinate axes, not the relative coordinates after the previous rotation. This explains the odd behaviors I was finding, where in one direction mouse movement worked as it should and 90° to the right or left it made me roll about the axis.
So I need a way to calculate the direction of the new axis after the first rotation. This means calculating x and z components (y component remains 0). The components, for a given rotation will be the cosine of that angle on the x axis, and the sin of it on the z coordinate.
There we go, that's much better. I'm going to need to work out all three coordinates when I add rolling to the camera. Now, we add camera movement. Moving forwards and backwards is easy, since we can ignore camera roll as well. I need to first figure out the vector on which I'm moving (camera's z axis), normalize the vector, and move my speed along that vector.
Yes! There we go. Got the forward and back working correctly at last. As I said a couple of times, the way the coordinates work here is kind of confusing. Vertex positions are relative to your last translation, but glTranslate and glRotate work on absolute coordinates. Need to streamline the camera functions, but I have a good start now.
Next up, I'm going to try to draw some of the variables on the screen, for debugging purposes. One of the problems my current implementation is having is that if you do half a turn going up, all the controls end up wonky. Probably a sin or cosine somewhere giving me a negative number where I'm expecting a positive or something like that. One of the reasons I didn't want to muck too much with trigonometry.