Torque towards Quaternion.

Well, a quaternion isn't exactly an absolute rotation in itself. It describes the difference between two rotations in a way that avoids gimbal-lock. In any case, you probably want some kind of damping algorithm- I assume you want it to slow down and stop at the correct value?

Yes, but setting the rotation seems to make the simulation less smooth. I have had the most success setting all the vectors using forces and velocities only, now I just have to focus on rotation.