How to make the turret of a tank in a 3rd person game look at the mouse cursor?

I find the best way to do things like this is to think about it realistically, and work from there.

With an actual tank turret, they generally have two separate pivot points. The first moves the whole 'head' around the Y-axis, allowing the turret to turn 360 degrees around the tank. The second pivot is usually near where the barrel starts, and allows the barrel to be pitched up or down a limited amount.

So, you want to set up your scene as such. Include an empty GameObject in the hierarchy for each pivot and store a reference to each.

Then you want to take the Transform.forward vector from the camera and get the barrel to match that as close as you can using the provided pivots and constraints. You're gonna need a little trig here:

First, you want to find the yaw for the first pivot. I suggest feeding in the transform.forward's x and z components to Mathf.Atan2 to get an angle in radians. Convert it to degrees and apply it as a y-axis euler rotation to the first pivot.

Next, you want to find the pitch. First you'll need the combined horizontals for the vector, using the Pythagorean equation with the x and z components. Then you can just use Atan2 again with the y component and the x/z combined horizontal component to get the pitch in radians. Depending on how your second pivot is oriented, you can apply that with another simple local euler rotation on the second pivot. Since the second pivot is a child of the first pivot, it will inherit it's transform and combined, your turret will be pointing the correct direction.