RelativeTorque and Vector3.Cross to look at an object

Hi there, thanks for looking at my question.

I looked around and googled a fair bit for the information I need, but sadly wasn't able to come up with much beyond what I already have.

In order to use rigidbody physics to move the object, I wanted to make a physics solution for looking at an object (in this case, the player). This is a snippet of the code I'm using:

var player     = GameObject.FindWithTag("Player").transform;
rigidbody.AddRelativeTorque(Vector3.Cross(-transform.forward,player.forward)*rotationSpeed;

Now in this case, it looks at the player if the player is looking at it, but of course if the player looks away so does the object. So the obvious question is, how would I go about acheiving the actual behaviour I'm looking for, rather than this one?

Thank you for any help you can provide. =)