Object rotates normaly while moving, then does zig zags?

I have an object of mass (Black hole) with a circle collider attached, and small yellow object (Photon) that moves in the direction that it is pointing at a constant velocity. When the Photon enters the cricle collider, it rotates twards the black hole at a constant rate to simulate gravity (this seems to be more optimal than to have a force acting on it). It seems that when it passes the middle of the circle it decides to rotate in the other direction faster than my set velocity, then zig zag for a bit. I had a point affector that simulated gravity that worked, but, it was very inconsistant( the photon landed somwhere eles every time I played). I am lost on how to fix this, If you can help it would be very apriciated. Thank you for looking at my issue! :)

code for rotation:

void OnTriggerStay2D(Collider2D z)

 {    
     if (z.gameObject.tag=="Photon")
         {
         Debug.Log ("hit");
     Vector3 vectorToTarget = gameObject.transform.position-photon.transform.position;
     float angle=Mathf.Atan2 (vectorToTarget.y,vectorToTarget.x)*Mathf.Rad2Deg;
     Quaternion q=Quaternion.AngleAxis (angle,Vector3.forward);
     photon.transform.rotation=Quaternion.Slerp (transform.rotation,q,Time.deltaTime*speed);
 }
     }

the gameObject is the black hole.