[Unity Vuforia] Rotating object with mouse drag

Hello everybody, i'm developing a game in Unity that has augmented reality in it. This is a picture of my objects so i can explain the aim of the game:

The goal is to make each specific pointer point the the ball with the same color as its own. The user has to click and drag each pointer to rotate it towards the ball that it wants. The catch is that the pink colored pointer rotates by itself when dragged, but the blue pointer rotates itself and the pink one as well when rotated. The green rotates itself and the other two, and the yellow one rotates all of them. My problem is i don't know how to make them rotate!!

This is the code i have for the pink pointer that rotates alone.

 public int position;
 public float smooth = 1f;

 private Quaternion targetRotation;
 private bool isRotating;

 // Use this for initialization
 void Start()
 {
     targetRotation = transform.rotation;
 }

 void Update()
 {
     if (isRotating)
     {
         //Get the Screen positions of the object
         Vector2 positionOnScreen = Camera.main.WorldToViewportPoint(transform.position);

         //Get the Screen position of the mouse
         Vector2 mouseOnScreen = (Vector2)Camera.main.ScreenToViewportPoint(Input.mousePosition);

         //Get the angle between the points
         float angle = AngleBetweenTwoPoints(positionOnScreen, mouseOnScreen);

         //Ta Daaa
         transform.rotation = Quaternion.Euler(new Vector2(0f, angle));
     }

     
      }

 float AngleBetweenTwoPoints(Vector3 a, Vector3 b)
 {
     return Mathf.Atan2(-(a.y - b.y), a.x - b.x) * Mathf.Rad2Deg;
 }

 private void OnMouseDown()
 {
     isRotating = true;
 }

 private void OnMouseUp()
 {
     isRotating = false;
 }

One of the problems is that if i click near the pink ball it rotates 180 degrees immediately and then lets me rotate as i want, and if i click the bottom part it rotates fine. I want them both to rotate fine. The second problem is i don't know how to make the blue and pink pointers rotate together.

If anyone can help me i would really appreciate it!!! Thanks.