Some issue with Raycast and mouse position

Hello, I'm trying to make a little game. The purpose is to collect some crashing meteors. So I drew a circle on the ground and, with the mouse, we have to click to move it. Here comes the issue. In fact it works quite well if the circle is small, but as soon as the circle gets bigger (the player can increase the area of the circle by collecting meteors) the precision gets lower. In fact I think I know the problem, but I don't know how to correct it. The Raycast comes from the camera and there is some kind of perspective. So when I click on the ground the middle of the circle doesn't move perfectly on the move. It goes a bit below of the mouse position, and bigger the circle is, worst is the precision.

Here is the script :

 using System;
 using System.Collections;
 using System.Collections.Generic;
 using UnityEngine;
 using UnityEngine.AI;
 
 public class PlayerController : MonoBehaviour
 {
     private float speed = 10.0f;
     public Camera cam;
     private Vector3 targetPosition;
 
 
     private void FixedUpdate()
     {
         SetPosition();
         Move();
     }
 
 
     //Locate the destination
     void SetPosition()
     {
         if (Input.GetMouseButton(0))
         {
             Ray ray = cam.ScreenPointToRay(Input.mousePosition);
             RaycastHit hit;
 
             if (Physics.Raycast(ray, out hit))
             {
                 targetPosition.x = hit.point.x;
                 targetPosition.z = hit.point.z;
             }
         }
     }
 //move the circle
     void Move()
     {
         transform.position = Vector3.MoveTowards(transform.position, targetPosition, speed * Time.deltaTime);   
     }

Do you have any idea how to be sure the middle of the circle will go exactly on the mouse position ?