When player move the camera flickers and it looks janky

I made a Camera Script that follows the player using a lerp for making it look more smooth, but for some reason, it looks laggy when walking in some sudden areas.

The ground the player is walking on is tilemap, and at first, I thought there were tiny cell gaps that caused the problem, but it still happens even when I changed the ground to one solid block. So I concluded it must be something with Camera Script that cause this problem.

Here is a video clip of what I'm talking about: https://youtu.be/mmBMHWuHpxo

I think the problem stems from this part of my camera script:

   void SetTargetPos()
     {
         // By default the target x and y coordinates of the camera are it's current x and y coordinates.
         targetX = transform.position.x;
         targetY = transform.position.y;
         // If the player has moved beyond the x margin...
         if (CheckXMargin())
         {

     targetX = Mathf.Lerp(transform.position.x, transformPlayer.position.x, xSmooth * Time.deltaTime); 
         }
 
         // If the player has moved beyond the y margin...
         if (CheckYMargin())
         {

     targetY = Mathf.Lerp(transform.position.y, transformPlayer.position.y, ySmooth * Time.deltaTime);
         }
 
             targetX = Mathf.Clamp(targetX, currentMinBounds.x, currentMaxBounds.x);
             targetY = Mathf.Clamp(targetY, currentMinBounds.y, currentMaxBounds.y);
        
         
         // Set the camera's position to the target position with the same z component.
         transform.position = new Vector3(targetX, targetY, transform.position.z);
 
         TestOutOfCamBounds();
     }

So my question is how can I make the camera look less janky and more smooth. Just in case of the issue is not from this part of the camera script, I will post the entire script. The rest of the camera script:

 public class Camera_Controller : MonoBehaviour
 {
     private static Camera_Controller _instance;
     public static Camera_Controller instance;
     [Header("Put player here")]
     public GameObject player;
     public Transform transformPlayer;
     [Space]
     Vector2 currentMinBounds;
     Vector2 currentMaxBounds;
     public Vector3 targetPos;
 
     [Space]
     [Header("Camera Properties")]
     public float xMargin = 1f;
     public float yMargin = 1f;
     public float xSmooth = 8f;
     public float ySmooth = 8f;
 
     private float targetY;
     private float targetX;
 
     [Header("Smooth Transition")]
     public Vector3 oldPosition;
     public Vector3 targetPosition;
     public float transitionsTime;
     public bool switchingCamera;
 
     private void Awake()
     {
 
 
         if (_instance != null && _instance != this)
         {
             Destroy(gameObject);
             return;
         }
         else
         {
             _instance = this;
         }
 
         instance = _instance;
     }
 
     void Start()
     {
         targetPos = transform.position;
     }
 
     private bool CheckXMargin()
     {

         return Mathf.Abs(transform.position.x - transformPlayer.position.x) > xMargin;
     }
 
 
     private bool CheckYMargin()
     {

         return Mathf.Abs(transform.position.y - transformPlayer.position.y) > yMargin;
     }
     void Update()
     {
         if (!switchingCamera)
         {
             SetTargetPos();
         }
 
     }
 
     public void SetCamBounds(Vector2 minBounds, Vector2 maxBounds) //Called from Game Events Trough WarpController as trigger
     {
         currentMinBounds = minBounds;
         currentMaxBounds = maxBounds;
     }
     //SetTargetPos() should be causing the problem
     void SetTargetPos()
     {
         // By default the target x and y coordinates of the camera are it's current x and y coordinates.
         targetX = transform.position.x;
         targetY = transform.position.y;
         // If the player has moved beyond the x margin...
         if (CheckXMargin())
         {

             targetX = Mathf.Lerp(transform.position.x, transformPlayer.position.x, xSmooth * Time.deltaTime); 
         }
 
         // If the player has moved beyond the y margin...
         if (CheckYMargin())
         {

             targetY = Mathf.Lerp(transform.position.y, transformPlayer.position.y, ySmooth * Time.deltaTime);
         }
 
             targetX = Mathf.Clamp(targetX, currentMinBounds.x, currentMaxBounds.x);
             targetY = Mathf.Clamp(targetY, currentMinBounds.y, currentMaxBounds.y);
        
         
         // Set the camera's position to the target position with the same z component.
         transform.position = new Vector3(targetX, targetY, transform.position.z);
 
         TestOutOfCamBounds();
     }
 
     void TestOutOfCamBounds() //Set Camera some boundaries.
     {
         if (targetPos.x <= currentMinBounds.x)
         {
             targetPos.x = currentMinBounds.x;
         }
         if (targetPos.x >= currentMaxBounds.x)
         {
             targetPos.x = currentMaxBounds.x;
         }
         if (targetPos.y <= currentMinBounds.y)
         {
             targetPos.y = currentMinBounds.y;
         }
         if (targetPos.y >= currentMaxBounds.y)
         {
             targetPos.y = currentMaxBounds.y;
         }
     }
 
 }

Btw I have the exact same script on different Unity Projects with the same variable inputs. There is a small difference, and that just the overall size of everything in the second project is a lot smaller than the first one. Here is a video clip of that one: https://youtu.be/baJmKehYfG0