Shaky player effect when I apply a damping camera follow script?

I stumbled across this script on Unity answers, and I decided to stick it on rather than just make the camera a child object of the player. It works perfectly, except when the player is moving it makes the player look like it's shaking. I think the term for that is "clipping" or something like that. Anyway, on the script, when I change the damping variable to 0, the player doesn't shake anymore, so it has to do with the damping. But I want the damping. Is there anyway to have the damping but not have the player appear to be shaking? Here's the script:

 using UnityEngine;
 using System.Collections;
 
 public class CameraFollowFoReal : MonoBehaviour {
     
     public float dampTime = 0.15f;
     private Vector3 velocity = Vector3.zero;
     public Transform target;
     
     // Update is called once per frame
     void Update () 
     {
         if (target)
         {
             Vector3 point = camera.WorldToViewportPoint(target.position);
             Vector3 delta = target.position - camera.ViewportToWorldPoint(new Vector3(0.5f, 0.5f, 20f)); //(new Vector3(0.5, 0.5, point.z));
             Vector3 destination = transform.position + delta;
             transform.position = Vector3.SmoothDamp(transform.position, destination, ref velocity, dampTime);
         }
         
     }
 }

Any ideas what's wrong with it? I think from reading online it has something to do with clamping, but I can't seem to figure it out. Any help is much appreciated.

Edit: I should probably add, my game is 2D. Here is a screenshot of it so you can visualize what I mean:

Nothing else shakes when the camera moves, the tree and ground look normal, it's just the player.