[2D - Top Down] Camera track / Border control

Hey, I'm having a bit of an issue with my camera script. The aim is so that it will gently track the player up until it hits the background boundary. I can get it to track the player statically and stop at the walls, I can slowly track the player, but I can't do both.

Here's what I'm using now:

 using UnityEngine;
 using System.Collections;
 
 public class Border : MonoBehaviour
 {
     private float rightBound;
     private float leftBound;
     private float topBound;
     private float bottomBound;
     private Vector3 pos;
     private Transform target;
     private SpriteRenderer spriteBounds;
     public float dampTime = 0.15f;
     private Vector3 velocity = Vector3.zero;
 
     // Use this for initialization
     void Start()
     {
         float vertExtent = Camera.main.orthographicSize;
         float horzExtent = vertExtent * Screen.width / Screen.height;
         spriteBounds = GameObject.Find("BG").GetComponentInChildren<SpriteRenderer>();
         target = GameObject.FindWithTag("Player").transform;
         leftBound = (float)(horzExtent - spriteBounds.sprite.bounds.size.x / 2.0f);
         rightBound = (float)(spriteBounds.sprite.bounds.size.x / 2.0f - horzExtent);
         bottomBound = (float)(vertExtent - spriteBounds.sprite.bounds.size.y / 2.0f);
         topBound = (float)(spriteBounds.sprite.bounds.size.y / 2.0f - vertExtent);
     }
 
     void FixedUpdate()
     {
         Vector3 point = Camera.main.WorldToViewportPoint(target.position);
         Vector3 delta = target.position - Camera.main.ViewportToWorldPoint(new Vector3(0.5f, 0.5f, point.z)); 
         Vector3 destination = transform.position + delta;
 
         var pos = new Vector3(target.position.x, target.position.y, transform.position.z);
         pos.x = Mathf.Clamp(pos.x, leftBound, rightBound);
         pos.y = Mathf.Clamp(pos.y, bottomBound, topBound);
         //transform.position = pos;
         Debug.Log(pos);
         transform.position = Vector3.SmoothDamp(pos, destination, ref velocity, dampTime);
         
     }
 
 
 }

 

If anyone could shed some light as to why this is happening I'd be very grateful. Thanks!