Need help getting mouse controlled multiplayer camera to work

So basicly I'm spawning my player (a simple Sphere). It's a prefab, so I can't just make the camera a child object of it as Unity will complain there's no Main camera. So instead I need to use C# to do it. This is what I've tried:

 public Transform target;
 public float smoothing = 5f;
 
 Vector3 offset;
 
 public GameObject playerPrefab;
 
 private void SpawnPlayer()
 {
     Network.Instantiate(playerPrefab, Vector3.up * 5, Quaternion.identity, 0);
     offset = transform.position - target.position;
 }
 
 void LateUpdate()
 {
     Vector3 targetCamPos = target.position + offset;
     transform.position = Vector3.Lerp(transform.position, targetCamPos, smoothing * Time.deltaTime);
 }

So it does transform the camera position at the start towards the player location. But after that, it refuses to follow the player. It just sits there. I've tried everything I could find on the internet so far. Never getting errors. Just simply not following. And now I'm just at a loss. Anyone able to help out?