Jittery movement with camera following player

I'm working on a game similar to Factorio. The camera is locked to the player while the player navigates around the world. The camera is updated to the player's position (minus some offset and zoom level) in LateUpdate. Player movement feels buttery smooth.

However, the player has a swarm of ships that follow it, and these ships experience a lot of jitter. The jitter goes away if I stop the camera from moving. I've tried different camera movement strategies (lerp, moveTwards, SmoothDamp), none of these work. Probably because there is some more fundamental solution to this problem involving the movement of the swarm around the player. Nothing uses physics bodies (or rigidbodies). I'm just updating positions of everything directly on their translation/transform.

Here is a demonstration of the problem:

https://www.youtube.com/watch?v=JoXCGZDGrIM

Here's the camera code:

       var dirVector = DirectionBuilder.Create(Direction).Vector; // Vector representing up, down, left or right, depending on camera rotation
       var followObjectPosition = _followObject.position;
       
       var basePos = new Vector3(
         followObjectPosition.x - dirVector.x * _cameraZOffset,
         _cameraHeight,
         followObjectPosition.z - dirVector.z * _cameraZOffset);
       
       transform.position = basePos + (_followObject.transform.position - transform.position).normalized * _zoomFactor;

Pretty basic stuff. The swarm movement code is fairly complex to make their movement feel natural, but I need a solution that will allow arbitrary entities following the player to move with the player at around the same speed as the player, without jittering (might be worth noting that my game is DOTS, the player is the only thing in the game that has a monobehaviour. The swarm around the player are all pure entities).