Using lerp to smooth network movement causes rubber banding

I have a very simple networking script I use to keep track of the position of my player in game. I am attempting to use Vector2.loop to smooth out the players movements over the network, however this causes the player to rubber band up and down as opposed to moving smoothly. I have no problem directly setting the players position from what I receive over the network, except of course that the movement is not smooth. Am I missing something here?

 void Update () 
     {
         if (!MyPhotonView.isMine)
         {
             transform.position = Vector2.Lerp(transform.position, CorrectPlayerPos, LerpSpeed * Time.deltaTime);
             //transform.position = CorrectPlayerPos;
         }
     }
 
     void OnPhotonSerializeView(PhotonStream stream, PhotonMessageInfo info)
     {
         // We own this player: send the others our data
         if (stream.isWriting)
         {
             LoadAnimationStates();
             stream.SendNext(transform.position);
           //  stream.SendNext(AnimationStates);
         }
         else  // Network player, receive data
         {
             
             this.CorrectPlayerPos = (Vector3)stream.ReceiveNext();
             //this.AnimationStates = (int[])stream.ReceiveNext();
 
             ApplyAnimationStates();
         }
     }