Character moving at constant speed after hitting a wall

When my character hits a wall it starts to slowly move. This effect is mostly unnoticable, but you can see it when you look at the values in inspector or if you hit the wall many times. I've checked my code, but I don't think it is the one causing it. If you want to check for yourself then you have it here:

 public float Speed = 10;
 public GameObject Character;
 float horizontalMovement;
 float verticalMovement;
 Vector3 movedDistance;
 
 void Update() {
   if(Character == null)
         return;
 
   horizontalMovement= Input.GetAxis( "Horizontal" );
   verticalMovement= Input.GetAxis( "Vertical" );
 
   movedDistance= new Vector3( horizontalMovement * Speed * Time.deltaTime, 0, verticalMovement * Speed * Time.deltaTime );
   Postava.transform.Translate( movedDistance );
   networkView.RPC( "PozitionOfCharacter", RPCMode.OthersBuffered, Character.name, movedDistance, Character.transform.position );
 }

The character I use has Capsule collider (because I don't have any character to use yet) and Rigidbody and the wall has Box collider.