Authoritative Networking 2D Movement

Alright, so a little about the game I am working on: It is a completely 2D game, and everything is square. Your player is a square, the environments are squares, and the map is a rectangle (Sue me). You move using Left/Right Arrows, and jump using Space. There is physics, and it is completely handled by the Server (Authoritative).

So all Clients send RPC's of their movement to the server which simulates the movement (Setting the Velocity for Left/Right movement, and using AddForce for jumping). The new position of the player is then sent back to all Clients, which is then updated as their Transform Position. And this is where I am having problems: On the Server, movement looks perfectly fine because it is using Physics and it's smooth. But on the Client, movement is extremely choppy. I have tried Lerping the Client's position between the current position and the new one set by the Server, but this gives a "linear" effect which is not the type of movement I want.

Example: On the Server when you hit Left, you go left instantly and when you let go, your player stops instantly by setting the Velocity to 0. On the Client with Lerp, when you hit Left you slowly accelerate left and when you let go, you decelerate even slower to the position where you should be.

PlayerMovement.js

 function FixedUpdate() {
     if (networkView.isMine) {
         //JUMPING SCRIPT
             if (Input.GetAxis('Horizontal')) { //IF HORIZONTAL (LEFT/RIGHT) INPUT IS DETECTED
                 if (Input.GetAxis('Horizontal') > 0.0) { //IF MOVING RIGHT
                     facingRight = true;
                     if (Network.isServer) {
                         ServerSimulateHorizontalMovement(networkView.viewID, Input.GetAxis('Horizontal')*moveSpeed);
                     } else if (Network.isClient) {
                         networkView.RPC('ServerSimulateHorizontalMovement', RPCMode.Server, networkView.viewID, Input.GetAxis('Horizontal')*moveSpeed);
                     }
                 } else if (Input.GetAxis('Horizontal') < 0.0) { //IF MOVING LEFT
                     facingRight = false;
                     if (Network.isServer) {
                         ServerSimulateHorizontalMovement(networkView.viewID, Input.GetAxis('Horizontal')*moveSpeed);
                     } else if (Network.isClient) {
                         networkView.RPC('ServerSimulateHorizontalMovement', RPCMode.Server, networkView.viewID, Input.GetAxis('Horizontal')*moveSpeed);
                     }
                 }
             } else { //IF NO HORIZONTAL MOVEMENT (LEFT/RIGHT) IS DETECTED
                 rigidbody2D.velocity.x = 0.0; //STOP THE MOVEMENT INSTANTLY
             }
 }

 @RPC
 function ServerSimulateHorizontalMovement(clientViewID:NetworkViewID, input:float) {
     var clientToMove: Transform = NetworkView.Find(clientViewID).transform;
 
     clientToMove.rigidbody2D.velocity.x = input;
     networkView.RPC('SendClientHorizontalMovement', RPCMode.Others, clientViewID, clientToMove.transform.position.x);
 }
 
 var newXPosition: float;
 
 @RPC
 function SendClientHorizontalMovement(playerViewID:NetworkViewID, xPos:float) {
     var playerToMove = NetworkView.Find(playerViewID).transform;
 
     playerToMove.GetComponent(PlayerMovement).newXPosition = xPos;
 }
 
 function Update() {
     if (newXPosition) {
         transform.position.x = newXPosition;
     }
 }

Right now I am focusing on Left/Right movement, as Jumping will be it's own beast to take down. And I assume that once I figure out how Left/Right works, jumping will be pretty similar. I hope.

Any help would be appreciated. I have been working with Networking in Unity for about 2 months now, and I must say... This is a fucking headache.

Thanks, DarKTower