World to local Inversion not working as I expect

Okay, so I have a chunk of code in the OnDrag, which I use for swiping a gameobject.

     public void OnBeginDrag(PointerEventData eventData)
     {
             GrabPointDir = Camera.main.ScreenToWorldPoint(Input.mousePosition) - transform.position;
     }

     public void OnDrag(PointerEventData eventData)
     {
         if (allowSwipe && Input.touchCount <= 1)
         {
             var scrToWorld = Camera.main.ScreenToWorldPoint(Input.mousePosition);
             var targetWorldPos = new Vector3(scrToWorld.x - GrabPointDir.x, scrToWorld.y - GrabPointDir.y, transform.position.z);
             var targetLocalPos = transform.InverseTransformDirection(targetWorldPos);
             var clampedLocalPos = new Vector2(Mathf.Clamp(targetLocalPos.x, -100f, 100f), targetLocalPos.y);
             transform.localPosition = clampedLocalPos;
         }
     }

What I expect from this is that I make all calculations in world coordinates, then use InverseTransformDirection to make it local coords and set constraints, then apply the result. But what I get is chaotic movement, jumping etc. I tried both InverseTransformDirection and InverseTransformPoint, but at some point it worked when I set the result to transform.position, not localPosition. So I must be missing something about those transforms.