Jittery pan touch controls

So I'm working on some code to pan the camera around on my phone with one or two fingers. My plan was to get the initial touch position, and then offset the camera by the difference between the current touch position and the initial one. The camera is orthographic, with position (0,0,-10) and faces (0,0,0), with an initial size of 5. But... the object at (0,0,0) doesn't stay under my finger as I pan. Am I misusing the Camera.main.ScreenToWorldPoint function? Additionally, there's some jitter. Anything I can do to reduce this?

 using UnityEngine;
 using System.Collections;
 
 public class PinchZoomPan : MonoBehaviour
 {
     private float INITIALSCALE, INITIALDIST;
     private Vector3 DEFAULTPOS;
     private Vector3 INITIALCAMPOS1, INITIALFINGPOS1, CURRENTFINGPOS1;
     private Vector3 INITIALCAMPOS2, INITIALFINGPOS2, CURRENTFINGPOS2;
     public float MINSCALE = 1f;
     public float MAXSCALE = 6f;
 
     void Start()
     {
         //Camera.main.transform.position = DEFAULTPOS + new Vector3(0, 1, 0);
         DEFAULTPOS = Camera.main.transform.position;
     }
 
     void Update()
     {
         // Initializing variables when a one finger pan begins
         if (Input.touchCount == 1 && Input.GetTouch(0).phase == TouchPhase.Began)
         {
             INITIALFINGPOS1 = Camera.main.ScreenToWorldPoint(Input.GetTouch(0).position);
             INITIALCAMPOS1 = Camera.main.transform.position;
         }
 
         // One finger pan
         if (Input.touchCount == 1 && Input.GetTouch(0).phase == TouchPhase.Moved)
         {
             CURRENTFINGPOS1 = Camera.main.ScreenToWorldPoint(Input.GetTouch(0).position);
             Camera.main.transform.position = INITIALCAMPOS1 + INITIALFINGPOS1 - CURRENTFINGPOS1;
         }
         
     }
 
 }