How to make a child transform not move relative to parent?

I am building a character select that swipes. I have a parent gameobject that moves with finger. The children of said gameobject are the characters. when the characters are center of screen they scale up and have a 0.5f cushion of both sides of x. This allows some swiping while character stays center. I have character sent to stay at zero while scaled up. Everything works perfect, except on thing while swiping the character twitches a little bit. I know why. I have child position set with world position, but it's local position still moves with swipe. Basically the character slightly moves with local position, then sets to zero world position. Is there a way to make the character not a child while center of screen then set back to child when he isn't in position to be selected? Is there a way to ignore local position while centered? Or maybe another option? Thanks.

     public Vector3         minScale;
     public Vector3        maxScale;
     public float        lerpTime;
 
     Transform            character;
 
     // Use this for initialization
     void Start () 
     {
         foreach(Transform child in transform)
         {
             character = child;
         }
     }
     
     // Update is called once per frame
     void Update () 
     {
         if(transform.position.x < 0.999f && transform.position.x > -0.999f)
         {
             Vector3 characterPos = character.position;
             character.localScale = Vector3.Lerp (maxScale, minScale, 0);
             characterPos = new Vector3(0f, -1.0f, -5f);
             character.position = characterPos;
         }
         else{
             character.localScale = Vector3.Lerp (minScale, maxScale, 0);
             character.localPosition = Vector3.zero;
         }
     }