How to limit AudioClip Buffer for real-time voice chat ?

I'm working on a real-time voice chat application. I get audio samples from NativePlugin. 320 samples per frame.

Here is the code.

Get Audio Samples from Native Plugin

 [AOT.MonoPInvokeCallback(typeof(C.audioDataCallbackFunc))]
 private void _onAudioData(String handle, IntPtr dataPtr, UInt32 dataSize)
 {
     int len = Convert.ToInt32(dataSize / 2);
     short[] audioDatamanaged = Utility.IntPtrAndSizeToShortArray(dataPtr, len);
     for (int i = 0; i < len; i++)
     {
         _pcmQueue.Enqueue(AudioConverterConstant * (float)audioDatamanaged[i]);
     }
 }

PlayAudio

 void Start()
 {
     AudioClip clip = AudioClip.Create("remoteAudio", 320, 1, 16000, true, OnAudioRead);
     sourceAudio.clip = clip;
     sourceAudio.Play();
 }
 
 public void OnAudioRead(float[] data)
 {
      Debug.Log("Queue:" + QueueCount);
      if (QueueCount >= 320)
      {
           for (int i = 0; i < data.Length; i++)
           {
                data[i] = _pcmQueue.Dequeue();
           }
      } else {
          for (int i = 0; i < data.Length; i++)
        {
             data[i] = 0.0f;
         }
      }
 }

I print QueueCount when OnAudioRead called. The QueueCount sometimes greater than 9000. This makes the audio has about 1 second latency.