GC lags cause sound to stutter (buffer underrun) when using OnAudioFilterRead

I am using OnAudioFilterRead for a simple music crossfade. The effect performs smoothly, it's FPS independent, of course, and works even when activating a new loaded scene (which can take several seconds). But the GC spikes seem to block all threads, causing the audio buffer underrun. Yet this doesn't seem to happen when using built-in filters (such as low-pass). I assume that these run in the unmanaged memory world and are not affected by the GC spikes. Since GC spikes can be reduced, but not really elliminated, does this mean that OnAudioFilterRead is generally unusable, if I want to avoid annoying underruns and looping sound? Any tips?

Here's my fading thingy audio code

 private void OnAudioFilterRead(float[] data, int channels)
         {
             if (m_dsp_fadeEndVolume != targetVolume)
             {
                 m_dsp_fadeStartVolume = m_dsp_currentVolume;
                 m_dsp_fadeEndVolume = targetVolume;
                 m_dsp_fadeOffset = 0;
                 var volumeDifference = Mathf.Abs(m_dsp_fadeEndVolume - m_dsp_fadeStartVolume);
                 m_dsp_fadeLength = Mathf.RoundToInt((volumeDifference / fadeSpeed) * m_dsp_sampleRate);
             }
 
             if (m_dsp_fadeOffset < m_dsp_fadeLength)
             {
                 for (int i = 0; i < data.Length; ++i)
                 {
                     var ch = i % channels;
                     if (ch == 0)
                     {
                         var fadeProgress = (float)m_dsp_fadeOffset / m_dsp_fadeLength;
                         m_dsp_currentVolume = Mathf.Lerp(m_dsp_fadeStartVolume, m_dsp_fadeEndVolume, fadeProgress);
                         ++m_dsp_fadeOffset;
                     }
 
                     data[i] *= m_dsp_currentVolume;
                 }
             }
             else
             {
                 m_dsp_currentVolume = m_dsp_fadeEndVolume;
                 for (int i = 0; i < data.Length; ++i)
                 {
                     data[i] *= m_dsp_currentVolume;
                 }
             }
             m_currentVolume = m_dsp_currentVolume;
         }