koobas.hobune.stream
Audio Spatializer implementation and distribution challenges
During the development of a spatializer plugin for Unity3D, some questions arose that remain unanswered in the documentation:
1) The maximum input magnitude of data for the spatializer. We have observed that pitching is applied before the spatializer and we know that distance attenuation is also applied before the spatializer. I.e. the max input data magnitude can only exceed full scale (1.0) if there is no distance attenuation and resampling after pitching causes overshoots. Not all resampling strategies will generate overshoot. Can we be sure that the input never exceeds full scale?
2) The maximum recommended magnitude of output audio data from the Spatializer plugin. Since the output data are floats, it is possible to output data greater than full scale (magnitude 1.0). Ideally, the perceived loudness of the audio after HRTF processing should match the perceived loudness of the input data but in some cases, the output data will then overshoot full scale with several dB. What is the maximum Spatializer output magnitude you support without causing clipping?
3) In your documentation it says: “Output sample rate. If set to 0, the sample rate of the system will be used. Also note that this only serves as a reference as only certain platforms allow changing this, such as iOS or Android.” Do you have a rule of thumb for what sample rates should be supported for Android and iOS to cover the majority of devices?
4) Our tests have shown that in Unity 5.3.2f1 the “Doppler Level” for an audio source does not have any effect when “Spatializate” is enabled. There is no Doppler effect. As a workaround we implemented the Doppler effect by changing the pitch in a script. Is it intentional that there is no Doppler effect or is it a bug?
5) Our Spatializer plugins are proprietary and are not going to be free. We have noticed how easy it is to extract a spatializer plugin from for instance an Android apk file. Furthermore, there is no copy protection when delivering it in a Unity package. Is there any way to prevent illegal copying of our plugins?
Best Regards, Kim Therkelsen
Hi Kim,
1) In general you shouldn’t make any assumptions about the dynamic input range, as it depends on the codec used. Also future versions may provide a way to allow effects before spatialization.
2) Same here. Users could be processing this signal by additional effect components or when the spatialized signal gets routed into the audio mixer, so there are many potential sources for further dynamic processing. At the final output stage just before hitting the sound card everything gets clipped to the [-1; 1] range. This is done in order to make the editing consistent with whatever clipping certain output platforms may apply (i.e. Windows/OSX/mobiles have a built-in limiter, wheres consoles don’t).
3) You should always query this through UnityAudioEffectState and try not make any assumptions about it. In that regard our own Plugin_Spatializer.cpp example is horribly wrong by not resampling the IRs to the target system rate. Given the huge amount of Android devices, I don’t know a rule of thumb for device capabilities.
4) Yes, this is a known bug that we already have a fix for. Can’t put any ETA on the release though.
5) Copy-protection is not anything we provide a solution for. I also think it's a bit beyond of the scope of the audio plugin SDK as such. Since the audio SDK is based on the generic native plugins you may be able to take a look at how other Unity extensions handle this. Also note that since all your functions of the plugin are within the same process space, you may be able to make some extra DllImport functions accessed from C# to handle the unlocking of your audio callback processing code.
Jan
Thanks Jan, your answers are very useful to us. Regarding sample rate handling, I think I have discovered a bug, case 780788: https://fogbugz.unity3d.com/default.asp?780788_4o0qlf8se32vk5r9
If we return UNITY_AUDIODSP_ERR_UNSUPPORTED in CreateCallback, the Unity app and editor crash at shutdown. If we do not support the sample rate, I assume it should be okay to return UNITY_AUDIODSP_ERR_UNSUPPORTED? It seems to work fine (beside crashing at shutdown) because the spatializer falls back to panning.
