Errors referencing DirectX.dll from custom library DLL

I'm looking at integrating voice chat into my Unity game, and have made a custom C# library to handle this for me. It uses DirectX to read voice data from the mic and then transfers it over UDP.

However, every time I try to use any DirectX functions in my custom DLL, (and then compile it and update the DLL in my Assets/ folder) Unity gives me the following error:

Internal compiler error. See the console log for more information. output was: Unhandled Exception: System.TypeLoadException: Could not load type 'VoiceComm.VoiceCommHandler' from assembly 'VoiceComm, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'.

If I don't link in Microsoft.DirectX.dll my custom DLL imports fine and I can call functions on it no problem.