Slower code executing in Unity then outside of Unity,Code execution is slower in Unity then outside of Unity

I have a .NET standard class library, and I am using this in Unity. One of the methods of my class library is a "ToTensor" image to tensor conversion method. When I benchmark this method with BenchmarkDotNet in the class library it takes around 4 ms to convert a (1080p) image to a tensor. However, this is not the case in Unity. Calling the same method, from unity, it takes about 28ms to convert the same 1080p image to a tensor. Why is this? And how can I improve on this? Because 28ms is waaay to slow.

In case someone is wondering about the code: https://gist.github.com/luuk777w/3258040454e80ca0bdaf7c601e1c6965