Can you somehow track why UnloadUnusedAssets doesn't unload a specific asset?

We're over a year in development of one of our games and we've recently found that some assets just won't unload (when changing scenes or when calling UnloadUnusedAssets). In the description of the function UnloadUnusedAssets it states that "An asset is deemed to be unused if it isn't reached after walking the whole game object hierarchy, including script components. Static variables are also examined."

Since we've been developing for quite a while and we're using a couple of plugins - it's quite difficult to track what object might still be referencing the asset (we've been searching for a looong time now). Is there a way to "follow" the UnloadUnusedAssets process and pin point where it's referenced from? Are there any tools that might help us - maybe a third party memory profiler that works with unity?

The weird thing is that if we unload it manually (FindObjectsOfTypeAll(Material/Texture2D) and then unload the specific one that's "leaking") the game runs fine - meaning that whatever that is referencing the asset doesn't really need to (since we're not seeing any side effects).

Best regards, Mattias Hakulinen.