AssetBundle prefab with missing material

I have a project where I'm building an AssetBundle and loading a prefab from said assetbundle. This prefab consists of an empty, with a single child object that contains a mesh, meshfilter, collider and a material. When I load this prefab, the material is missing. However, if I remove the empty parent so the prefab consists of only the single object with the components previously listed, the prefab has everything expected.

Any idea why this might be happening and how I can fix it?