Physics based bullet penetration collision detection issues

I'm not very dumb, but I may be overlooking something insanely obvious.

I have been working on a basic project to work with some semi-realistic bullet trajectories in VR, for this, I want to implement some penetration mechanics. I understand the calculations and the programming, but I am having some major issues deciding how to simultaneously allow the bullet entity to pass through the level architecture, while also still calling OnCollisionEnter. A trigger would not work in this case, as the player character still has to interact with these solid objects, and I cannot understand how the system can work otherwise. This for some reason has kept me stuck for a few hours now and I'm about ready to give up.

 void OnCollisionEnter(Collision collision)
 {
     hitObject = collision.gameObject;
     Debug.Log(hitObject.name);
     hitPoint = transform.position;
     Debug.Log(hitPoint);
     //if(fleshDamage < 1)
     //{
     //    Destroy(gameObject);
     //}
 }
 void OnCollisionExit(Collision collision)
 {
     exitPoint = transform.position;
     Debug.Log(exitPoint);
     length = Vector3.Distance(hitPoint, exitPoint);
     Debug.Log(length);
 }