Collider2D entering other colliders

So I have a 2D project where the player can pick up and move objects. All have Collider2D and Rigidbody2D components. The player is on one layer, the objects are on another, any held objects are moved to a third layer that doesn't collide with the player layer. Because the position of any held items is locked while being carried, it ends up being carried through other items, even if they shouldn't be able to collide. I'd like to be able to prevent this, because it means players can move the items through walls or place them on other objects.

I appreciate any help!