FixedJoint Stopping Rigidbody Physics

I have been trying to make a system for the player to be able to pick up items, the pick up system works fine when I use FixedJoints, but for some reason I can't get the rigidbody physics to work unless the joint has a connectedBody. I don't know what to do to remedy this issue.

I have tried attaching the fixedJoint to both the player and the item but whichever object it's connected to stops gravity and non-kinematic collisions. If I attach the joint to the player, it can't move and floats in the air, if I attach it to the item, it just sits in the air and can't be pushed over, I don't know what to do.

Note that all physics works fine once the player picks up the item, it acts as normal, it's only when the item doesn't have a connectedBody that the weirdness happens.