Colliders don't move when timescale = 0

I've got a set up where the player can place objects in a scene when time.timescale = 0. The objects either exist in a pooling system or are instantiated if there is nothing in the pool to pull from. Problem is that later the player may need to interact with those tasks while time is still at 0. I'm finding that the colliders for these objects don't seem to move when timescale = 0 so I'm unable to cast rays against these objects. Is it possible to force the colliders to move to their object when the timescale is 0?

As a side note, I know the colliders aren't moving because I've got the prefabs set to vector3.zero and when I instantiate them somewhere other than that position, I can return them from a raycast at vector3.zero but not the actual objects position.