Collider act at position offset from what is displayed,Collider position is offset from it's object?

As you can see in the screenshots, I have a model to which is attached a collider (exact structure is I have a parent player object on which are all the scripts and as a child of that is the GFX object holding the mesh and the boxcollider)

The player is a prefab, the parent object is centered 0,0,0, the gfx is 0,0,0 in the first picture and 0,2,0 in the second picture. As you can see the collider correctly encapsulate the model, however it's not where it should be (for both screenshot I let the character fall and you can see it where it stopped). The floor has a mesh collider working correctly with the objects you can see in the background (the have mesh colliders and rigidbodies)

I realize now that the picture isn't all that clear but in this second screenshot the character is not finishing his fall to the floor and is now floating around (If I walk to the edge it will fall to its doom so the collider is doing things, just in the wrong place)

For the moment I use the GFX offset as a workaround but it's not great because it's not precise and I have to change it everytime I change the character's size. I would also like to understand this behavior.

Any idea what is happening here and what I can do about it?