Collision Problem

Ok, i have a problem with my colliders.

I have a space ship, where i can fly with. It has a box collider. And when I fly and collide with a meteorite it pushes the meteor away with a rocky movement. the meteor has a cirkel colider and a rigid body attached, the rigid body has no gravity.

When i remove the rigid, i fly through it, if i set the position to freeze, i fly through it. And if i adjust the mass to something higher, i fly through it as well... What can I do ?

I want to fly against it and I don't want the meteor to move away.

anyone ?