Unity's collision system VS designing my own basic collision system

Hello there.

I've been looking at the collision system in Unity (the space game I've been designing has a LOT of space in it, so collisions have only just got to the top of my 'to do' list) and when I make my player/camera's RigidBody non-kinematic it seems to go out of control. I've turned gravity off, but it still seems to be being affected by forces beyond my control.

Given that my game effectively doesn't use gravity and has a slightly unique frame-of-reference system (0, 0, 0 in the game universe is always the currently-nearest-gravity-well-to-the-player and those gravity wells (planets and moons) are constantly but slowly orbiting eachother) would it be better to ignore the in-built Unity collision system of SphereColliders/RigidBodies and just implement a very simple system of checking if a particular object is closer to the player than its radius to check for collisions?

And if I do that, what do I lose? Is there some other benefit to using the Unity collision system, given that I'm prepared to have collisions calculated at a sphere-based level of detail?

Any pointers, well appreciated.