Problems implementing a 3rd person static camera

I am building a space game that I would like to be controlled in third person.

Basically I would like the camera and the ship to remain tied together and remain the same distance from each other but for the camera to remain in the same position relative to the ship from the world perspective. This position of the camera relative to the ship can only be altered by mouse movement and not by ship movements.

Example of the functionality required would be:

• User can rotate the the ship on the x, y, z axis and the camera can observe these rotations

• User can position the camera on the right side of the ship using mouse movements and activate thrusters to observe the ship in the center of the screen and the background objects passing

• User can fly ship into an asteroid and observe rigidbody collider cause the ship to bounce back and start to rotate violently in a direction but the camera will remain stable and at a constant distance to the ship

Explanation from technical sense:

• The camera must remain at (0,0,-100) from the ship in world coordinates and this position only changed using mouse movement using local position

• Putting camera and ship in the same player object allows to control local coordinate to (0,0,-100) but from world perspective the positing of the camera to ship actually changes

Currently the main problem is that I can only either have the camera stuck in a constant position relative to the ship (cannot change perspective with mouse) OR I can change perspective with mouse but the rigitbody physics get applied to camera (if I hit an asteroid the camera rotates with the ship)

I also tried to use a dummy camera to track mouse movements and to just add it to the constant camera but when the ship rotated very quickly it wasn't very accurate.

Any suggestions for what I could use in order to get accurate response?