Rotating around a fixed point at a fixed distance?

Hey there!

I am pretty new to Unity, and I am having some issues getting the right result out of some ship movement In a prototype I am working on. I'm not sure if it's better to try and use the built in physics engine, write my own, or subvert physics entirely and just use transforms to get what I want.

Basically, I have a top down 2.5D space shooter where I have a space ship object travelling a specific direction at a fixed speed. I want to be able to rotate in a circle around a fixed point (another object). As the ship flies past the fixed point on the map, I want to be able to press a key to make the ship rotate around the point in a circle, with the radius of the circle being the same distance the ship was away from the point when I pressed the key. Then, when I let off of the key, I want the ship to continue flying in the direction that it was point when it detached. I want the ship to be moving at about the same speed this whole time, if possible.

It's kind of hard to explain, so I made an ugly diagram. The triangle represents the ship and the star the fixed point to rotate around:

I have something that works about 60% the way I want using a HingeJoint. Basically I have a rigidbody on the ship and a hingejoint on the fixed point. When space is pressed, the ship rotates correctly, but it jumps to a weird location on the circumference rather than where it should. I made a gif, but it won't let me post here. If people are curious, I can give a dropbox link or something.

Any help or suggestions are appreciated! Be as detailed as possible as I am new even to the basics of Unity.