Trying to prevent click& drag camera from going through mesh, rigidbody makes it bounce like crazy

Backstory- I've got a terrain mesh and a camera object I'm trying to prevent the camera from being able to go through the mesh so I've added a Collider to both the terrain and the camera, and added a rigidbody to the camera (with Gravity being set to 'off'). The problem I'm facing now is that each time the camera hits my terrain, it bounces off and starts spinning like crazy. How do I go about fixing this?

I'm using damien_oconnell's Camera Controller (http://forum.unity3d.com/threads/39513-Click-drag-camera-movement)

Many thanks!