Prevent Camera from going off terrain

I have an orthographic camera that can zoom in and out to the ground. I can control the zoomfactor by Mathf.Clamp, and that works perfectly well.

 Camera.main.orthographicSize = Mathf.Clamp (Camera.main.orthographicSize, minZoom + 1, maxZoom -1);

Noe I want to clamp the camera view to the terrain it is looking at. So my first try was to clamp the transform to the transform of the terrain. But that won't work, because the camera can see a bit to the right and to the left, depending on the zoom factor (orthographicSize). So I could set an hard coded offset to that, but that only works for one zoomfactor, not for all.

So my current try is this to use raycast:

     int mask = (1 << LayerMask.NameToLayer ("Terrain"));
     RaycastHit terrainHitInfo = new RaycastHit ();
     Ray rayLeft = Camera.main.ScreenPointToRay (new Vector3 (0, Camera.main.pixelHeight / 2, 0));
     Ray rayRight = Camera.main.ScreenPointToRay (new Vector3 (0, Camera.main.pixelHeight / 2, 0));
     Ray rayTop = Camera.main.ScreenPointToRay (new Vector3 (Camera.main.pixelWidth / 2, Camera.main.pixelHeight, 0));
     Ray rayBottom = Camera.main.ScreenPointToRay (new Vector3 (Camera.main.pixelWidth / 2, 0, 0));

     if (!Physics.Raycast (rayLeft, out terrainHitInfo, mask)) {
         Camera.main.transform.Translate(new Vector3 (camSpeed, 0, 0) * Time.deltaTime, Space.World);
     }

     if (!Physics.Raycast (rayRight, out terrainHitInfo, mask)) {
         Camera.main.transform.Translate(new Vector3 (-camSpeed, 0, 0) * Time.deltaTime, Space.World);
     }
     if (!Physics.Raycast (rayTop, out terrainHitInfo, mask)) {
         Camera.main.transform.Translate(new Vector3(0, 0, -camSpeed) * Time.deltaTime, Space.World);;
     }
     if (!Physics.Raycast (rayLeft, out terrainHitInfo, mask)) {
         Camera.main.transform.Translate(new Vector3(0, 0, camSpeed) * Time.deltaTime, Space.World);
     }

The idea is to recognize with raycast, whether the left, right, top and bottom view of the camera still hits the terrain. And if not, move back.

But that doesn't work. I don't even understand why. An effect of this code is, that the if statement is true, when there is a mountain or hill in the terrain. But not, when there is nothing. Very strange.