Keep Camera from player at a certain distance while orbiting around player

So far I have everything working perfectly with the following code. The problem is, using transform.forward while keeping distance sets the camera's Y-axis to the same height as the player's Y-axis. I want the camera to be a bit higher than the player rather than equal to the player.

The reason I am not simply attaching the camera to the player is because the player is a sphere that is constantly rotating. The method I am using keep the camera separate from the player while still keeping its distance. The player's movement is based on the direction the camera is facing, if that matters at all.

using UnityEngine; using System.Collections;

 public class RotateCamera : MonoBehaviour
 {
     public float rotateSpeed = 300.0f;
     public float distance = 3.0f;
 
     public GameObject player;
 
     // Use this for initialization
     void Start ()
     {
 
     }
     
     // Update is called once per frame
     void Update () 
     {
         // If A key is pressed, rotate camera to the left
         // If D key is pressed, rotate camera to the right
         // if S key is pressed, rotate the camera 180 degrees around player
         if (Input.GetKey(KeyCode.A))
         {
             transform.RotateAround(player.transform.position, Vector3.down, rotateSpeed * Time.deltaTime);
         }
         else if (Input.GetKey(KeyCode.D))
         {
             transform.RotateAround(player.transform.position, Vector3.up, rotateSpeed * Time.deltaTime);
         }
         else if (Input.GetKeyDown(KeyCode.S))
         {
             transform.eulerAngles = Vector3.Lerp(transform.eulerAngles, transform.eulerAngles + 180f * Vector3.up, 100 * Time.deltaTime);
         }
     }
 
     void LateUpdate ()
     {
         // Keep camera a specified distance away from player
         transform.position = player.transform.position - transform.forward * distance;
     }
 }

EDIT: basically I want to combine these two lines of code:

transform.position = player.transform.position - transform.forward * distance;

Combined with

 transform.position = player.transform.position - transform.up * distance;