Sphere on cube floor doesn't rotate after too much delay

I made a simple area where you rotate the actual ground to roll a sphere around. If you start rotating (by using WASD) immediately then everything is good and the sphere rolls fine. But if you wait for a second or two after the ball falls onto the ground of the area and try it, the ball stays still and the floor phases through it as you rotate the ball.

Here is a picture of the ball and the area

The second pic is what happens if you wait a few seconds before rotating the ground. The ball is stuck in the same place. If I kept rotating, the ground would go through the ball until I mess with it the the ball would roll on the underside of the ground.

Here is the script I'm using to rotate the ground.

 using UnityEngine;
 using System.Collections;

 public class moveGroundScript : MonoBehaviour {

 public int speed = 35;

 // Use this for initialization
 void Start () {
 
     

 }
 
 // Update is called once per frame
 void Update () {

     if (Input.GetKey(KeyCode.A)) transform.Rotate(Vector3.forward * speed * Time.deltaTime);

     if (Input.GetKey(KeyCode.D)) transform.Rotate(-Vector3.forward * speed * Time.deltaTime);

     if (Input.GetKey(KeyCode.W)) transform.Rotate(-Vector3.left * speed * Time.deltaTime);

     if (Input.GetKey(KeyCode.S)) transform.Rotate(Vector3.left * speed * Time.deltaTime);

 }

 }

The ball has a rigidbody component with a mass of 1 and the problem only happens when you wait before rotating. Can someone please tell me what is happening and how to fix it? Thanks.