Dice going through walls using Raycast hit distance

Hey, I got a question.

I am trying to roll a dice in a closed box of 4 walls.

The user will swipe the dice make it roll in the direction of the swipe and then bounce off the wall, and I am calculating the raycast distance to apply the force (Velocity) to the dice.

The issue is if I swipe too much, the dice will go through the walls ( colliders ).

I am attaching the code below:

 GameObject gobj = null;
 Plane objPlane;
 Vector3 mO;
 Vector3 Velocity;
 Vector3 lastPos;

 Ray GenerateMouseRay()
 {
     Vector3 mousePosFar = new Vector3(Input.mousePosition.x, Input.mousePosition.y, Camera.main.farClipPlane);

     Vector3 mousePosNear = new Vector3(Input.mousePosition.x, Input.mousePosition.y, Camera.main.nearClipPlane);

     Vector3 mousePosF = Camera.main.ScreenToWorldPoint(mousePosFar);
     Vector3 mousePosN = Camera.main.ScreenToWorldPoint(mousePosNear);
     Debug.DrawRay(mousePosN, mousePosF - mousePosN, Color.green);
     Ray mr = new Ray(mousePosN, mousePosF - mousePosN);
     return mr;
 }

 // Update is called once per frame
 void Update()
 {

     if (Input.GetMouseButtonDown(0))

     {
         Ray mouseRay = GenerateMouseRay();
         RaycastHit hit;

         if (Physics.Raycast(mouseRay.origin, mouseRay.direction, out hit) && hit.transform.gameObject.tag == "Dice")
         {

             gobj = hit.transform.gameObject;
             objPlane = new Plane(Vector3.up * -1, gobj.transform.position);

             //Cal mouse offset
             Ray mRay = Camera.main.ScreenPointToRay(Input.mousePosition);
             float rayDistance;
             objPlane.Raycast(mRay, out rayDistance);
             mO = gobj.transform.position - mRay.GetPoint(rayDistance);


         }

     }

     else if (Input.GetMouseButton(0) && gobj)
     {
         Ray mRay = Camera.main.ScreenPointToRay(Input.mousePosition);
         float rayDistance;
         if (objPlane.Raycast(mRay, out rayDistance))
         {
             gobj.transform.position = mRay.GetPoint(rayDistance) + mO;
             Velocity = gobj.transform.position - lastPos;
             lastPos = gobj.transform.position;
         }
     }

     else if (Input.GetMouseButtonUp(0) && gobj)
     {
         gobj.GetComponent<Rigidbody>().AddForce(Velocity*500);
         gobj = null;
     }
 }

I appreciate your help on this.