gameObject goes through wall with onMouseDrag

The issue at hand is that whenever I use OnMouseDrag() function on a GameObject (a ball), too fast, the ball is then dragable through the wall. When I do a slow drag this doesn't happen. The gameObject has a rigidBody attached to it and all of my walls have box colliders attached to it.

I have been wrapping my head on this for the last two weeks and tried different solutions without resolving my issue.

I'm using a raycastHit to get this working and use a transform.position to move the ball to a new vector whenever the ray hits within a certain distance from my walls This works fairly well but dragging needs to be done slow or normal. Fast drag will move the ball through the wall.

Tried using the settings in Time manager in order for the physics engine too.

The only thing that may work is perhaps to include FixedUpdate() within onMouseDrag() function but that results in an error. Not sure how that would work and not familiar with coroutines ans such. Below is my script. Perhaps someone in the community is able to help me out?

 #pragma strict
 
 private var screenPoint:Vector3 ;
 private var offset:Vector3;
 private var speed : float = 10;
 static private var currTrans : Transform = null;
 var cursorTexture : Texture2D;
 var cursorMode : CursorMode = CursorMode.Auto;
 var hotSpot : Vector2 = Vector2.zero;
 
 function  OnMouseOver() { 
  
     screenPoint = Camera.main.WorldToScreenPoint(gameObject.transform.position);
     offset = gameObject.transform.position - Camera.main.ScreenToWorldPoint(new Vector3(Input.mousePosition.x, Input.mousePosition.y, screenPoint.z));
     currTrans = transform;
     renderer.material.color = Color.green;
 }
 
 function OnMouseEnter () {
     Cursor.SetCursor(cursorTexture, hotSpot, cursorMode);
     screenPoint = Camera.main.WorldToScreenPoint(gameObject.transform.position);
     offset = gameObject.transform.position - Camera.main.ScreenToWorldPoint(new Vector3(Input.mousePosition.x, Input.mousePosition.y, screenPoint.z));
     currTrans = transform;
     renderer.material.color = Color.green;
 }
 
 function OnMouseExit () {
     Cursor.SetCursor(null, Vector2.zero, cursorMode);
       renderer.material.color = Color.white;
       print ("");  
 } 
 
 function OnMouseDrag(){  
 
     if (currTrans != transform) return;{
     }
                                                                                                                                                                                                                                                                                                                    
     var curScreenPoint:Vector3 = new Vector3(Input.mousePosition.x, Input.mousePosition.y, 9.2f);
     var curPosition:Vector3 = Camera.main.ScreenToWorldPoint(curScreenPoint) + offset;
     transform.position = curPosition;
     
       
     //Debug.Log("Position of ball is " + curPosition.x + ", " + curPosition.z);
     
     var hit : RaycastHit;
         if (Physics.Raycast (transform.position,Vector3(curPosition.x,curPosition.y,curPosition.z), hit)) {
             var distanceObject = hit.distance;
             if (distanceObject < 0.4f){
             gameObject.transform.position=new Vector3(curPosition.x+0.3f,curPosition.y,curPosition.z+0.3f);            
             }
         }
 }