Prevent a fast moving ball from passing through a wall

I need help with preventing a fast moving ball from passing through a wall. For a testing environment I have a basic sphere with a sphere collider radius .5, and a rigid body that is set to continuous collision detection. The wall is 1 unit thick with a box collider, and the paddle is .8 units thick and also has a box collider. The distance from the paddle to the wall is 57. Every time the ball hits the paddle its velocity is increased by 1. With out fail every time the velocity of the ball reaches 54 it will pass through the wall. I have tried all the built in collision detection methods, but regardless of which is selected it will pass through the wall at velocity total of 54. Here is an example of code used to increase the velocity of the ball, this code is attached to the paddle; I have cut out all the math for dealing with the angles of the bounce, and would like to just focus on the speed issue.

 using System;
 using UnityEngine;
 using System.Collections;
 public class PaddleScript : MonoBehaviour
 {
 public float ballSpeed = -20f;
 public float velX = 0;
 public float velY = -20f;

    void OnCollisionEnter(Collision collision)
    {

       if(velY>0)
       {
          velY = ballSpeed - Math.Abs(velX);
       }
       else
       {
          velY = (ballSpeed - Math.Abs(velyX));
       }
       collision.rigidbody.velocity = new vector3(velX,velY,0);
       ballSpeed += 1;
     }
 }