Picked up cube passes through walls

Hey guys, Been searching the internet for hours and I can't figure out why this is happening. I have a scene with a first person character controller and a cube. The floor and 4 walls are all just stretched cubes. I have this script attached to the camera within the first person controller

 using UnityEngine;
 using System.Collections;

 public class PickUpObjectTest : MonoBehaviour {
 
 private Rigidbody rigid;                //For storing the rigidbody if successfully hit
 private int holdDistance;                //The distance that the object will be held away from the player
 private enum holdState {Free, Held};    //States in which the player can be regarding external objects
 private holdState state;                //State variable
 private float increment;                //The speed at which the object moves towards the camera
 private Quaternion rotation;            //The desired rotation of the object when picked up
 
 // Use this for initialization
 void Start () {
     holdDistance = 3;
     state = holdState.Free;
     increment = 1.0f;
 }
 
 // Update is called once per frame
 void FixedUpdate () {
     
     switch (state){
         
     case holdState.Free:
         if (Input.GetKey ("e")){
             Ray ray = camera.ScreenPointToRay(new Vector3(camera.pixelWidth/2, camera.pixelHeight/2, 0));
                Debug.DrawRay(ray.origin, ray.direction * 10, Color.yellow);
             RaycastHit hit;
             if (Physics.Raycast(ray, out hit, 10)){
                 if (hit.transform.tag == "Pickable"){
                     rigid = hit.rigidbody;
                     state = holdState.Held;
                 }
             }
         }
         break;
         
     case holdState.Held:
         
         Vector3 nextPos;
         
         if (increment < 100.0f)
             increment++;
         
         rigid.transform.parent = transform;
         
         nextPos = Vector3.Lerp(rigid.position, transform.position + transform.forward * holdDistance, increment);
         
         if (!Physics.Raycast(rigid.position, nextPos, 2)){
             rigid.position = nextPos;
         }
         
         //Rotation of the object
         Vector3 direction = transform.position - rigid.transform.position;
         rotation = Quaternion.LookRotation(direction);
         rigid.transform.rotation = Quaternion.Slerp(rigid.transform.rotation, rotation, increment);
         rigid.transform.rotation = Quaternion.Euler(rigid.transform.rotation.x,
             rigid.transform.rotation.eulerAngles.y, rigid.transform.rotation.eulerAngles.z);
         
         rigid.useGravity = false;
         
         if (Input.GetKey ("e")){
             increment = 1.0f;
             rigid.useGravity = true;
             rigid.transform.parent = null;
             state = holdState.Free;
         }
         break;
     }
   }
 } 

I can pick up the cube and move around the scene with it however when I look down at the ground or walk near a wall the cube passes right through it. I have tried walking very slowly towards the wall and the cube doesn't immediately pass through, rather it pops through when I get close enough. Any help would be IMMENSELY appreciated as I am trying to hold on to the few remaining strands of hair I have left :(

Thanks Guys