Slow Motion Physics Lag Spikes

Hello. I'm creating a game where the player is able to slow down time to dodge bullets, etc. (bullet time effect). I'm doing this with the code below:

 public bool timeModify = false;
 
 void Update(){
 if (Input.GetKeyDown (KeyCode.CapsLock)) {
                 timeModify = !timeModify;
    }
 if (timeModify) {
                 blur.blurAmount += 0.01f;
                 Time.timeScale -= 0.02f;
                 Time.fixedDeltaTime = Time.fixedDeltaTime * Time.timeScale;
                 bgm.pitch -= 0.01f;
                 if(Time.timeScale <= 0.4f){
                     Time.timeScale = 0.4f;
                 }
                 if(bgm.pitch <= 0.65f){
                     bgm.pitch = 0.65f;
                 }
             }
             if (timeModify == false) {
                 blur.blurAmount -= 0.05f;
                 Time.timeScale += 0.02f;
                 Time.fixedDeltaTime = 0.02f * Time.timeScale;
                 bgm.pitch  += 0.01f;
                 if(Time.timeScale >= 1f){
                     Time.timeScale = 1f;
                 }
                 if(bgm.pitch >= 1f){
                     bgm.pitch = 1f;
                 }
             }
 }

This creates a gradual slow down effect when the player hits capslock, as opposed to an immediate slow down. The trouble is, I'm running into extremely bad physics performance. It's strange, because this project even runs smoother in the Editor than in the actual build. The profiler physics graph spikes like crazy when I press capslock to manipulate the physics as well. This brings the FPS from 55-60 to a ridiculous 5-10, and then it will rise back up.

Does anyone have any idea on how to optimize/fix this? I had no idea playing with Unity's physics would be so costing to performance.

Here is a sample build of the project to test out (zipped): https://dl.dropboxusercontent.com/u/261955553/PhysicsLag.zip