Changing Time.fixedDeltaTime to create slowmotion messes up with physics.

I am trying to create a simple slow motion changing timeScale but I have some issues.

First I used Time.fixedDeltaTime so I can properly watch the slowmotion itself, but is changes something with physics. The next Gif shows a ball getting a force and passing through a tube without slowmotion and then with slowmotion.

I followed Brackeys video about slow motion but it ends to the same problem.

 void ToggleSlowMotion()
     {
         slow = !slow;
 
         if(slow == true)
         {
             Time.timeScale = slowdownFactor;
             Time.fixedDeltaTime = Time.fixedDeltaTime * slowdownFactor;
         }
         else
         {
             Time.timeScale = 1;
             Time.fixedDeltaTime = Time.timeScale * 0.02f;
         }
 
       
     }