other.contacts[0].normal changes over time

I'm working on a "Breakdown"-ish game, to brush up on my coding, and ran into this issue, that I can't seem to find a solution to.

My problem lies in the bounce, when the ball hit the walls (and blocks). I use

 void OnCollisionEnter(Collision other)
 {
     Vector3 normal = other.contacts[0].normal;
     modifier = Vector3.Reflect(modifier, normal);
 }

 void FixedUpdate ()
 {
     rigidbody.velocity = new Vector3(transform.forward.x*modifier.x, transform.forward.y*modifier.y, transform.forward.z*modifier.z)*speed;
 }

to move and bounce the ball, and my problem lies in the "other.contacts[0].normal".

The problem is that the normal should stay constant for the various walls and objects, as they don't move or rotate, but they change over time.
e.g. the normal for the left wall, start out as (-1.0, 0.0, 0.0) as it should be, but it then changes to (-0.9, -0.5, 0.0), then (-0.8, 0.6, 0.0), (-0.8, -0.6, 0.0), (-0.7, 0.7, 0.0), and so on.

I tried using this

 Vector3 normal = other.contacts[0].normal;
 int normX = normY = normZ = 0; 
 if(normal.x > 0.75f) { normX = 1; } else if(normal.x < -0.75f) { normX = -1; } else { normX = 0; }
 if(normal.y > 0.75f) { normY = 1; } else if(normal.y < -0.75f) { normY = -1; } else { normY = 0; }
 if(normal.z > 0.75f) { normZ = 1; } else if(normal.z < -0.75f) { normZ = -1; } else { normZ = 0; }
 normal = new Vector3(normX, normY, normZ);
 modifier = Vector3.Reflect(modifier, normal);

as a workaround, but when the normals go outside my limits, the problem returns.

what makes this even more frustrating is that this only happens if I spawn a new ball from code, or restart unity.
If I create a new ball manually, the normals stay the same, untill I restart or build the game.
EDIT
Specified the issue