False collision at the edge of hole when using high gravity value

I'm developing a little board game. I have a board, which I modeled in Blender. The board has round holes in it. Player moves a ball around the board and tries not to drop into the holes. The ball can move very slow or very fast.

I had to increase game's gravity a lot to make fast moving ball drop to a hole. Otherwise the ball just flew over the holes with a little bump. After increasing the gravity, I noticed that sometimes the ball bounces off from the edge of a hole. It's like some false collision and it happens mostly when the ball is moving very slowly.

I created a test project to reproduce the problem. In it I have different kind of holes to test if the vertice count of the hole's edge matters. But it seems that it makes no difference. Holes are made in Blender using a cylinder and a boolean modifier. Holes in this test model have 8, 12, 16, 20, 24, 28, 32 vertices and the last one is smoothed.

Low gravity and low speed, works fine:

High gravity and high speed, works fine:

Low gravity and high speed, ball flies over the hole:

High gravity and low speed, ball bounces from the edge:

So to make this work, I have to fix the bouncing problem or to use default gravity and figure out another solution for fast moving ball. I tried to increase the solver iteration count and min penetration for penalty values from Unity's PhysicsManager to fix the bouncing problem. It may have helped a little, but not solve the problem completely. I'm now running out of ideas how to fix this.