make a wedge moveable

Hello i am making a basic ball rolling game and am having a few issues with the wedge you have to move it with your ball to get into a tube when i add mesh collider with rigid body you move it when you get on it how would i fix this so it doesnt warp around a bunch