Rotation and moving object simultaneously problem?

Hello Everyone, I am new on UNITY and i am making a one game in which simple swipe ball will swipe in X and Y Direction..But the problem is when i am trying to swipe as well as rotating my ball on that time some variations or hick ups are coming..

For applying force in Y direction i am using

              ball.rigidbody.AddForce(-touchDeltaPosition.x * 0.004f,touchDeltaPosition.y * 0.013f, 0, ForceMode.Impulse);

`For Rotation i am using

ball.rigidbody.AddRelativeTorque(touchDeltaPosition.x * 28.0f,0,0);

On ball i attached rigidbody and my Constraints Z are checked, When i am swiping simply it is working but when some Bricks(Obstacles) come after collision with that ball is moving some time like Zig-Zag or hick ups are coming.

Anyone can help me please...? Thank You