iPhone joypad interfering with MouseLook

I'm developing an iPhone/Android FPS which is controlled with a single joypad for movement and using mouselook for rotating the camera. This works fine in that the user can look around by swiping their finger, but whenever a finger is on the lower-left joypad this causes the mouselook to instantly point the camera in that direction.

Is there a way to stop joypad touches interfering with mouselook?