How to lock a slider unless the handle is being interacted with?

At the moment, I want my slider which controls the rotation of an object to only move when the handle has been clicked and dragged. I don't want it to be able to jump to wherever I click on the slider?

Any ideas?