Scroll Rect not scrolling content

I've added a Scroll View game object and I've added several Image objects under the Content object. I disabled Vertical movement and I have the Movement Type set to Elastic. I removed the Mask from the viewport, since I don't want the content to be cut off.

For some reason, the scroller is not working. It's scrolling when I drag my pointer, but when I release the pointer it goes immediately to the beginning. When I set the movement type to Unrestricted, it's scrolling fine, but it never stops.

What could be wrong?