Polybrush and Blender meshes

I'm trying to use polybrush on an object that I've made in Blender. As soon as I touch the object with the brush the object just screws up. Demonstrated in image below, before and after. Anyone knows that could be causing it?