Generating a sphere with irregularities

I wonder how to go about generatic a procedural sphere with irregularities.

The idea is to generate a spheric planet. So it's basically a sphere, but it would have some cliffs and things like that. I know that perlin noise is a good way to generate irregularities, but how would I apply it to the sphere, and more precisely to the geometry (creating the corresponding polygons, etc)?

The planet is of little size (imagine Animal Crossing villages or mario galaxy levels) so I don't need any realistic environment creation, just something that creates a little bit of variation.

I also need to generate a custom collider that has the same polygons as the mesh. Is this possible?