Is it possible to create physics pushable particles?

I'd like to be able to simulate dirt and small stones. I've seen it done before in other games, where you have pile of dirt that can be brushed away or small stones that can be moved - although the stones are more likely objects, but dirt feel s like it would best be simulated with particles. The ultimate goal would be to use a broom or brush or some similar tool to clean up the dirt.

The other part would be to somehow 'freeze' the particles so that they don't keep replacing themselves while they're being brushed away.