Object movement like in hologram app

I want to move rotate my objects using gestures. I want to achieve the behaviour as it is there in the default Holograms app in the hololens. Though I am able to get the movement and rotation working to some extent, but my holograms bypass the spatial surface. I want them to behave like rigid bodies. I tried adding rigid body component but that doesn't seems to be working. Can someone guide me in right direction of what components/scripts I can use/write to get this working?