Network.Destroy () Not working.

So i am making this netoworking game and it is just like minecraft at the moment but i plan on tweaking it a ton. But one problem i am having right now is that when i try to Network.Destroy a raycast hit object, it says "could not destroy gameObject because no network View is attached to it." But if i attach a network view to it, the game gets suuuuper laggy and then nothing works. Any ideas on how to destroy the objects?