Return Objects hit by raycast, check against a array of all objects?

At start, I would retrieve a array of all objects in a specific layer. I would like to use multiple raycasting within in that layer (<---- done all that so far) to return an array of objects that were hit, and then check which objects of the layer were NOT hit, by creating a third array, by taking the original array, and for loop checks against the hit array. Then destroy the second(hit) and third array after they do their business.

Advice?

Thank you