methods for mouse picking that do not use physics?

Are there any methods for using a mouse to select a gameObject without using physics? I'm building a gizmo so I can drag an object around on the screen with 3d object handles. When you click on the object the handles appear. But if the handles have to have colliders then they create all sorts of havok with other physical objects in the scene. Especially if the original object I'm moving has joints. I know I can ignore collisions between pairs of objects, but I never know which gameObject might be nearby. Is there an alternative to Physics Raycast for choosing objects with the mouse or a way to ignore all collisions for a collider except the raycast collision?

Thanks, Dan