Renderer.isVisible returning true with back side of Quad Primitive

I'm creating a grid system in a box, and would like to display the grid on the interior walls only when the the walls are being rendered.

I am creating 6 Quad Primitives, rotating and positioning them such that the interior of the box is visible, while the exterior walls are culled and therefore invisible.

The problem I'm running into is when trying to check to see if the Main Camera can see the Renderer of a wall -- it will always return True if the wall is within view of the camera, even if it is the exterior of a wall and thus 'invisible' as it is culled.

The code snippet I'm using to test this specifically with the Main Camera is below:

 public static bool CheckVisible(Renderer renderer, Camera camera)
         {
             Plane[] planes = GeometryUtility.CalculateFrustumPlanes(camera);
             return GeometryUtility.TestPlanesAABB(planes, renderer.bounds);
         }

When the wall is totally out of view from the camera, it does and work properly as my debugging tells me, however my concern is to ensure if this wall is not visible to the eye, even if it is technically still there, it won't be to the camera.

Suggestions? Thanks!