What is happening with my Multiple Display setup and clear flags?

Setup

In my project I am utilizing multiple displays and these have been working just fine.
The first display shows my 3d content on a main camera with deferred rendering and I'm just using the second display to show some accompanying text information only in a screen space UI with a forward rendered camera.

Recently I needed to utilize some ray casting to detect mouse over specific 3d objects and this worked great in isolation. However upon integrating that into my multiple monitor setup, I began to experience some issues. I have a 3d knob that when rolled over pops up a set of arrows that allow the user to interact with them and turn them and then another object further out that hides everything as the user moves away from the knob. Again, these work fine in isolation on one display, only when adding a second display does the problem appear.

Issue

At first I thought my mouse was not being detected but upon further investigation and deep dive into the problem, I've discovered the problem is related to display 2 and it's camera. If I change the camera target display 2 over to display 1 it works. Also if I change the camera 2 Clear Flags setting to either Depth only or Don't clear it works fine as well. This however causes my text to pile up and is unreadable.

Question

Why would the Clear Flags set to solid color on a camera output to display 2 interfere with a 3d object or mouse over on display 1?

I can post code if necessary but in this case there probably isn't much code related to what is going on I don't believe. I have been stumped and working on this issue for a couple of days now trying to understand and figure out the solution. If more information is needed, I can add additional information as requested.

Thanks in advance.