How to detect click on naked panel?

In my game I have a panel opening up to show stats and allow configuration. The panel should be closed again when it is clicked at a position no other input element exists.

• When clicking on pure panel, close

• When clicking on some text, close

• When clicking on a button, don't close

• When clicking on a toggle, don't close

Already tried EventSystem.current.IsPointerOverGameObject(), sadly this also closes the panel when clicking on a button or checking the toggle.