I'm having a problem with UI responsiveness

So I have a Canvas and several panels in it with 2D sprites as UI elements.

My Canvas scale mode is set to "Scale With Screen Size" which scales my elements correctly however, depending on the screen size, some elements either overlap like the image below

or it creates unwanted space between panels like the following example

The ideal case would be this one (sorry can't attach more than 2 pics):

How it should be on all screen sizes

Am I missing something? Should I do it via a script? If so, what would you recommend?