[4.6 UI] Layouts messes with my object with/height even though force width/height is unchecked

Namely they force my objects to be same size as the image used for them (e.g. 144x144 in case of my panels), even if size before applying layout (activating/adding component) is smaller/bigger. Because of that...

instead of

How to fix that? I want text to fit nicely between corners of the panel and I can't do that while still having that unwanted resize.