Insane amount of CPU power used to activate UI inventory and equipment windows

Hey everyone,

I've been having random issues lately, that never occured before, with my inventory UI. I have not touched it for legit like a month and yet sometimes opening inventory results in massive fps drop on a frame when the key to open inventory is pressed.

The whole process sometimes takes over 40 ms! Which is just insane. I don't know how this thing triggers, but when I restart unity, the spikes usually vanish for a while, but after some time of testing and working with my code the issues happen again. However even when the issue does not appear, opening invenotory takes around 7-8 ms (4-5 ms to update canvas and 2-3 ms to activate Inventory and Equipment elements on the canvas)

Here on the image you can see the actual results from the profiler. The huge spikes happen when I open the Inventory (which activates game object Inventory and Equipment on my canvas)

Image with the better scenario: https://prnt.sc/25s7d35

And here is my main canvas in the hierarchy. Basically this one only has hp bar, mana bar, inventory and equipment UI. Inventory has 40 slots.

Since I can't add anymore attachments to this post, I'm sending the rest of the information via links to the images.

https://prnt.sc/25s54pi

This image contains the structure of my inventory slot. Inventory slot parent has inventory slot script, that handles adding items into the inventory slot, updating icon, showing remove button and updating item stack size if needed.

ItemButton contains image and button component and on click script that has IPointerClickHandler to detect clicks in given interval.

RemoveButton has image and button with a function that is called in the inventory slot script on the InventorySlot object.

ItemStackSize has text and an outline.

None of the inventory uses update method or anything similar. Inventory only updates when items are added into it, when they are consumed (potions) or when they are equipped or unequipped.

These fps drops only happen when you open the inventory.

This is how my UI looks: https://prnt.sc/25s6mrn

Anyone know how to fix these issues? They just don't make any sense at all to me. I know that changing one thing at canvas forces the game to recalculate whole canvas again, but this problem only appears when I activate the inventory object. Equipment object in fact has no effect at all on the game. As you can see on the profiler image, in the worst case scenario for some reason it takes whole 12 ms to just activate game object and 28 more to rebuild the canvas. Even in the best case scenario (which is majority of the time, especially after I restart unity) the activation of UI takes 8 ms which is still insanely crazy and I just have no idea why is this happening.

Thanks to anyone for any help provided.