Performance of thousands of unmoving, deactivated gameobjects

,I am creating a visual novel game, and I'm in the process of rolling my own dialog engine for it. I've tried to use techniques such as prefabs and scriptable objects for the different conversation nodes, but because a conversation has many conditions controlling its "available" state, and each conversation has variations based on yet other conditions, it seems to be easiest to just store everything in the scene permanently.

I am concerned with the performance of this solution as my game grows. None of the objects have any physical components and they only run a small piece of initialization code, and they are all deactivated 95% of the time.

Do I need to be worried about this? Is there a better solution? Remember that I need to keep track of complex unlock relationships between the elements.