Large serializable field in a MonoBehaviour causing poor performance in Inspector

I have asset bundles where I group 3d models with additional meta data. A key requirement is I need an image of each 3d model. The simplest solution I've found is to generate these images/snapshots in the Unity Editor using Editor APIs, and storing the result with the model.

So I have a couple classes that look like this (simplified here):

 [Serializable]
 public class Model {
     [HideInInspector]    //Serialized, but not shown in inspector
     public byte[] texture;
     public GameObject model;
 
     public string name;
     public string description;
 
     public void GenerateTexture() { ... } //This is called with a custom Inspector
 
     public Texture2D Texture {
         get {...}    //Converts this.texture to an actual Texture2D
         set {...}    //Converts a Texture2D to a byte[]
     }
 }
 
 public class ModelPack:MonoBehaviour {
     public List<Model> gameModels;
 }

ModelPack(s) are stored in asset bundles which are then loaded at runtime.

This works perfectly, except one minor-ish problem: performance in the Unity Inspector is terrible. The Unity (Editor) profiler shows a large amount of garbage collection happening in every UI update causing the whole Unity UI to freeze and skip. For the longest time I thought there was an issue with the custom inspector I wrote for the ModelPack/Model classes, but even completely disabling that the performance issue remains. I've finally come to the conclusion that the problem seems to be the texture byte array. If I mark it as NonSerialized then performance is fine in the Inspector.

I suspect the problem is that it's constantly getting/setting and serializing/deserializing the value of the whole object in the inspector, even though I have ensured the texture value itself isn't changing unless the associated model is changed. Since the texture is rather large (megabytes in size) that could create quite a strain.

Does this make sense? Is there a recommended limit on how much data should be stored when serializing a Unity MonoBehaviour? Is the entire MonoBehaviour serialized in one go every time? Is there a way to filter out that one field, or will I have to find an alternate way of storing these large byte arrays such as saving them to files?