CustomPropertyDrawer: foldout issues when switching properties (simulating weak references)

Intro: I want weak references to avoid infinite serialization depth. E.g.

 public class Foo
 {
     public Foo[] children; 
 }

The above of course triggers unity alerts for serialization depth. So let's change it to the following:

 public class FooId
 {
     public int id;
 }
 
 public class Foo
 {
     public FooId[] children;
 }
 
 public class SomeBehaviour: MonoBehaviour
 {
     public List<Foo> pool;
     public FooId root;
 }

So, now I have my tree hierarchy of arbitrary depth, so unity doesn't complain. But, viewing the tree in the inspector is problematic, as I have to manually jump back and forth. So, to tackle this, I tried to create a custom property drawer for the FooId, which looks like this:

 [CustomPropertyDrawer(typeof(FooId))]
 public class FooIdDrawer : PropertyDrawer
 {
     void SwitchProp(ref property)
     {
           var e = property.GetValue<FooId>();
           var so = new SerializedObject(SomeBehaviour.Instance); // it's singleton
           var prop = so.FindProperty(string.Format("pool.Array.data[{0}]", e.id));
           property = prop;
     }

     // Draw the property inside the given rect
     public override void OnGUI(Rect position, SerializedProperty property, GUIContent label)
     {
         SwitchProp(ref property);
         EditorGUI.PropertyField(position, property, label, true);
     }
 
     public override float GetPropertyHeight(SerializedProperty property, GUIContent label)
     {
         SwitchProp(ref property);
         return base.GetPropertyHeight(property, label);
     }
 }

So, in the above, SwitchProp replaces the FooId property with the Foo property. BUT the foo property in the tree is a foldout that never expands... What could be the problem?

I tried changing OnGUI function to the following:

     public override void OnGUI(Rect position, SerializedProperty property, GUIContent label)
     {
         var propOrg = property;
         SwitchProp(ref property);
         property.isExpanded = propOrg.isExpanded;
         EditorGUI.PropertyField(position, property, label, true);
         propOrg.isExpanded = property.isExpanded;
     }

This made the foldout work, but the rect is all wrong and even if I set it manually to something tolerable, the children foldouts do not work (click does nothing). So, clearly the isExpanded swap works only for a single level, locally.

Any ideas?

Thanks.