How to use the Undo snapshot system?

Undo.RegisterUndo basically says "this item is about to change, so please remember its state so I can undo it later." It should only be called when you know the target object is about to be changed.

In contrast, the snapshot system lets you handle the case where you don't know if the object will be changed or not -- you can save the state of the object and decide later whether or not to register that state for an undo operation.

Note that the simple:

 Undo.RegisterUndo(target, "Modify");

Is equivalent to:

 Undo.SetSnapshotTarget(target, "Modify");
 Undo.CreateSnapshot();
 Undo.RegisterSnapshot();

You can use the snapshot system like this in your custom editor:

 public override void OnInspectorGUI()
 {
     // Create the snapshot before you start editing.
     Undo.SetSnapshotTarget(target, "Modify");
     Undo.CreateSnapshot();

     // Do whatever you want to edit your object.
     DoMyFunkyCustomEdits();

     // If something changed then register the snapshot for undo.
     // You can either use Unity's GUI.changed or keep track of your
     // own flag indicating that your code modified something.
     if (GUI.changed)
     {
         EditorUtility.SetDirty(target);
         Undo.RegisterSnapshot();
     }
 }

Note that Unity is clever enough to group related Undo operations -- a sequence of operations on the same object with the same name (e.g. "Modify" above) will turn into a single undo-able operation.

Note that this answer is relevant to the question, but the solution seems more complex than necessary. What I've documented here is working for me, I hope it helps others.

Alex, it sounds like this is almost exactly what you're doing, and that AndyP's suggestion of calling EditorUtility.SetDirty may be all you're missing. (Disclaimer: in my custom editor I also call SerializedObject.SetIsDifferentCacheDirty, though I'm not certain it's needed.)

Use Register before the change: http://unity3d.com/support/documentation/ScriptReference/Undo.RegisterSceneUndo.html

But it sounds like one or the other: user Register to capture the entire state of the editor, use Snapshot on particular objects.

I'm assuming you are trying to make undo work with handles that update something continuously as the user moves them. You didn't make it absolutely clear in your question, but I'll answer with what I know anyway.

Have you tried calling EditorUtility.SetDirty? I found adding a call to this magically makes snapshots work without anything other than SetSnapshotTarget. Here's an example:

Undo.SetSnapshotTarget(target, "move point");

for (i = 0; i < target.points.Length; ++i) { Vector3 newPoint = Handles.FreeMoveHandle( target.points[i], Quaternion.identity, handleSize, pointSnap, Handles.DotCap); }

if (GUI.changed) EditorUtility.SetDirty(target);

Disclaimer: What I'm doing is not how I read it was supposed to be done, but it works for me. However, I currently have a weird problem with the function that contains this code and I'm not sure whether or not it is the cause.