StopCoroutine() - how to release memory to the garbage collector?

I'm not sure what you're asking here. If a coroutine is finished (is literally finishes by reaching the end or hit a "yield break;") or stopped, the object that represents the coroutine and the IEnumerator object will be both up for garbage collection. There's no memory leak I'm aware of.

Though some people might do things the wrong way or manually keep references to either the Coroutine object or the IEnumerator object beyond the livetime of the coroutine. Of course as long as you have a reference to one or both of those objects they won't be garbage collected until those references are cleared / go out of extent.

Such general questions are easier to answer when the question has a general example code snippet that demonstrates the issue you talk about. Keep in mind that the managed memory pool might grow as necessary while the program runs, but that pool usually never shrinks again. So in terms of system memory that memory is not released until you close / terminate your application. However the memory is "free" or can be freed inside the C# appdomain when the GC actually runs. So that memory is most likely be reused for other objects within the managed memory. So watching the memory usage in the task manager of window can be misleading. Unless you have a constant and unbounding growth over time there's no issue at all.

As I mentioned earlier many people use StopCoroutine wrong. That's because they usually have the wrong idea about what a coroutine is. A coroutine is not a method. A coroutine is a statemachine object that is created when you call your "generator" method. That's why such methods are often called "generators" because they just generate the statemachine object which actually represents the code inside your IEnumerator. When you start a coroutine you essentially do two things: First you call your generator / coroutine method which generates an instance of the compiler generated statemachine class. The second thing you do is passing this object to Unity's coroutine scheduler by using StartCoroutine. Unity will internally create an instance of its "Coroutine" class and stores the statemachine object alongside somewhere on the native side. The scheduler will now take care of "stepping" / iterating through your statemachine and depending on what values you yield the scheduler will determine when to continue the "next step". Once the IEnumerator reaches the end / a yield break statement the coroutine itself is considered done and Unity will get rid of all references it holds on the native side. At that point the whole thing is purely a managed thing. As long as you don't keep references to any of those instances, they will eventually be garbage collected.

With that information in mind it should be clear that a setup like this WON'T WORK:

 StartCoroutine(MyCoroutine());
 
 // somewhere else
 
 StopCoroutine(MyCoroutine());

That's because each time you call your generator method MyCoroutine, you will create a new statemachine object that is in no way connected to the first one. So that StopCoroutine call has no effect at all since you never started any coroutine with that particular instance of your IEnumerator. If you want to stop a running coroutine you have to pass the SAME IEnumerator object to StopCoroutine that you passed to StartCoroutine, or usually simpler pass the Coroutine instance that StartCoroutine returned to StopCoroutine. So if you want to be able to stop a coroutine externally (which I wouldn't recommend anyways) you have to store either the IEnumerator object or the Coroutine object in a variable and use that reference to stop the coroutine. Keep in mind once you stopped the coroutine you probably want to "null" your reference otherwise, even the coroutine has been stopped, you still hold a reference to the coroutine object so it can not be collected.

The two general approaches to stop a coroutine are those:

 private IEnumerator co;
 
 // start the coroutine:
 co = MyCoroutine(); // create the statemachine and save a reference
 StartCoroutine(co); // pass it to the coroutine scheduler
 
 // stop the coroutine
 if (co != null)
 {
     StopCoroutine(co);
     co = null;
 }

The second and usually simpler way is storing the Coroutine instance

 private Coroutine co;

 // start the coroutine:
 co = StartCoroutine(MyCoroutine());
 
 // stop the coroutine
 if (co != null)
 {
     StopCoroutine(co);
     co = null;
 }