Issue removing element from List

I have the following code:

 if(remainingObjectives.Count > 0)
         {
             randomNumber = Random.Range(0, remainingObjectives.Count);

             //This defaults to -1 when on the last element
             element = remainingObjectives.IndexOf(randomNumber);
 
             Debug.Log("Random Num: " + randomNumber);
             Debug.Log("Element: " + element);
 
             for (int i = 0; i < remainingObjectives.Count; i++)
             {
                 if (i == randomNumber)
                 {
                     if (!activeObjectives.Contains(element))
                     {
                         activeObjectives.Add(element);
                         remainingObjectives.Remove(i);
                         break;
                     }
                 }
             }
         }

I have two elements in the list remainingObjectives, I have it setup so that when I press Q it will execute this code, when I press Q, it chooses a number between 0 and 1, I have the element variable to tell me what the value at the index is.

If 0 is chosen first, the first element is removed from the list and added to another. But when I try and execute it again, it won't remove the last element, also, the element variable is set to -1 when it should be 1.

Hard to get my point across on this but if anyone sees an oversight on my end, I'd appreciate it.