Toggle with key input only turning off

Is this script attached to the "glasses" GameObject itself? If so, an inactive GameObject won't run its Update() function.

For reference, you could also give yourself some more feedback in the console to ensure things are working as expected:

 void Update()
 {
     if(Input.GetButtonDown("glassesKey"))
     {
         glasses.SetActive(!glasses.activeSelf);
         Debug.Log(string.Format("{0}: \"glassesKey\" pressed: {1}.activeSelf = {2}", gameObject.name, glasses.name, glasses.activeSelf));
     }
 }

^{Edit: gave example a little more detailed diagnostic output}