UI Toolkit screen reader

Hey, answering because this still shows up high in search results.

Yes, it's possible but...

Unity has no built in support for screen readers: It would be natural to assume it could because the new UXML and USS system seems so similar to HTML and CSS, and all web browsers support Screen Readers. It's also not really that far of a stretch to expect one of the biggest engines in the world with a worth of 13~ Billion dollars to support Screen Readers in SOME way.

But it doesn't, at all, for Runtime or Editor.

In order to support it across all platforms, Unity would have to make an API Layer that that can send UI navigation events to the screenreader on the platform, and do this automatically when using UI Toolkit. Unity would have to manage the drivers for each (Android, Iphone, Xbox, PS5, PC) so that developers only had to deal with the Unity internal API, no matter what platform they are targeting.

This is what Tolk does for Windows screen readers.

Here's an abridged history of Unity threads asking about Screen readers

and Unity sometimes answering perhaps-maybe-at-some-point-thinking-about it:

August 2014: "Is the new UI system going to support screenreaders?" (No answer from Unity)

Context: this is back when the old UI system was "new". History has since repeated itself with the new UI Toolkit system. https://forum.unity.com/threads/screen-reader-access.264938/

May 2015, Visually Impaired player says Unity would be accessible to blind players if only the Editor windows could be selected properly and read to screenreader. (No Answer) https://forum.unity.com/threads/screen-reader-accessibility.328623/

June 2015 Newly visually impaired developer asks if there is some way to navigate Unity Editor UI with screenreader. (No Answer) https://forum.unity.com/threads/ui-accessible-via-keyboard-and-screen-reader.334751/

March 2017, blind programmer asks if Unity editor is blind accessible. (No Answer) https://forum.unity.com/threads/screen-reader-support-of-unity-engine.462689/

• I've cut away a lot of threads with no answer

  ---

Jan 2018, Developer asks about Editor and Screenreader accessibility, says Unity is not in compliance with CVAA (No Answer) https://forum.unity.com/threads/any-updates-on-accessibility-becoming-a-first-class-citizen-in-unity.512669/

March 2018 Student accuses Unity of breaking the Americans with dissability act, and quote:

"the americans with disabilities section 508 act, which in part, says that apps, products, services must be accessible for the blind."

(Unity answers 2 years later after thread is bumped twice) https://forum.unity.com/threads/screen-reader-accessibility-for-unity.521432/?gq=screen%20reader

Sep 2018, developer asks if the new UI Toolkit will be blind accessible, since it will be XML Based. (FINALLY Unity Answers in 2018! but their answer is vague and non-promising: Quote: "

It is indeed out intention to increase overall Accessibility in both the Editor and at Runtime, and you're right to say that adding support for it in UIElements makes a lot of sense. The how and when is still not defined though.

June 2019, Hackathon Accessibility test, some Unity devs add engine-level Screen Reader support to unity as part of a Hackathon, is never published or released anywhere. https://twitter.com/superpig/status/1145381804389666976

June 2019-December 2021: (Last response from Unity as of now was May 2021): Accessibility and inclusion Thread. https://forum.unity.com/threads/accessibility-and-inclusion.694477/

In the meantime, if you want implement some sort of UI Toolkit TTS Support yourself for UI Toolkit for runtime, here is the simplest implementation I could get to work. This ONLY works on Windows (as far as I know, because I think the TTS drivers included in Tolk are Windows only).

Known limitations:

• if you don't have NVDA or something else installed, this sollution will use Windows SAPI. This WILL lag your game when navigating UI.

• If your UI often changes (if you dynamically add elements to a UI List or something) then you need to call the EnableTTSForMenus() method on the Visual Element container to grab them.

• If you update a label or button text, you need to register the event again, or else it will read the old text.

  How to Implement

1. Download the Tolk DLLs and include the drivers, drop them in the Assets/Plugins folder

2. Add the script component below to a gameobject in the scene.

3. Drag your UI DocumentComponent into it.

4. Play the game.

 using System.Collections;
 using System.Collections.Generic;
 using UnityEngine;
 using DavyKager;
 using UnityEngine.UIElements;
 namespace Stealcase.TTS
 {
 
 
     public class TTSManager : MonoBehaviour
     {
         public UIDocument document;
         public bool useSAPI = true;
         /// <summary>
         /// Whether the player wants to use TTS.
         /// </summary>
         public bool useTTS = true;
         /// <summary>
         /// The actual STATE of the TTS.
         /// </summary>
         private bool ttsEnabled = false;
         public int MaxQueueCount = 10;
         public bool TTSEnabled { get => ttsEnabled; }
         private int currentIndex = 0;
         private string lastHeader = "";
         private Queue<string> SpokenSentences;
         private Coroutine shuttingDownTTS;
 
         public static TTSManager instance;
         void Awake()
         {
             SpokenSentences = new Queue<string>(MaxQueueCount);
             if(instance != null)
             {
                 Destroy(this);
             }
             else
             {
                 instance = this;
                 if(useTTS)
                 {
                     EnableTTS();
                 }
                 DontDestroyOnLoad(this);
             }
         }
         public void ToggleEnabled()
         {
             useTTS = !useTTS;
             if(useTTS)
             {
                 EnableTTS();
             }
             else
             {
                 if(shuttingDownTTS == null)
                 {
                     Tolk.Speak("Shutting down TTS. Press T at any time to enable");
                     shuttingDownTTS = StartCoroutine(ShutDownTTS());
                 }
             }
         }
         void Start()
         {
             var root = document.rootVisualElement;
             EnableTTSForMenus(root);
         }
         IEnumerator ShutDownTTS()
         {
             yield return new WaitForSeconds(0.4f);
             while(Tolk.IsSpeaking())
             {
                 yield return new WaitForSeconds(1f);
             }
             DisableTTS();
         }
         void EnableTTS()
         {
             Debug.Log("Enabling TTS");
             if(!TTSEnabled)
             {
                 Tolk.Load();
                 Tolk.TrySAPI(useSAPI);
                 ttsEnabled = true;
                 Debug.Log("TTS ENabled");
                 Speak("Screen Reader Loaded",true);
             }
         }
         void ChangeTTS(bool enable)
         {
             if(enable)
             {
                 EnableTTS();
             }
             else
             {
                 DisableTTS();
             }
         }
         void DisableTTS()
         {
             if(TTSEnabled)
             {
                 Tolk.Unload();
                 ttsEnabled = false;
                 shuttingDownTTS = null;
                 Debug.Log("TTS SHUTDOWN");
             }
         }
         void OnDisable()
         {
             Debug.Log("Destroying Object");
             if(instance == this)
             {
                 DisableTTS();
                 instance = null;
             }
         }
         void OnEnable()
         {
             if(instance == this && useTTS)
             {
                 EnableTTS();
             }
         }
         public void Silence()
         {
             if(TTSEnabled)
             {
                 Tolk.Silence();
             }
         }
         public void Speak(string textToSpeak, bool interupt = false)
         {
             if(TTSEnabled)
             {
                 SpokenSentences.Enqueue(textToSpeak);
                 currentIndex = SpokenSentences.Count;
                 if(SpokenSentences.Count > MaxQueueCount)
                 {
                     SpokenSentences.Dequeue();
                 }
                 Tolk.Speak(textToSpeak, interupt);
             }
         }
         public void Speak(string header, string body, bool interupt = false, bool reverseOrder = false)
         {
             if(header == lastHeader)
             {
                 Speak(body, interupt);
             }
             else
             {
                 lastHeader = header;
                 if(reverseOrder)
                 {
                     string newLine_reversed = $"{body}. {header}";
                     Speak(newLine_reversed, interupt);
                 }
                 else
                 {
                     string newLine = $"{header}. {body}";
                     Speak(newLine, interupt);
                 }
             }
         }
         /// <summary>
         /// Grabs a Visual Element and queries it for elements that have USS with "tts" in them, then subscribes to navigationevents to trigger tts.  
         /// </summary>
         /// <param name="root"></param>
         public void EnableTTSForMenus(VisualElement root)
         {
             var tts = root.Q<Toggle>("text-to-speech");
             if(tts != null)
             {
                 tts.RegisterValueChangedCallback(e => ChangeTTS(e.newValue));
             }
             UQueryBuilder<Button> builder_buttons = root.Query<Button>(classes: new string[]{"unity-button"});
             builder_buttons.ForEach(RegisterButtonEvents);
 
             UQueryBuilder<Label> builder_labels = root.Query<Label>(classes: new string[]{"unity-label"});
             builder_labels.ForEach(RegisterLabelEvents);
 
             UQueryBuilder<SliderInt> builder_sliders = root.Query<SliderInt>(classes: new string[]{"unity-slider-int"});
             builder_sliders.ForEach(RegisterSliderEvents);
             builder_sliders.ForEach(v => v.RegisterCallback<NavigationMoveEvent, SliderTTS>(IncrementSlider,v)); //Allowing Slider to navigate with WASD
 
             UQueryBuilder<Toggle> builder_toggles = root.Query<Toggle>(classes: new string[]{"unity-toggle"});
             builder_toggles.ForEach(RegisterToggleEvents);
 
             UQueryBuilder<ImageTTS> builder_images = root.Query<ImageTTS>(classes: new string[]{"tts-image"}); //Custom class with Alt Text
             builder_images.ForEach(RegisterImageEvents);
         }
         /// <summary>
         /// Unsubscribe from all children elements.  
         /// </summary>
         /// <param name="root"></param>
         public void DisableTTSForMenus(VisualElement root)
         {
             UQueryBuilder<Button> builder_buttons = root.Query<Button>(classes: new string[]{"unity-button"});
             builder_buttons.ForEach(UnRegisterButtonEvents);
 
             UQueryBuilder<Label> builder_labels = root.Query<Label>(classes: new string[]{"unity-label"});
             builder_labels.ForEach(UnregisterLabelEvents);
 
             UQueryBuilder<SliderInt> builder_sliders = root.Query<SliderInt>(classes: new string[]{"unity-slider-int"});
             builder_sliders.ForEach(UnRegisterSliderEvents);
             builder_sliders.ForEach(v => v.UnregisterCallback<NavigationMoveEvent, SliderTTS>(IncrementSlider,v));
 
             UQueryBuilder<Toggle> builder_toggles = root.Query<Toggle>(classes: new string[]{"unity-toggle"});
             builder_toggles.ForEach(UnRegisterToggleEvents);
 
             UQueryBuilder<ImageTTS> builder_images = root.Query<ImageTTS>(classes: new string[]{"tts-image"});
             builder_images.ForEach(UnRegisterImageEvents);
         }
         #region subscribe events
         void RegisterButtonEvents(Button btn) => btn.RegisterCallback<FocusInEvent, string>(NavigateButton, btn.text);
         void UnRegisterButtonEvents(Button btn) => btn.UnregisterCallback<FocusInEvent, string>(NavigateButton);
         void RegisterLabelEvents(Label label)
         {
             label.focusable = true;
             label.RegisterCallback<FocusInEvent, string>(NavigateLabel, label.text);
         }
         void UnregisterLabelEvents(Label label)
         {
             label.focusable = false;
             label.UnregisterCallback<FocusInEvent, string>(NavigateLabel);
         } 
             
         void RegisterImageEvents(ImageTTS image)
         {
             image.focusable = true;
             image.RegisterCallback<FocusInEvent, string>(NavigateImage, image.Description);
         }
         void UnRegisterImageEvents(ImageTTS image)
         {
             image.focusable = true;
             image.UnregisterCallback<FocusInEvent, string>(NavigateImage);
         }
         void RegisterSliderEvents(SliderInt slider) => slider.RegisterCallback<FocusInEvent, SliderInt>(NavigateSlider, slider);
         void UnRegisterSliderEvents(SliderInt slider) => slider.UnregisterCallback<FocusInEvent, SliderInt>(NavigateSlider);
         void RegisterToggleEvents(Toggle toggle) => toggle.RegisterCallback<FocusInEvent, string>(NavigateToggle, toggle.label);
         void UnRegisterToggleEvents(Toggle toggle) => toggle.UnregisterCallback<FocusInEvent, string>(NavigateToggle);
         #endregion
         # region EventWrappers
         void NavigateButton(FocusInEvent e, string body) => Speak(TTSElementType.Button.ToString(), body, true, true);
         void NavigateLabel(FocusInEvent e, string body) => Speak(TTSElementType.Label.ToString(), body, true, true);
         void NavigateImage(FocusInEvent e, string body) => Speak(TTSElementType.Image.ToString(), body, true, true);
         void NavigateSlider(FocusInEvent e, SliderInt slider) => Speak(TTSElementType.Slider.ToString(), $"{slider.label}. {slider.value} percent. {slider.tooltip}", true, true);
         void NavigateToggle(FocusInEvent e, string body) => Speak(TTSElementType.Toggle.ToString(), body, true, true);
         #endregion
 
         /// <summary>
         /// Allowing Left and right UI navigation events to increment value. 
         // only Directional Arrows are supported automatically, this method allows WASD to be supported.
         /// </summary>
         /// <param name="root"></param>
 
         void IncrementSlider(NavigationMoveEvent m, SliderInt slider) {
             switch (m.direction)
             {
                     break;
                 case NavigationMoveEvent.Direction.Left:
                     slider.value -= 1;
                     break;
                 case NavigationMoveEvent.Direction.Right:
                     slider.value += 1;
                     break;
                 default:
                     break;
             }
         }
     }
     public enum TTSElementType
     {
         Button,
         Label,
         Slider,
         Input,
         Toggle,
         Image,
     }
     /// <summary>
     /// Image class that supports Alt Text for Images in unity. 
     /// Be sure to use this class when adding images to unity, or creating your own custom visual Elements. 
     /// </summary>
     public class ImageTTS : Image
     {
         public new class UxmlFactory : UxmlFactory<ImageTTS, UxmlTraits> { }
         public new class UxmlTraits : Image.UxmlTraits 
         {
                 UxmlStringAttributeDescription m_description = new UxmlStringAttributeDescription { name = "description", defaultValue = "Needs description!" };
                 public override IEnumerable<UxmlChildElementDescription> uxmlChildElementsDescription
                 {
                     get { yield break; }
                 }
 
                 public override void Init(VisualElement ve, IUxmlAttributes bag, CreationContext cc)
                 {
                     base.Init(ve, bag, cc);
                     var _l = ve as ImageTTS;
                     _l.Clear();
                     _l.Description = m_description.GetValueFromBag(bag, cc);
                 }
         }
         public string Description { get; set; }
         public ImageTTS() : base()
         {
             AddToClassList("tts-image");
         }
         public ImageTTS(string description) : this()
         {
             Description = description;
         }
 
     }
 }