Screen Outline Demo

Yesterday I mentioned using the UIA Viewer to make an outline of the entire screen so that a low-vision user could easily find items on the screen. Well because the architecture was already in place it only took a few hours to implement a prototype.

Below are a few screen shots of the generated outline view. With the UIA Viewer, I could select any application or part of an application and generate an outline view. The outlines have a black background with white rectangles representing items on the screen. The purpose of the outline view is to display an overview of the layout of the screen, so details are not displayed.


This is an outline of Windows Explorer. I had a random folder open. At the top and left are various toolbars with buttons and links. At the bottom right is the list of files. Individual files are not displayed in the outline. However, they could have been if focus were placed on the file list instead of the entire window when the outline was generated.



This is an outline of Microsoft Outlook. At the top are the toolbars with buttons, search boxes, and drop down lists. Under the toolbars are the three different panes of outlook: The Folder List, the Folder Items, and the Opened Email.



This is an outline of Microsoft Visual Studio 2005 which was used to develop this application. In the outline, the running program has just thrown an exception. The reactangle in the lower right is a dialog window that contains the information about the exception. At the top are the toolbars and at the right and bottom are expandable panes that shrink when they do not have focus. This outline shows that the shrinking panes are actually positioned just on the edge of the application. In this case, the outline viewer is actually displaying elements that are not visible on the screen. This is one situation where additional work is needed.



This is an outline of the Start Menu. It has four sections: The Start button, the quick launch icons, the running tasks, and the system tray with its icons and clock.




This is an outline of my desktop. Each rectangle is an shortcut or other item located on my desktop. Obviously, it is extremely cluttered. I intend to organize everything soon.


Hopefully, these screenshots will give you an idea of how this could be used to help a person find items on the screen.

The next step is to figure out how to use this tool in conjuction with a screen magnifier. My idea is as follows:

1. The user activates the outline view with a simple but unique shortcut (like holding down the middle mouse button or holding down the windows key)


2. Any running screen magnifier is temporarily disabled


3. The outline is displayed on the screen


4. The user moves their mouse (which is represented by a large yellow X) to the desired part of the screen


5. The user deactives the outline view


6. The screen magnifier is enabled

The difficulty is turning a running screen magnifier on and off.

Please comment if you have any suggestions.

Screen Magnifiers

Working on the UIA Viewer Tool has brought some intersting ideas concerning screen magnification. The technique we are using to reconstruct the user interface could also be used to reconstruct a user interface for screen magnification. This would enable a tool for overcoming one of the most difficult problems with screen magnifiers.

With a screen magnifier, the user is focused on one part of the screen at a time. This makes it very difficult to find or even notice some elements on the screen. Pop up balloons, dialog boxes, status bars, and other visual indicators are often missed by a screen magnifier. In addition, when the user knows they are present, it is often difficult to locate these elements in order to read them.

A sighted person does not have this problem because he or she can see the whole screen at once. So a solution is to allow a user of a screen magnifier to do the same.

The important part of this solution is to remove all the details and give an outline view of the major elements on the screen. Elements that have changed or have focus could have a stronger outline or some other way to emphasize them. With this tool, a user could quickly switch to this view, visually move the mouse to the desired part of the screen, and switch back to zoom view.

With the ability to see the whole screen in high contrast without unneccessary details, the user could greatly enhance thier user experience.

This solution could also be used for those with tunnel vision. For example, if a user could only see about 3 by 5 inches of the screen. This screen preview could be sized to a small window that he or she could see all at once. It would serve the same purpose, and it would allow the user to find the relevant parts of the screen and know where to look on the monitor.


This technique is actually relatively easy to implement. I expect it will be one of the earliest practical uses of the UIA Viewer. The main implementation difficulty will be allowing it to work in conjuction with a screen magnifier.

UIA Viewer Tool

The UIA Viewer, is a tool to help show what a screen reader can see. UIA (User Interface Automation) is the latest accessibility technology that Microsoft has created as part of Windows Vista (the next version of Windows). Thankfully, Microsoft has also made the technology backwards compatible for Windows XP and Windows Server 2003. So the UIA Viewer tool uses this technology to show what a screen reader can see. In order to accomplish that task, the UIA Viewer uses the information to reconstruct the user interface.

For example: The current window I am using to write this blog entry has a drop down list to choose the blog, a label and textbox to enter the title, a toolbar to choose formating options, a large text box to type in the text of the blog and 2 buttons to either save this as a draft or publish the post. All of this is being displayed to the screen in Firefox using various fonts and graphics. If I were to open this window in Internet Explorer, it might look a little different. And if this were a windows application instead of an html form, it would look very different.

The UIA Viewer takes all the information about this window and recreates it in a standard way that simulates how a screen reader would read the screen. Instead of displaying the graphics, it displays the tooltip for each, like Bold, Italics, Colors, etc. as a text button.

Right now, we are in the process of implementing this tool. At this point every user interface element is reconstructed as a text box. However, as we improve it, each element will more closely resemble its original.

Although, the tool is not self-voicing, it is accessible for a screeen reader to use. In fact there are three of us working together and we have an intersting diversity. One is sighted, one uses Jaws a screen reader, and one uses ZoomText a screen magnifier. A great mix for both testing and brainstorming.

This tool will be open source, but right now it is more in a prototype stage. Come back or sign up to the RSS feed to hear about the progress of the UIA Viewer.

Why are Applications Still Not Accessible?

Many technologies exist today that enable developers to make an application accessible. With some frameworks like Microsoft .Net all a developer must do is add an "Accessible Name" to each item on the screen that is not automatically labeled. However, even in .Net, many applications are still developed that have unlabeled text boxes and unorganized tab navigation.

Therefore, some extra steps need to be done to get past this problem. First, developers need to "see" what a screen reader sees. If a sighted developer could easily see where a screen reader is missing something in the User Interface, they could easily fix the problem during development instead of during testing or after the product ships. Second, screen readers must take more responsibility in enabling applications that work for sighted users. For example, a screen reader could detect a nearby label and associate that with the textbox as the accessible name if the accessible name was left blank by the developer.

We will begin developing a small tool that reconstructs a form to show what the screen reader sees through the User Interface Automation (UIA). The purpose of this will be for testing the UIA and for easily allowing a sighted developer to understand what the screen reader can actually see.

Next Generation Blind User Experience

How would the blind user experience be different?

I have been thinking about this question over the past year. I teach C# at a blind school. I myself am sighted, but I must use a screen reader in order to teach my students. Therefore, I am in a unique position because I know how simple the user interface is for a sighted "mouse-wielding" user, and how frustrating it can be for a blind user who relies on keyboard navigation and the screen reader.

There is a huge gap, but I believe some awesome experiences can happen with a screen reader. Below are a few ideas which I have. Many of the ideas are inspired by similar technologies that already exists.

• Stability:
  
• The screen reader "actually works" without locking up, crashing, or interfering with other applications
  
• If the screen reader ever stops responding, a small backup screen reader automatically starts that enables the blind user to fix the problem.
  
• Security:
  
• The screen reader does not introduce any security holes.
  
• For example, a blind user's passwords remain safe.
  
• Friendly Navigation:
  
• An application can be customized so that it navigates in a standard way.
  
• No applications require the mouse cursor to work.
  
• Friendly Speaking:
  
• The user can make the screen reader repeat anything it has said.
  
• The user can repeat a message that was interrupted by another message.
  
• Friendly Shortcuts:
  
• Uses multiple press shortcuts instead of shortcuts that require extreme/insane key combinations.
  
• Provides a shortcut auto-complete list that helps the user find their shortcut and teaches the user additional shortcuts.
  
• Provides keyboard layout help for beginners who do not know where all the keys are located on the keyboard.
  
• Sleep Mode:
  
• The user can turn off the screen reader for temporary purposes.
  
• For example, the screen reader can sleep while a sighted user is sitting at the computer.
  
• No Useless Features
  
• Every feature makes the computer more usable for a blind user.
  
• No features exist simply because they are cool or fun.
  
• Open Source
  
• The screen reader is free.
  
• A thriving open source community makes the screen reader truly useful and allows it to grow.
  
• No corporatations taking advantage of the system.
  

I believe all of these goals are possible. In addition, I believe even better ideas will emerge and make a great open source screen reader that enables blind users beyond what is possible today.

Next Generation Screen Reader

About a year ago, many people got excited about a project for an open source screen reader for Windows Vista. I was one of them. However, the project seems to have disappeared. I hope to revive that flame and help get the ball rolling again.

A side note: The open source screen reader could also run on Windows XP and Windows Server 2003.

I have created a few technology demos to test the accessibility technology that is built into Vista. The technology is called User Interface Automation (UIA) and it is awesome. It replaces MSAA which is the current accessibility technology. The UIA provides a standard interface for all applications.

Currently, a screen reader must work differently with a program depending on what programming framework it uses. If the program was written in Java it will work differently than if it were written in C++. Also, there are sometimes multiple user interface frameworks used by the same language. This makes the job of a screen reader very difficult and because of that, some programs are simply not accessible or require additional work to make them accessible.

The UIA is a solution to the problem. Instead of a screen reader trying to get its accessible information directly from each application, it can get the information from the UIA. Therefore, the screen reader can treat every application the same.


What Does This Mean for a Blind User?

For a person who uses a screen reader this means that the screen reader will work with every program in a consistent manner. For example: A button or link on a web page will work the same as a button on a program written in Java, a C++ button, a C# button, etc. A button is simply something that the user can click and the UIA makes all buttons work the same way.

Finally, blind users will be able to have a consistent feel to the user interface. They won't have to learn how to use the screen reader with every new application they try. A sighted user can figure out how to use a new application because it looks the same as what they have already learned. But often even when applications look the same, the screen reader does not work with it the same way. Therefore, the UIA will help make the user experience enjoyable.

I believe the time has come for a new breed of screen readers. Why should a blind user require training in order to use a screen reader even when they are already familiar with computers? Why should a screen reader require the user to learn 100+ shortcuts in order to function? The time has come for a screen reader that makes the user experience enjoyable. I believe we can create it and that is what we will do.

Welcome to the Next Generation Accessibility Blog

Welcome to the blog for the Next Generation Accessibility Wiki found at: http://nextgenerationaccessibility.wikispaces.com/