Agapeware: Logic Recognition

Agapeware: Logic Recognition

Check out this article in my Coding blog about an alternative to Image Recognition Systems.

Creating a UI Automation Provider for a .Net Control

In this post, I will give an overview of the process of creating a client side UI Automation Provider for the MenuStrip Control. This will focos on how to get access to the data, not how to do each detailed step.

1. Create the Provider Assembly
   
2. Get a MenuStrip reference from the Windows Handle of the Control
3. Implement the MenuStrip Provider using the MenuStrip object
   
4. Register the Provider
   

Create the Provider Assembly

1. Create a project with any name
2. Add a static class called UIAutomationClientSideProviders
   
3. Add a static field called ClientSideProviderDescriptionTable which is a ClientSideProviderDescription[]
4. For each provider add a ClientSideProviderDescription to the array with a delegate used to create the provider and the UI "class name" of the control. The UI "class name" can be found with the UISpy tool or Spy++.

The code for a MenuStrip is below:

using System.Windows.Automation;
using System.Windows.Automation.Provider;

// The namespace must be the same as the name of the DLL,
// so that UI Automation can find the table of descriptors.
// In this example, the DLL would be "Auxsr.CustomClientSideProviders.dll"
namespace Auxsr.CustomClientSideProviders
{
  // The assembly must implement a UIAutomationClientSideProviders class
  public static class UIAutomationClientSideProviders
  {
    // Implementation of the static ClientSideProviderDescriptionTable field.
    // In this case, only a single provider is listed in the table.
    public static ClientSideProviderDescription[] ClientSideProviderDescriptionTable =
    {
      new ClientSideProviderDescription(
        new ClientSideProviderFactoryCallback(MenuStripProvider.Create),
        "WindowsForms10.Window.8.app.0.378734a")
    }; 
  }

}

Get a Control reference from the Windows Handle of the Control

1. The ClientSideProviderFactoryCallback method takes a windows handle as the first parameter.
2. Call the Control.FromHandle method to get a reference to the MenuStrip.
3. Try to cast the Control to a MenuStrip reference
4. If it succeded, you now have acces to the MenuStrip object
   

internal static IRawElementProviderSimple Create(
  IntPtr hwnd, int idChild, int idObject )
{
  // Get a reference to the managed control from the windows handle
  Control controlFromHwnd = Control.FromHandle( hwnd );

  // If the control is a MenuStrip
  if( controlFromHwnd is MenuStrip )
  {
    // Create a menu strip provider
    return new MenuStripProvider( controlFromHwnd as MenuStrip );
  }
    // Otherwise, return the default provider
  else
  {
    return AutomationInteropProvider.HostProviderFromHandle( hwnd );
  }
}

Implement the MenuStrip Provider

Implement the following interfaces using the MenuStrip object:
IRawElementProviderSimple
IRawElementProviderFragment
IRawElementProviderFragmentRoot

See examples in the Windows SDK by searching for each of the previous interfaces.


Register the Provider

In the client, register the provider assembly at runtime.

ClientSettings.RegisterClientSideProviderAssembly(
  typeof( 
    Auxsr.CustomClientSideProviders.UIAutomationClientSideProviders
  ).Assembly.GetName() );

As an alternative, a client could automatically load all the providers assemblies from a certain directory on the hard drive. This could also be made secure by using an AppDomain with limited permissions. I have not tested this, but I believe it would be the optimal solution for accessibility technology in order to extend the client with additional client side providers without actually modifying the client.

On Architecture

Overview
This post will provider an overview of the architecture for a UI Automation Client (like a screen reader). There are four layers involved when using the UIA. Here is an example of the layers with a screen reader that is focused on a Drop Down Box in Firefox:

• The ScreenReader
• The UI Automation Framework
• Firefox Html Drop Down Box Provider
• Firefox with a page open that is currently focused on a drop down box

The four layers are:

• The UIA Client Application
• The UI Automation Framework
• The UI Automation Providers
• The "Actual" Application

The UIA Client Application
The Client Application is any application that uses the UIA. Anotherwords, any application that might control or read the user interface.
Some examples include:

• Screen Reader
• Screen Magnifier
• UI Testing for Software Development
  
• System Macros
• User Monitoring Software

The UI Automation Framework
The framework is the access point for the Client Applications. It allows the client to access the user interface in a consistent way. It reduces the complexity of the user interface and presents everything in well defined terms. For example, a screen reader can access all buttons the same way no matter if it is an Html Button, a Java Button, an MFC button, or a .Net Button.

The UI Automation Providers
The UI Automation Providers are responsible for interpreting every specific type of user interface control. It accesses the data of a specific control and presents it to the UIA as an Automation Element. For example, a Java Button would have a Java Button Provider which would interprete the button so the UIA can work with it as a standard button.
Most controls already have a UIA Provider. If the control is a standard Win32 control or supports MSAA, then it already has some level of functionality. However, custom controls that have additional information need to have their own providers so that UIA can get to that data or interact with the control.


The "Actual" Application
The Application that is being driven by the UIA. There is no limit to what types of applications the UIA can work with. All that is required is that the application's controls have sufficient UI Automation Providers in order to make the application accessible.

UI Automation Viewer Progress

The past few weeks have reaveled much about the UI Automation framework and how powerful it has proved.

The UI Automation Viewer tool we have been creating has made great progess. It allows the user to choose what type of element view they desire. Right now it can present three different views:

• Outline View - Creates an easy to see outline of the screen or an application
• Reconstruction View - The main goal of the UIA Viewer which reconstructs the UI in order to present what is actually visible with the UIA
• Auditory Layout View - An experiment that uses stereo audio, pitch changing, and volume to help a blind user visuallize the layout of the screen

Currently, the Reconstruction View can reconstruct the following types of controls:

• Buttons
• Hyperlinks
• ComboBoxes
• ListBoxes/ListViews
• TreeViews
• ScrollBars
• Sliders
• Menus (In Progress)
  
• Progress Bars (In Progress -Ironic)
  

Everything else is reconstructed as a text box. This accounts for about 60% of the different types of controls. Some of them only allow the information to be read, while others allow interaction.

For example, a reconstructed button can actually be clicked which causes the real button to be clicked. However, changing the selection in a list does not actually do anything to the original list.

Last week, I figured out how easy it is to overcome the lack of a provider for certain controls:
The MenuStrip control of .Net 2.0 does not have any UI Automation Provider. Therefore, the UIA cannot access anything but its name. However, I was able to overcome this limitation by providing a client side provider - more on that in a later post. The point is, in a short time, I was able to make my own provider to make an application accessible WITHOUT changing the program.

Let me say that again:
I made an application accessible without changing anything about the application itself.

This is pretty revolutionary because it would be easy to create new UI Automation providers that are external to both the application and the accessibility technology. Neither the application nor the screen reader would need any modifications to make the application more accessible.

Awesome!

Tactile Monitor - The Ultimate Accessibility Tool

Digression

I just wanted to mention a dream I have had - something that I think would be a cool tool - a tactile monitor.

What I mean by a tactile monitor is a grid of pins that can change their height to form a tactile image. A blind person could quickly feel the overall pattern on the monitor. It would not need to communicate letters or words, just shapes. This could be done with a grid of 100 * 80 and would greatly open a blind users world on the computer.
With current technology a blind user cannot do anything with graphics or any visual information like the layout of the screen. With a tactile monitor a blind person could:
- Identify the objects in an image
- With zooming possibly recognize people in photographs
- Scan images and identify them
- Feel a programs User Interface (UI) layout
- Have a backup technique when the screen reader will not read an application correctly

I believe if this were mass produced, it could cost under $1,000 per unit.

I am submitting this idea to Cambrian House. Who knows, maybe some electrical engineers might think this is possible and feasible.

End Digression

Screen Outline Demo

Yesterday I mentioned using the UIA Viewer to make an outline of the entire screen so that a low-vision user could easily find items on the screen. Well because the architecture was already in place it only took a few hours to implement a prototype.

Below are a few screen shots of the generated outline view. With the UIA Viewer, I could select any application or part of an application and generate an outline view. The outlines have a black background with white rectangles representing items on the screen. The purpose of the outline view is to display an overview of the layout of the screen, so details are not displayed.


This is an outline of Windows Explorer. I had a random folder open. At the top and left are various toolbars with buttons and links. At the bottom right is the list of files. Individual files are not displayed in the outline. However, they could have been if focus were placed on the file list instead of the entire window when the outline was generated.



This is an outline of Microsoft Outlook. At the top are the toolbars with buttons, search boxes, and drop down lists. Under the toolbars are the three different panes of outlook: The Folder List, the Folder Items, and the Opened Email.



This is an outline of Microsoft Visual Studio 2005 which was used to develop this application. In the outline, the running program has just thrown an exception. The reactangle in the lower right is a dialog window that contains the information about the exception. At the top are the toolbars and at the right and bottom are expandable panes that shrink when they do not have focus. This outline shows that the shrinking panes are actually positioned just on the edge of the application. In this case, the outline viewer is actually displaying elements that are not visible on the screen. This is one situation where additional work is needed.



This is an outline of the Start Menu. It has four sections: The Start button, the quick launch icons, the running tasks, and the system tray with its icons and clock.




This is an outline of my desktop. Each rectangle is an shortcut or other item located on my desktop. Obviously, it is extremely cluttered. I intend to organize everything soon.


Hopefully, these screenshots will give you an idea of how this could be used to help a person find items on the screen.

The next step is to figure out how to use this tool in conjuction with a screen magnifier. My idea is as follows:

1. The user activates the outline view with a simple but unique shortcut (like holding down the middle mouse button or holding down the windows key)


2. Any running screen magnifier is temporarily disabled


3. The outline is displayed on the screen


4. The user moves their mouse (which is represented by a large yellow X) to the desired part of the screen


5. The user deactives the outline view


6. The screen magnifier is enabled

The difficulty is turning a running screen magnifier on and off.

Please comment if you have any suggestions.

Screen Magnifiers

Working on the UIA Viewer Tool has brought some intersting ideas concerning screen magnification. The technique we are using to reconstruct the user interface could also be used to reconstruct a user interface for screen magnification. This would enable a tool for overcoming one of the most difficult problems with screen magnifiers.

With a screen magnifier, the user is focused on one part of the screen at a time. This makes it very difficult to find or even notice some elements on the screen. Pop up balloons, dialog boxes, status bars, and other visual indicators are often missed by a screen magnifier. In addition, when the user knows they are present, it is often difficult to locate these elements in order to read them.

A sighted person does not have this problem because he or she can see the whole screen at once. So a solution is to allow a user of a screen magnifier to do the same.

The important part of this solution is to remove all the details and give an outline view of the major elements on the screen. Elements that have changed or have focus could have a stronger outline or some other way to emphasize them. With this tool, a user could quickly switch to this view, visually move the mouse to the desired part of the screen, and switch back to zoom view.

With the ability to see the whole screen in high contrast without unneccessary details, the user could greatly enhance thier user experience.

This solution could also be used for those with tunnel vision. For example, if a user could only see about 3 by 5 inches of the screen. This screen preview could be sized to a small window that he or she could see all at once. It would serve the same purpose, and it would allow the user to find the relevant parts of the screen and know where to look on the monitor.


This technique is actually relatively easy to implement. I expect it will be one of the earliest practical uses of the UIA Viewer. The main implementation difficulty will be allowing it to work in conjuction with a screen magnifier.