Tuesday, July 11, 2006

Creating a UI Automation Provider for a .Net Control

In this post, I will give an overview of the process of creating a client side UI Automation Provider for the MenuStrip Control. This will focos on how to get access to the data, not how to do each detailed step.

  1. Create the Provider Assembly
  2. Get a MenuStrip reference from the Windows Handle of the Control
  3. Implement the MenuStrip Provider using the MenuStrip object
  4. Register the Provider


Create the Provider Assembly
  1. Create a project with any name
  2. Add a static class called UIAutomationClientSideProviders
  3. Add a static field called ClientSideProviderDescriptionTable which is a ClientSideProviderDescription[]
  4. For each provider add a ClientSideProviderDescription to the array with a delegate used to create the provider and the UI "class name" of the control. The UI "class name" can be found with the UISpy tool or Spy++.
The code for a MenuStrip is below:

using System.Windows.Automation;
using System.Windows.Automation.Provider;

// The namespace must be the same as the name of the DLL,
// so that UI Automation can find the table of descriptors.
// In this example, the DLL would be "Auxsr.CustomClientSideProviders.dll"
namespace Auxsr.CustomClientSideProviders
{
// The assembly must implement a UIAutomationClientSideProviders class
public static class UIAutomationClientSideProviders
{
// Implementation of the static ClientSideProviderDescriptionTable field.
// In this case, only a single provider is listed in the table.
public static ClientSideProviderDescription[] ClientSideProviderDescriptionTable =
{
new ClientSideProviderDescription(
new ClientSideProviderFactoryCallback(MenuStripProvider.Create),
"WindowsForms10.Window.8.app.0.378734a")
};
}

}



Get a Control reference from the Windows Handle of the Control
  1. The ClientSideProviderFactoryCallback method takes a windows handle as the first parameter.
  2. Call the Control.FromHandle method to get a reference to the MenuStrip.
  3. Try to cast the Control to a MenuStrip reference
  4. If it succeded, you now have acces to the MenuStrip object

internal static IRawElementProviderSimple Create(
IntPtr hwnd, int idChild, int idObject )
{
// Get a reference to the managed control from the windows handle
Control controlFromHwnd = Control.FromHandle( hwnd );

// If the control is a MenuStrip
if( controlFromHwnd is MenuStrip )
{
// Create a menu strip provider
return new MenuStripProvider( controlFromHwnd as MenuStrip );
}
// Otherwise, return the default provider
else
{
return AutomationInteropProvider.HostProviderFromHandle( hwnd );
}
}


Implement the MenuStrip Provider

Implement the following interfaces using the MenuStrip object:
IRawElementProviderSimple
IRawElementProviderFragment
IRawElementProviderFragmentRoot

See examples in the Windows SDK by searching for each of the previous interfaces.


Register the Provider


In the client, register the provider assembly at runtime.

ClientSettings.RegisterClientSideProviderAssembly(
typeof(
Auxsr.CustomClientSideProviders.UIAutomationClientSideProviders
).Assembly.GetName() );


As an alternative, a client could automatically load all the providers assemblies from a certain directory on the hard drive. This could also be made secure by using an AppDomain with limited permissions. I have not tested this, but I believe it would be the optimal solution for accessibility technology in order to extend the client with additional client side providers without actually modifying the client.

On Architecture

Overview
This post will provider an overview of the architecture for a UI Automation Client (like a screen reader). There are four layers involved when using the UIA. Here is an example of the layers with a screen reader that is focused on a Drop Down Box in Firefox:
  • The ScreenReader
  • The UI Automation Framework
  • Firefox Html Drop Down Box Provider
  • Firefox with a page open that is currently focused on a drop down box
The four layers are:
  • The UIA Client Application
  • The UI Automation Framework
  • The UI Automation Providers
  • The "Actual" Application

The UIA Client Application
The Client Application is any application that uses the UIA. Anotherwords, any application that might control or read the user interface.
Some examples include:
  • Screen Reader
  • Screen Magnifier
  • UI Testing for Software Development
  • System Macros
  • User Monitoring Software

The UI Automation Framework
The framework is the access point for the Client Applications. It allows the client to access the user interface in a consistent way. It reduces the complexity of the user interface and presents everything in well defined terms. For example, a screen reader can access all buttons the same way no matter if it is an Html Button, a Java Button, an MFC button, or a .Net Button.

The UI Automation Providers
The UI Automation Providers are responsible for interpreting every specific type of user interface control. It accesses the data of a specific control and presents it to the UIA as an Automation Element. For example, a Java Button would have a Java Button Provider which would interprete the button so the UIA can work with it as a standard button.
Most controls already have a UIA Provider. If the control is a standard Win32 control or supports MSAA, then it already has some level of functionality. However, custom controls that have additional information need to have their own providers so that UIA can get to that data or interact with the control.


The "Actual" Application
The Application that is being driven by the UIA. There is no limit to what types of applications the UIA can work with. All that is required is that the application's controls have sufficient UI Automation Providers in order to make the application accessible.

UI Automation Viewer Progress

The past few weeks have reaveled much about the UI Automation framework and how powerful it has proved.

The UI Automation Viewer tool we have been creating has made great progess. It allows the user to choose what type of element view they desire. Right now it can present three different views:
  • Outline View - Creates an easy to see outline of the screen or an application
  • Reconstruction View - The main goal of the UIA Viewer which reconstructs the UI in order to present what is actually visible with the UIA
  • Auditory Layout View - An experiment that uses stereo audio, pitch changing, and volume to help a blind user visuallize the layout of the screen

Currently, the Reconstruction View can reconstruct the following types of controls:
  • Buttons
  • Hyperlinks
  • ComboBoxes
  • ListBoxes/ListViews
  • TreeViews
  • ScrollBars
  • Sliders
  • Menus (In Progress)
  • Progress Bars (In Progress -Ironic)

Everything else is reconstructed as a text box. This accounts for about 60% of the different types of controls. Some of them only allow the information to be read, while others allow interaction.

For example, a reconstructed button can actually be clicked which causes the real button to be clicked. However, changing the selection in a list does not actually do anything to the original list.

Last week, I figured out how easy it is to overcome the lack of a provider for certain controls:
The MenuStrip control of .Net 2.0 does not have any UI Automation Provider. Therefore, the UIA cannot access anything but its name. However, I was able to overcome this limitation by providing a client side provider - more on that in a later post. The point is, in a short time, I was able to make my own provider to make an application accessible WITHOUT changing the program.

Let me say that again:
I made an application accessible without changing anything about the application itself.

This is pretty revolutionary because it would be easy to create new UI Automation providers that are external to both the application and the accessibility technology. Neither the application nor the screen reader would need any modifications to make the application more accessible.

Awesome!

Sunday, July 09, 2006

Tactile Monitor - The Ultimate Accessibility Tool

Digression

I just wanted to mention a dream I have had - something that I think would be a cool tool - a tactile monitor.

What I mean by a tactile monitor is a grid of pins that can change their height to form a tactile image. A blind person could quickly feel the overall pattern on the monitor. It would not need to communicate letters or words, just shapes. This could be done with a grid of 100 * 80 and would greatly open a blind users world on the computer.
With current technology a blind user cannot do anything with graphics or any visual information like the layout of the screen. With a tactile monitor a blind person could:
- Identify the objects in an image
- With zooming possibly recognize people in photographs
- Scan images and identify them
- Feel a programs User Interface (UI) layout
- Have a backup technique when the screen reader will not read an application correctly

I believe if this were mass produced, it could cost under $1,000 per unit.

I am submitting this idea to Cambrian House. Who knows, maybe some electrical engineers might think this is possible and feasible.

End Digression