Apple's human interface research - a new user interface?
Written by David Tebbutt, MacUser 11/91 item 05 - scanned
In our last look at the research being carried out by Apple's Human Interface Group, we enter the realm of high-end computing and the search for a new system interface.
This series so far has looked at a number of difficult areas of research carried out by the Human Interface Group (HIG) within Apple's Advanced Technology Group in Cupertino. In this final article, we take a look at the work that is being done beyond the Mac - projects on high-end computing and on accessing remote information databases more effectively.
Not all of the new technologies being developed within Apple's Advanced Technology Group can be cleanly integrated into the existing user interface. Accordingly, one design effort is centred on the creation of a brand new system interface incorporating technologies which have not yet seen the light of day.
Current developments include prototypes of ideas such as component tools, delegation techniques, support for temporal media such as audio and video, tools for annotation and 3D manipulation, new ways of retrieving, browsing and displaying information, and support for connectivity and collaboration.
Some of the ideas in the high-end computer interface, such as annotation, layering and movable icons for marking up documents, have been described earlier in this series. These ideas are being simultaneously developed by several HIG teams and have been adapted and extended for use in a high-end interface.
The high-end project was launched in May 1990 by brainstorming with members of HIG for three weeks to examine key user issues and requirements for a new interface. A year of design, prototyping, and coding followed.
As always with HIG, the new system interface is being driven by user needs. Gitta Salomon and her team spend a lot of time interviewing users to understand their problems, both on their computers and in other parts of their lives. Prototypes of new ideas are built so that they can be tried out on users at the earliest opportunity. Although the project is designated `high-end computing', one of the design aims is ultimately to make a consistent interface which can scale applications across a variety of platforms from the smallest machines upwards. Clearly, each successively smaller machine will implement fewer capabilities or fewer aspects of a given function, but operation across the range of machines must be straightforward because of the need to support common features in the interface.
The Basic Screen
A typical opening screen might have a tool drawer at the top left, a stationery drawer below it, and a mail area on the top right hand side.
The tool drawer contains tools for drawing, typing, painting and selection. There are also tools for sound, video, scanning, classifying documents and collaboration. Users can personalise their machines with additional tools they've acquired or adapted. Plain paper documents, which are obtained from the stationery drawer, can be operated on by any tool, be it video, paintbrush or a marking icon.
The team has been exploring how people typically use tools and is applying this information to the design. Salomon points out that artists and cooks have tools appropriate to particular functions. They don't set brush colour and thickness, or knife sharpness and blade length. She suggests a parallel in computing where the specific tools are available all the time. And rather than a single brush that you have to modify, why not have a computer equivalent of a bunch of them in a mug? The mail area takes care of any form of electronic communication between users, including electronic mail, voicemail and videophone.
A Document Page
In the high-end interface, documents assume a number of new dimensions. Documents are layered, so that, for example, users can annotate them without changing the underlying text by sliding a translucent layer above the document and applying the annotations to the new layer.
The layer controls include a number of tabs at the edge of the document, with each tab representing a layer. The layering sequence can be altered by dragging these tabs to new positions - if you drag them below a `divider' tab, the layer disappears from view. New layers are dragged out of a `well' by pulling the new layer tab above the divider.
The document of the high-end machine and its annotation layers will be able to incorporate any data type. For example, sound, video, and text can all be included in the same document. Mark-up icons can be applied as they were in the hand-held machine mentioned in the first article in this series.
As the number of documents increases, users will need to quickly recognise those of interest. Work is being done to allow the user to expand the document icon into an ever-increasing view area. A click on the icon throws up four controllers which govern the degree of `zoom'. While the zoom view is small, certain aspects of the document are exaggerated. For example, while text may be `greeked' and shown as straight-sided blocks, annotations and pictures will show up more clearly. A text-only document might show the line layout more clearly. Like the video `poster', perhaps, the icon will show a unique page or play a sound when activated. These views will vary according to whether or not the user is familiar with a document.
File Organisation
As systems get bigger, the number of files and the potential confusion for the user increases. Most users find it difficult to organise their files and folders beyond a certain point, so a simple way of marking, storing and retrieving files and folders is needed.
The team is developing a search system which asks users to provide clues about the files and folders they're looking for. `The system then goes off and, using a range of sorting mechanisms such as mark-up icons, textual clues and example documents, retrieves those items it thinks might fit the bill. In one window it places the matching documents, in the other it places those which have similarities to the found documents. The mark-up icons at the foot of the window each show how many of the found documents they match.
The team believes that it is vital not to misrepresent the system's intelligence to the user. The word `clues' is used instead of either `arguments' or `criteria' as it reminds the user that the system is fallible.
Other research taking place into how to represent very large bodies of information and how to navigate through them with least effort is, regrettably, on the classified list.
The team is also studying ways in which people organise their everyday information to see if they can find extensions for the user interface. For example, some people know exactly where something is, even if their office is covered with heaps of paper documents. Perhaps the deeper in a stack it is, the older it is. Or perhaps certain files are recognised by their shape or colour.
Information Access
Another research effort has addressed the problem of giving users access to information on remote computers over the network. `The project is a collaboration between a number of groups within the Advanced Technology Group, Tom Erickson of the Interfaces For Small Computers team, and Gitta Salomon representing HIG.
The research was motivated by the ever increasing number of on-line databases, most with their own unique and arcane interfaces. While these databases give users access to information, they rarely provide support for using the found information. When this team looked at how ordinary people used information, they noticed a number of things: people rapidly scanned new data, discarding or ignoring most of it; when they discovered something relevant, they often annotated it and then filed it away; finally, anywhere from days to months later, they had to search their filing cabinets, often having difficulty locating it.
Based on these observations, the team designed a system which gives users access to a variety of databases through a single interface, and which supports scanning, annotation, organisation, and retrieval of information. In addition, the system allows users to create agents which, on a daily basis, will automatically look for information in particular databases.
The metaphor used by the access software was designed from observation of the publishing world: Agents within the system are called `reporters', which you can create, summon and dismiss. You give them assignments to locate certain types of information and to publish summaries of their findings in a newspaper. The user can either scan the newspaper or reach the synopses through a front-page table of contents, requesting full reports for items of interest. Users will have several reporters working for them at one time, all publishing their results in the same electronic newspaper.
The screen where the assignment is made shows the information to search for, the databases or online systems to be searched, the frequency of searches, and the number of items which should be retrieved. The found items are ranked in order of relevance, so only the most important ones find their way into the newspaper.
The newspaper synopses are either the first 50 words of the found document, or the `best paragraph', if known. Some on-line services provide their own short synopses at the beginning of items. The index may have headings to match the synopsis headings. A list of idle reporters is also shown below the index. Special slimline scroll bars allow the user to scroll the column without getting in the way of the multi-column page design. Each synopsis has an indication of its relevance to the user based on the reporter's brief.
Whenever users find something of particular interest in the newspaper, they can retrieve the full document from the remote database and store it in a notebook. Notebooks are places where users can annotate, scan and organise the information.
A palette of annotation tools includes `post-it' notes, audio annotations and coloured highlighting `pens'. The searching facility lets the user find documents by content or by type of annotation highlight colour, post-it or voice annotation.
The scanning system shows a bird's eye view of the notebook contents. Highlights, pictures and the general shape of text are visible as you scroll through. Post-its and voice annotations are shown by icons alongside. Because users annotate the information that they are most interested in, annotations serve as good `Landmarks' when searching for things. By clicking on a spot in this bird's eye view, the main window of the notebook shows the selected page.
A second column allows the user to choose a sequence for the notebook contents. This could be by reporter, by subject, by date and so on. Most people have a feeling for when something happened, so a date outline is likely to be a popular choice. This can show years, expand the years to months, expand the months to days and so on. As the cursor is moved in time, the bird's eye window focusses on documents from that date. The main display of the notebook shows the current page with scrollbars, highlights and annotations.
The Face Of The Future
As the teams involved with the Human Interface Group will be the first to agree, their work is only part of the research that goes on within the Advanced Technology Group and other sections of the company. Having said that, the work the HIG teams carry out is vital to the future of the Mac and its successors. Without a human interface that is relevant to the needs of the day and tightly integrated to the operating system and the hardware, the Mac would be just another computer.
MacUser would like to thank Gitta Salomon, Tom Erickson, Leo Degen, Yin Yin Wong, Richard Mander, and S Joy Mountford for their help in researching this article.
THE HUMAN INTERFACE GROUP
Apple's Human Interface Group (HIG), located within the Advanced Technology Group, has co-designed interfaces such as the Help system and publishing metaphor for System 7.0, and the interface for QuickTime.
The group has two principal goals - to help design consistent, easy-to-use interfaces across Apple's existing products and to develop new research ideas and prototypes for future Apple interfaces.
It consists of an interdisciplinary group of professionals, comprising visual designers, psychologists and software and hardware prototypers.
HIG's skills are in presenting the user's point of view. Psychologists in the group often film users working. Then they develop these findings into software and hardware prototypes to match the users' needs. These prototypes are redesigned, based on user feedback and testing.
Visual designers provide HIG with the ability to storyboard scenarios, often captured in terms of a 'day in the life of' story. Rough visual scenarios involving people and stories are drawn so that precise visuals can then be made to accompany finished projects. The look and the feel of the interactions are prototyped using these visual designers' skills. Prototypes are defined with the aid of software specialists who understand the feasibility of different approaches and provide a working code. These skill sets must be resident in one group working together in order to accomplish reliable working interfaces.
Much of the group's work involves establishing an interdisciplinary design team up front, and collectively redesigning the interface. Individuals in the group debate over the best interface designs to use. Most of the individuals in the group specialise in more than one area, such as architecture, language, film, music, theatre production and mathematics. In the future, the manager of the group, S Joy Mountford, hopes to bring in even broader skill sets, including anthropologists and philosophers.