In the first in a series on Apple's future computers, MacUser visited its Advanced Technology Group in Cupertino, where researchers and scientists are developing machines for the next decade.
Written by David Tebbutt, MacUser 09/91 item 01 - scanned
Deep inside Apple Computer Inc, within its Advanced Technology Group (ATG), is a team of people dedicated to making sure that your interaction with the Mac is a fruitful and satisfying experience. This team is called the Human Interface Group, or HIG for short.
Headed by S Joy Mountford, the group is a mix of skilled professionals, most of them with at least two university degrees in subjects such as cognitive psychology, electrical engineering, visual studies or computer science. They make an interesting combination of visionaries, psychologists, prototypers and programmers. At the moment, the group is split into four teams, each concentrating on a different aspect of computing, but all working closely together and getting involved with each others' projects as their skills are needed.
In this series of features on technological developments at Apple, MacUser will give you an exclusive view of this important research establishment. In this and the next four issues of MacUser, you will learn about the human interface developments and issues connected with hand-held computing devices at one end of the scale and computers of unlimited power at the other. Along the way, we'll take in speech recognition, the integration of video into everyday computing and how we might cope with the huge volumes of information which are becoming available to computer users. We will also look at how animation may be integrated into the desktop to make computers easier to use and at ways of handling 2D and 3D information more naturally.
You will see how HIG works to make Apple's computers the most satisfying to use. You will gain an insight to the kind of thinking that lies behind each component of the human interface, and see some projects, most of which are, hopefully, destined to reach the marketplace in future Apple products. Others simply won't survive the ruthless testing and analysis.
Every scientist in the group believes in user-centred design. None of them care for the ``We've got 40mips, let's see what can we do with it'' approach. Through extensive study of people at work, they find out what artefacts and methods people use and enjoy. This enables the team to find the best ways of fitting the technology to the way people work. All the projects are logical extensions of today's Mac interface and are focused in potential growth areas.
We start, this issue, with hand-held computers.
A MAC IN THE HAND
How the hand held computers of the future are being designed to work like every day objects.
As far as the HIG team is concerned, a small computer is one which is hand held - anything from the size of an A4 pad downwards would qualify. Such a machine, without a conventional keyboard, forces you to change your thinking about what a computer is and how you might interact with it. The challenge for HIG was to design a series of prototypes with a new kind of interface.
Small Computer Interfaces
For three years, Apple researcher Laurie Vertelney has been looking at hand-held computing. She has been assessing what people would want to do with machines of this size. In conjunction with an ATG hardware and industrial design group, she initiated a project with students from the California State University, who made dozens of mock-ups of new kinds of portable machines.
This was followed by studies of people using traditional tools, such as paper notebooks. As a result, Apple decided that annotation, or mark up, would be one of the principal tasks for a series of machines that could be taken anywhere to perform different types of application.
Just over a year ago, Vertelney started looking at other applications and at new ways of interfacing to a device with no keyboard. S Joy Mountford, head of the HIG, had been to Japan with a still video camera and her frustrations with using it led to an idea for the Photologger.
Vertelney's prototype didn't even look like a computer, apart from its screen. It was an imaginary camera which stored its `photographs'' internally and displayed them on a touch-sensitive panel on the back of the camera. The operator controlled it with a finger and a stylus and could add voice notes through the built-in microphone.
The Photologger had to allow the user to flick from one picture to another, and Vertelney decided that other functions such as displaying 16 minipictures at a time would help with photograph searching. The device had a touch-sensitive display which would allow the user to use natural gestures, such as flicking and touching, to move between and select pictures. Many of the ideas which evolved were extremely relevant to a hand-held computer.
The Photologger could capture sound annotations up to 10 seconds in length. So, if a picture of a violinist was taken, a sample of the sound could be attached. The other requirement of the camera was to tailor presentations to suit the needs of different audiences. A means of classifying each photograph was also needed, according to whether it was of people, places or whatever.
The solution to the problem was to provide a row of functional icons at the foot of the screen, each relating to a classification category. To assign a category to a picture, the user would simply select the icon and slide a copy of it to the photograph. Whenever the user wanted to select only people and sights, they would activate the two icons by touch and this would instantly pull out the pictures bearing people or sights icons. These sketchy ideas worked well on the camera, but would they work on a more conventional, but small, computer?
Adapting The Interfaces To Real Life
The HIG's Small Computer Team, including Vertelney, was formed in January this year. An interdisciplinary team, drawing from different parts of ATG, was set up to develop prototypes of hand-held computers. Another group, called Systems Technologies, co-ordinates hardware integration, marketing, factory relationships and software integration. The end product of these alliances is that HIG gets working prototypes of its various inventions to try out on users.
HIG set about finding metaphors which linked to people's real-life experiences for these machines. They wanted to be sure that the end result fitted the Apple corporate identity. And, although Apple has other groups working on handwriting recognition, HIG felt that its brief should be to find a compelling user interface that could be implemented in the meantime, before handwriting recognition becomes a reality. It is considering applications for people on the move in, for example, medical, sales, marketing, teaching and shopping environments.
The researchers spent a lot of time studying the way people use conventional devices - computers, writing pads, notebooks, calculators, dictation machines and so on - in real life, especially when they were away from their desks, in meetings, for example.
When people use notebooks, HIG found that they usually have their own methods of highlighting actions and creating lists of things to remember. One user drew small telephone handsets by notes of the calls he had made and little bubbles to highlight new thoughts.
The researchers also found that people remembered where certain sections of a notebook began by attaching paperclips to the edge of the page, or by inserting slips of paper between the pages. Some, better organised than others, arranged their work into folders. All these things, and many more, were taken into account when deciding how to implement the natural interfaces to small machines.
Another conclusion was that many people don't like to use keyboards in social and other sensitive environments - for example, it would be easy for a counsellor or doctor to lose eye contact with their patients while typing.
The concept of finger control and marking documents with pre-prepared icons was brought forward from the Photologger to the new devices virtually unchanged. The research prototypes come in all shapes and sizes but, unlike the Photologger, keyboards can be attached when necessary.
The user of these machines is given the equivalent of an electronic paper pad with which to work, rather like the stack of photographs in the Photologger. The pad can be divided by named tabs poking out from the top, rather like the contents of a filing cabinet drawer. Icons, including paperclip items mentioned earlier, can be used to mark particular pages so that the user can quickly jump to the page of their choice. What actually happens is that the page concerned jumps to the top of the pad.
Like the Photologger, pages on these devices are moved forwards and backwards, up and down, by using the finger on the screen. This means that scroll bars, for so long a feature of the Mac, are no longer needed.
The machines need to feel natural to the user, and as many companies are discovering, nothing is more natural than a combination of finger and stylus. A stylus can be used as a surrogate pen, for writing or annotation. The aim is to make the changes of mode (from navigation to annotation, for example) as simple as possible.
The Problems Of Prototyping
The team has created a number of prototype handheld devices, which are usually linked by an umbilical cable to a huge desktop machine. They have built prototypes which have proved the viability of stylus and touch control. All the original prototype applications were built in HyperCard, although work has now started on real code. The team is conducting trials with real users in the laboratory and giving HIG valuable feedback on its ideas, which will lead to further refinements.
With such a series of mobile machines, security could become a real issue. If you leave a calculator sized machine around containing confidential information, you want to feel sure that no one else could gain access. One way round this problem would be to give the machine the ability to recognise different pens or the user's handwriting. As long as users keep their pens with them, the content of their machines could remain secure.
As well as new icons like the paperclips mentioned earlier, another new type of icon is an active one which could have a function, such as electronic mail access, attached to it. When activated, it could display a dialog box asking you which service you want to connect to and what you want to do when connected. Once an action is initiated like this, the task is executed next time the hand-held is connected to a telephone line. This idea of delegation of a task to an agent within the machine is one that crops up again in other HIG work.
If this all sounds rather slick, remember that the group is only creating prototypes. A problem which this research group shares with others is that prototypes look so good these days that people can't understand why it takes so long for real code to appear, and even longer for the finished product.
With touch sensitive screens the team faces a number of challenges. For example, when using a stylus, the edge of your hand tends to touch the screen. How can the computer tell that this is not a finger? How can it tell the difference between you pointing to something on the screen when talking to a colleague, and you touching the screen to trigger an event? These issues, and hundreds of others, need to be resolved and tested with users before a project like this can see the light of day.
It is very important for the team to build real applications now that they have thought through what they're trying to achieve. User testing on HyperCard prototype programs is giving a lot of feedback, but even more insights are gained when real code is implemented. At the moment, the team is coding these application ideas for further testing.
Future Possibilities
Despite the challenges the team faces in implementing today's designs, it is already looking ahead to ways of enhancing the functionality of the devices. It is considering the possibility of handling multiple simulated notebooks in the same machine. It is looking at including tools like calendars, diaries and calculators or, at a broader level, simply allowing users to add tools of their own choosing.
The team thinks it might be possible, one day, to buy products such as a CompuServe connect tool or an exotic paintbrush. Some would be low-cost items, say around $20, with which you would personalise your computer. Such additions would suggest a need for an expandable tool drawer, a theme we'll return to when we look at large-scale computing later in the series.