| Home | Up | Questions? |
Early in 1999, a number of trends came together to create a company, Netpliance, to produce one of the first Internet Appliances. The idea was to create a hardware product with software to allow a customer to surf the web and send and receive e-mail without needing to "own a computer". Half of the U.S. public do not own computers, and it was thought that they would be interested in a device that they could just use rather than have to set-up, configure, and support. The objective was to get "out of the box, and onto the net" as easily and quickly as possible.
The primary distinguishing characteristic of our offering is simplicity of the user experience for sending mail and accessing information from the Internet. We are able to do this by tightly coupling the device, in the user's home, and our ISP service. Knowing who our users are, what access devices they are using (the i-opener), and what applications they can use (e-mail and web surfing) enables us to eliminate all user configuration and device management -- the complexities that the PC world requires.
Our access device is not a general purpose PC. Our servers are fully aware of the device configurations it supports and, in fact, controls and manages these devices throughout their lifetime. The transparent remote software management of the device is necessary and desirable in order to offer a more reliable, simplified user experience.
Figure 1 shows a block diagram of the two primary components of our system. The system was designed to support large numbers of devices (millions) with potentially many server complexes consisting of multiple servers, each possibly containing multiple processors.
By designing both of the device and the servers, we greatly improve the overall user experience of accessing the Internet. Our long-term core value is the Server Complex and the custom software that connects the device to the portal. The i-opener device, however, is initially a critical component as the traditional PC with a standard operating system cannot provide the type of effortless user access device we envision. The i-opener device also serves as a reference implementation of our middleware and service for other device manufacturers.
Figure 1. A block diagram showing one instance of an i-opener device connected to the Netpliance server complex.
The i-opener hardware was designed as a consumer product. It has the appearance of just a flat display screen on a stand. The display is a 10-inch flat LCD DSTN screen, creating a basic 9 x 12 inch product, configured to a resolution of 800 (horizontal) by 600 (vertical) with 16 bits of color for each pixel. In addition to the attached keyboard, it has connectors for power, a telephone line, a printer and a mouse in the back. Behind the display, a circuit board contained the electronics -- a diskless single-board computer system.
Figure 2. The i-opener device.
The processor was a 200Mhz x86 chip providing the equivalent of a Pentium II. Several different, inexpensive, chips were used from several vendors. We changed vendors as necessary when chip sets were phased out as too old and slow.
Memory consisted of 32M of volatile RAM with 16M of compact flash memory. The compact flash memory was used for long-term storage, to avoid needing a disk, due to size, power, and reliability issues. Accordingly, there was no disk in the i-opener.
An on-board 56K serial modem supports communication. In addition, a parallel printer port and a USB port are provided.
A standard keyboard was customized by altering the function key tops to tie in with our software. The function keys were labeled to bring up mail, news, the web guide, and other functions. For example, the "ALT" key next to the space bar was relabeled "Pizza" and was a signal to the software to bring up a web page for a chain of Pizza restaurants (Poppa Johns by default).
The initial design included a small rubber disk about an inch in diameter, on the keyboard, used to drive the cursor. This was sometimes difficult to use, and most developers and many customers preferred a real mouse. We were able to add a mouse thru the USB port.
For development purposes, a number of facilities were available on the board that were not brought out for the normal customer. A second serial port was used for debugging, and an internal IDE port allowed a disk to be attached for initial software development. By using very standard hardware, we were also able to develop much of the system software on standard desktop PC's while the hardware was being designed and developed.
Figure 3. The inside of the i-opener device.
One difficulty that we had was caused by our comfortable hardware. Some early purchasers took the unit apart and determined empirically that by changing a few jumpers and attaching a disk to the IDE port, the i-opener could be made into a general purpose computer, running Linux or Windows. Since our business model was to sell the hardware at a loss initially to establish ourselves in the market place, charging only $100 to $300 for hardware that cost $500 to manufacture, it was necessary to modify the hardware to discourage this conversion capability.
A number of application packages were written for the i-opener device. The user interface of these applications was designed to be simple and easy to use, keeping in mind that our target audience was at the least unfamiliar with computers, if not computer-phobic. We designed specifically for "the 85-year-old grandmother".
A new device requires only two things to be useful: electrical power and a telephone line. When the device is first removed from the box, and turned on, it presented a tutorial, explaining with text and sound how to use the mouse, the keyboard, and the various applications. This provided the basic information needed for someone who might not be familiar with a graphical computer interface.
At the same time, while the user was busy with the out-of-box tutorial, the device would connect, over the phone line, with our servers. The device would report that it was in operation, and would be provided with up-to-date information concerning phone numbers, software, user information (entered into our servers when the device was sold and service started), and initial content. Thus, by the time the user had finished the tutorial, the system would be up-to-date and operational.
An on/off button on the front of the i-opener allowed the user to turn the device off. When the device was later turned on, it would go directly to the home page; the tutorial was only run once. The on/off button, while it had the appearance of turning the device off, did not remove electrical power from the device; the device was normally always powered on. Rather, when the power off button was pressed, a software event was generated that turned the screen off, and reset all applications to their home state. Even when the device was "off", it would occasionally dial out to fetch new content and e-mail messages, so that it would be up-to-date when it was turned back "on".
Figure 4. The Home Page of the i-opener.
When the i-opener was turned on, it went to the "Home" page (Figure 4). The i-opener home page was designed to provide easy access to the features of the i-opener. A set of on-screen buttons provided links to other pages designed to provide content (news, sports, entertainment, finance, and weather), as well as the e-mail application, a shopping page, and a web guide. One of the options on the web guide allowed users to type URLs, to browse the web.
Figure 5. Clicking on any of the circular links on the home page takes you to other content pages.
The links on the i-opener home page were easy to use with either a mouse or a touch screen. Special keys on the keyboard could also be used to get to many of these content pages.
One major aspect of the i-opener was "cached content". At regular intervals (originally every 4 hours, eventually every 8 hours), the device would dial out to our servers and download new cached content -- news, sports, finance, entertainment, weather. This meant that this information was available on the device and could be read off-line. We received our basic content from Reuters, and later from Go.com and reformatted it for the i-opener.
The weather needed to be localized for the location of the i-opener, but the rest of the content was the same nationwide.
If the device was idle for long enough, a screen saver would come up. The screen saver would display an updated number of unread e-mail messages. Headlines from the cached content was cycled on the screen when the screen saver was up, so that the customer might be attracted to a story when they were just around the idle device.
In addition to the on-device cached content, the device allowed web browsing. Given the nature of our audience, we created a short "Web Guide" and "Shopping" list of links, categorized by topic, to ease finding topics on the web. In addition, the user could simply type in URLs as desired. The user could "bookmark" sites by adding them to their "Favorites" list.
E-mail was supported by the IMAP protocol. The e-mail application displayed mail and allowed the user to compose mail and send it, using the "sendmail" protocol. The device checked for new mail whenever it was on-line, particularly at the same time it was downloading content (news, weather, ...). An LED on the front of the device would be turned on when new mail arrived.
We considered two alternatives to serve mail: IMAP and POP (since we had control over the client software, we could have developed our own protocol but that was not in our best interest). IMAP was chosen over POP because it allowed us to more easily develop an email client that:
The disconnected mode of operation was critical to our user's experience. We wanted the user to be able to read, compose, organize and send emails without having to worry about "going online" and connecting to the mail server. In addition, the user should be able to perform these operations on the same account but from different i-opener devices (this is the "roaming user" feature).
Note that to support disconnected roaming users, the client effectively becomes a cache and the problem of roaming users is reduced to that of multiple cached copies of the same data. Messages are copied from the server to the client, manipulated on the client, and later resynchronized with the originals on the server.
This fulfilled one of the original goals of the i-opener which was that all data on the device should be backed up on the server. This goal helped us (1) streamline the process for returning a failed device (we could simply furnish a new device to the user and all customizations, mail, etc would be reloaded onto the new device from the server), and (2) build a more fault tolerant system (when the mail client detected corrupt files due to things like filesystem corruption resulting from improper shutdowns it would simply reload the data from the mail server).
The fact that the mail client accessed cache copies of mail stored on the server also allowed us to conserve disk space. For example, the mail client could be configured to use only 2 megabytes of disk space for local storage even though the quota on the servers was 4 megabytes. When the 2 megabytes of local space was used up, the mail client could simply discard some of the cached items to make room for new mail. This feature is even more powerful given IMAP's ability to individually download a message's MIME parts. For example the client could elect to throw away images and keep text, and simply re-download the images when needed.
The IMAP protocol was designed with these features in mind, so it was a natural choice for our mail server.
Part of the initial design included support for different types of i-opener systems, for different target audiences: kids, teens, men, women, ..., but this was quickly limited to just a "standard" version with separate Web Guides and content categories for Kids and Teens. Kids and Teens were also sent through a proxy for web-surfing, which allowed their access to the web to be limited to a known list of acceptable sites.
The i-opener was developed on the QNX 4.25 operating system. QNX is a real-time operating system which allows processes to communicate by sending messages (using a request-reply model). It was chosen for many reasons: small, fast, mostly POSIX-like APIs. QNX supports its own windowing system (Photon) and its own web browser (Voyager). Some applications were provided in source form, allowing our developers to use them as models for our own application programs.
Various system development was necessary for our environment. The compact flash was used for the storage of file systems, modeled after a disk system. File systems were created on our development host computers, and then compressed. These compressed file systems were downloaded onto the device. A special file system driver was used to allow these compressed file systems to be uncompressed through a small memory buffer and made available to the system as a virtual file system. A read access to the file system would copy and decompress a block from compact flash memory into RAM. A write to the file system was cached strictly in RAM. This model was particularly valuable for all the system data files and executable files, as well as the cached content, all of which was effectively read-only.
A suite of application programs were written to provide e-mail and web browsing. These programs were written to directly create images using the Photon windowing system. A limited "window manager" was created to interpret the hot-keys and switch back and forth between the different applications as necessary. A protocol was developed to allow the applications to cooperate.
For example, if the user pressed the "Mail" button while the "News" application was running, the window manager would first send a message to "News", telling it that it was about to be quiesced (moved "off-screen"). That would allow "News" to save any state that it needed. Then the window manager would send another message to "Mail", telling it to "wake-up". Mail would respond with a message asking what was happening. The window manager would respond with a message that it was now the "current" application. Mail would activate and refresh its state, and then ask to be put "on-screen", at which point the window manager would move "News" off-screen and put Mail on-screen.
One process in the system is the event manager. Applications register with the event manager to be informed of events that each find interesting (for example, Power Off, User is Idle, Connection is Up, ...). Whenever an event happens, the code that causes or recognizes that situation, then posts that event with the event manager, causing other applications to be notified of the event.
Other than e-mail, most of the applications were designed to be the display of HTML pages. The home page, the shopping and Web guides, the display of cached content, were all reduced to pointing the web browser at a local file. Thus, e-mail and the browser became the two primary applications.
Cached content was initially downloaded as a specific set of files, formatted in HTML, which were presented by the browser. Most cached content was the same for all users, but some, such as the weather page, depended upon the i-opener's physical location. Rather than require the server to know, and download, the appropriate file, the software was designed to have the device request the correct file. A local HTML server process was run on each device, and the links to cached content were changed from file: links to requests from the local HTML server. The content pages were changed to a general XML format, including special executable tags, which could query device and user specific personalization information, and then generate personalized HTML pages from the XML. These generated HTML pages were passed from the local HTML server to the browser and displayed.
Personalization of information (such as the local weather page), is defined by a set of user-specific information. This information is entered into a data base on our servers and then provided to each specific i-opener. Initially, the i-opener, upon first use, calls our server and identifies itself by its device serial number. The device serial number is used as to query the server data base, extracting other information, including its physical location (for local weather and to define a local phone number for further access), as well as user e-mail account names, and so on.
In general, we expect there may be a need for a significant amount of device and user specific information. While most of it may be produced and used on the device, and so could be left there, that would mean that a failure of the device would lose all that information. So user and device specific information is kept on our servers, and transferred to the device as necessary. This function is provided by DSM, the distributed storage manager. DSM keeps information on the servers, downloads it as necessary to the devices, and then keeps the two copies of information, on the servers and the device, consistent as it is changed either by the user or by system administration.
As the system has developed, a large amount of information is stored in DSM for each device, including the information necessary to contact the servers (phone numbers, passwords, server IP addresses, ...), user information (name, email account), user data (list of favorite sites, list of visited sites, address book entries, ...), as well as configuration information (special services and accounts).
The i-opener software manages the connection to the servers using the built-in modem and a phone line. Since the phone line may also be used for other purposes, the i-opener uses a strict "pull" model -- the i-opener always calls the servers and pulls information down; the servers never call the i-opener to push information onto it.
The i-opener connects to the servers under two conditions: (a) whenever the user attempts to visit a page which is not on the device or asks to check for new e-mail, and (b) periodically, a couple of times a day, to check for new content. A connection manager takes "votes" from the applications, brings the connection up and down as necessary, and notifies applications when the connection state changes. Several applications, like e-mail, cached content and DSM, are opportunistic and will use the connection whenever it is up, but try not to force it up. At the same time, whenever the overhead of bringing the connection up occurs, we would like to use it as much as possible. This approach is designed to limit the time attached to the network to reduce costs and the amount of server support for each device.
Connection time was further reduced by compressing the cached content pages. The decompression was done by the local HTML server, allowing the cache content pages to be both transmitted and stored in compressed form, lowering both transmission and storage costs.
To improve the i-opener system, the various processes (e-mail, web browser, ...) create log entries to trace the use of each device. These log files are uploaded to the servers whenever a connection is made, allowing summary reports to be generated on the servers of user activities. The log files can be particularly useful for debugging and problem diagnosis.
For security, all logging message and all DSM information is encrypted, using an SSL connection, when it is sent between the i-opener device and the servers.
The initial design of the i-opener expected that, as new software was developed and improved, it would be necessary to update the software on the i-opener to improve the user experience. This was necessary even after the device was sold and in the field. The system was designed to allow its software to be updated in the field with no user intervention. Every part of the system can be remotely updated: data files, program files, the operating system itself, even the object code for the BIOS and the modem.
Updating an application involves more than simply downloading the appropriate new program data and executable files. Since the application is, almost certainly, currently running, updating requires the ability to stop the current application, update the appropriate files, and then restart it. A scripting language was invented to allow the update process to download the updated files, check that they are received correctly, wait until the system is idle (to avoid interference with the user), stop the affected processes, update the file system, and then start the new processes.
Replacing the operating system, on the fly, is an even more difficult operation. Updating the operating system requires stopping the system and re-booting the new operating system. A feature of the BIOS allows the system to be booted from either of two partitions in the compact flash. To update the operating system, the new system is downloaded into a second partition of the compact flash. Once this has been done, and the correctness of the new operating system verified, the BIOS partition table is modified to change the specification of the boot partition, and the system is stopped and rebooted, bringing up the new operating system and its applications.
Updating software on remote devices is a critical feature of the i-opener system. In the worst case, a bad update could cripple the device, making it unusable and require that it be physically sent back for repair. Accordingly, a number of steps have been taken to avoid this situation. For example, an update of the operating system requires that the new operating system reset a watchdog timer in the BIOS at regular intervals. It the new system fails to function, it will not reset the watchdog timer, and the BIOS will detect the failure of the new operating system. The BIOS will then automatically reset the boot partition information and reboot back to the previous operating system. This prevents an attempt to update to a bad operating system from crippling the device.
In addition, the entire software system has been designed so that the update process itself depends upon as little as possible. If a bad update occurs, which creates a system with a bad HTML server, browser, e-mail, window manager or other application, update will still be able to function, allowing our support personnel to roll the system back to earlier versions of the software. Since the update process itself is still functioning, the system can be restored to operation while still in the field.
All software, support files, and content are stored in the 16Mbyte compact flash, since there is no disk. The compact flash appears to the software as if it were a disk. It is partitioned into 4 disk partitions. These disk partitions are used to store the compressed file systems. Two partitions, each 2.5Mbytes, are used to store the operating system and base services (DSM, update, logging, and the connection manager). Two partitions allow one to be used while the other is updated, and then the two partitions are interchanged. The remaining compact flash is partitioned into two partitions, one for the application code (4Mbytes) and one for content (7Mbytes).
A continuing problem with the system is caused by the fixed, limited amount of memory (32K of RAM; 16K of compact flash memory). The use of memory was under constant scrutiny. Each application needed to be limited in it's memory usage. Memory leaks had to be found and fixed. All programs had to be ready to be told "no" when they asked for more memory.
Another approach towards reliability was to accept that at times, any application could fail. Thus every application, which ran as a (child) process, had some other process (a parent process) which was responsible for starting the child process and keeping it running. If the child process should crash, the parent process would log that fact, and restart it. Similarly, some other process watched over the parent process. One process (phoenix) was developed with no other function than to start and restart, as necessary, another process. Only the phoenix process did not have any other process watching it -- it was written to be as simple and correct as possible, and tested extensively.
XXX Server side provisioning.
Our hardware design has been exceptionally useful. Simple hardware design has resulted in very high hardware reliability. We stayed well back from the edge of the "envelope". For example, while our memory chips are designed to be run at 100Mhz, we run them only at 66Mhz. The processor is idle 95 percent of the time, since it is mainly constrained by user input and modem speeds, so there is little advantage to running the memory faster.
It would have been possible to pack the system components into about 1/4 the space that we use, but the size of the display defines a much larger size, and allows a much larger board with easy layout and no problems with heat.
The overall design of the system makes it very easy to use. The graphics are large and clear. However, the overall user experience is driven by the web browser. While an HTML 3.1 browser is very adequate for the display of our cache content, it has significant problems on the web.
Many HTML pages on the web have significant syntax errors in their content (missing beginning or end tags, unterminated strings, illegal values, ...). Both Netscape and IE tend to ignore and/or correct the errors in similar ways, which means other browsers must do the same. When the errors are pointed out to the web page designers, their response tends to be that it may be wrong, but it works with Netscape and/or IE, so why should it be changed?
A large number of sites check for and allow only the two main browsers: Netscape and Internet Explorer. These two browsers so dominate the design of web pages that sites look for specific versions of them to determine what features are expected on the browser. HTML 4.0 graphics, SSL, cookie support, and other features are determined by checking the name and version number of the browser (either from the HTTP user agent string, or the JavaScript navigator fields). Financial and e-commerce sites, in particular, will often check the version of the browser and then redirect to a page asking that the user upgrade their browser.
One option would be to contact the owners of all these web pages and request that they modify their pages to support other browsers. This tends not be very fruitful.
Another option is to masquerade as one of these browsers, generating an HTTP User Agent which identifies our browser as one of either IE or Netscape. This places our browser at some risk, since it is not what it claims to be. We understand, for example, that an IE browser which interacts with a Microsoft web server, uses non-standard communication ports for some functions.
The practical result is that to be fully able to support the full range of pages available on the web, it is necessary to actually use either the latest Netscape or IE browser. IE runs only on Windows platforms. Netscape runs on Windows, Macs, or Linux, but requires at least 64M of RAM. For the embedded, appliance environment Netscape is much too large.
The result is that, after developing and selling the i-opener base on the QNX 4.2.5 operating system and browser, it became apparent that it would be necessary to move to IE and, consequently, a Windows platform. To minimize requirements and be able to run on our current hardware, we chose Windows CE.
Moving our system to Windows CE presents significant problems. Windows CE is limited in the number of processes that it can run (only 32), so it has been necessary to redesign our system for extensive use of threads instead of processes. In addition, all of our graphics applications must be rewritten, since QNX Photon apps are not similar to Windows apps. Our solution here is not to rewrite them for Windows, but to port and rewrite them to run on Tcl/Tk, which then runs on Windows. This approach allows us to run the same apps on Linux and Windows NT for development and debugging.
The ability to modify and update systems in the field has been extremely useful, as we have learned and modified our user interface. Early ideas of a "channel" metaphor have been discarded as being too subtle and confusing for the user. The concept of separate versions for Men and Women (and other target audiences) was discarded, with the only changes being to server data base files, changing DSM values. The next time the user devices contacted the servers, they picked up the new configuration variables and presented a common version.
As experience with the system grew, software problems were identified and fixed, resulting in more capable and reliable operation. New graphic designs have increased function while simplifying the appearance. New applications (instant messaging and chat rooms) are under development. As the new systems were developed, tested, and accepted by operations, it was possible to easily upgrade the units in the field to the most recent system.
At the same time, the ability to update units remotely can be very dangerous. When a large number of devices are updated, problems may show up that were not expected. In one release, for example, we modified the system to update a file to capture the last time of an operation. This was done by a "touch" command on the file, updating its modification time. Since we wanted the file to exist after a reboot, we put it in the root directory. The result of this was that at frequent intervals (every few minutes), the file was touched, causing its directory entry to be modified. This required a write of the root directory to the file system, creating a small window of vulnerability. If the system was powered off while the root directory was being written, the file system would be corrupted, and be unreadable, and it turned out, unrepairable -- the file system could repair any problem except a corrupted root directory. While this never showed up in the lab, with tens of thousands of units in the field, it became a significant problem. Previously no file in the root directory was ever modified, and the root directory never needed to be written once the system was installed. A simple change to the path name of the file, moving it out of the root directory solved the cause of the problem; but it illustrates how complex and fragile the system can be.
Similarly, two server crashes which corrupted our e-mail IMAP data files, illustrate that additional work needs to be done to allow for problems and solutions on the servers.
| Home | Comments? |