A Graphical Approach to Some Shell Programs




James L. Peterson




Advanced Workstations Division
IBM
11400 Burnet Road
Austin, Texas 78758




27 March 1990


1.0 Pipes

One of the advances in programming environments which has come from the Unix system has been the concept and implementation of pipes. The pipe concept allows independently developed programs, run as separate processes, to communicate via a simple communication mechanism. One of the communication processes writes to the pipe; the other process reads from the pipe. The pipe is a simple FIFO queue, allowing an efficient implementation of producer/consumer systems.

The main Unix shells (command interpreters) provide a convenient syntax for creating pipes and linking together processes to form larger composite systems. The vertical bar character ("|") is used to separate commands which define the communicating processes. Data flows from left to right. Thus,

      cat file1 | sort | uniq > file2

creates three processes and two pipes. The "cat file1" command copies the contents of "file1" into a pipe. The "sort" command reads the contents of file1 from the pipe, sorts it and outputs it into another pipe. This second pipe, the output of sort, is the input to the "uniq" command which places its output into file2.

The use of pipes allows a new paradigm in program construction. Rather than attempting to write one large program to accomplish a task, it is often possible to write several smaller programs which can be connected with a pipe. The type of program composition supported by the pipe concept is not easily accomplished within a program. On systems which do not support pipes, it is generally possible to simulate the semantics of a pipe by intermediate files. For example, the above example could have been written, without pipes, as:

    cat file1 > /tmp/file.a
    sort < /tmp/file.a > /tmp/file.b
    uniq < /tmp/file.b > file2

However, notice that there are possible differences. The temporary file solution requires enough disk space for multiple copies of the intermediate files, while the pipe solution need not create disk copies of intermediate results. In addition error handling for the two solutions may differ.

2.0 Graphical Form

An immediate demonstration of the power of bit-mapped, window system based displays is the ability to display pipe-based compositions of programs in a graphical form. Rather than writing

	 sort file1 | uniq > file2

it is possible to create an editor which would allow such a pipe to be defined graphically:

	 /------\      ----------     -----------      /--------\
	/	 \     |	|     |         |     /	         \
	|  file1 |---->|  sort  |---->|  uniq   |---->|  file2	 |
	\	 /     |	|     |         |     \	         /
	 \------/      ----------     -----------      \--------/

The editor would allow both the components to be defined and their interconnection structure. The basic form would be a sequence of boxes, one for each command. Files could represent sink and/or source of the transformed information. A menu of standard commands (such as cat, sort, and uniq) and/or a "fill-in-the-blank" option can be used to define the contents of the commands to be piped together.

3.0 Extension to Non-Linear Forms

A short use of such a system, however, leads to obvious limitations of the system. The Unix commands "comm" and "diff" (which are used to compare two different files) require two inputs, not one. Thus, it would seem that they are not suited for use with pipes. The "comm" command, in particular, however, requires that its inputs be sorted. Hence, if we want to find the common elements of two files, we have to resort to scripts like:

       sort file1 > /tmp/f1
       sort file2 > /tmp/f2
       common -12 /tmp/f1 /tmp/f2 > outfile
This script cannot be handled as a simple pipe. However, a simple extension of the standard linear pipe format easily provides a simple pipe structure for this example:

          -----------	    -----------
	  |         |	    |	      |
	  |  file1  |       |  file2  |
	  |         |	    |	      |
          -----------	    -----------
	       |                 |
	       |                 |
	       |                 |
               v                 v
	     -----------------------
	     |                     |
	     |      comm -12       |
	     |                     |
	     -----------------------
	                |
	                |
	                |
			v
		   -----------
		   |         |
		   | outfile |
		   |         |
		   -----------

A graphical pipe editor can allow much more complicated systems to be built in a much clearer graphical representation. A graphical pipe editor provides a form of visual programming, allowing the program representation (the graph above) to clearly represent the actual flow of data in the program solution (data flow programs).

4.0 Implementation and Details

The graphical pipe editor would be a combination of a specialized drawing editor and a command interpretor (shell). As a drawing editor, we might design something like the "fig" editor. A row of items along the side would allow the user to select a box (for a command) or a circle (for a file), or to connect them together with directed arcs (for the pipes). A fill-in dialog window associated with each circle would allow the file to be specified as an absolute or relative path name. A similar dialog window for each command would allow the command name and options to be defined.

In addition to the generic commands, a set of specialized commands would need to be created. For example, the "comm" command takes two inputs (sorted files) and produces three outputs (the lines in the first, but not the second; the lines in the second, but not the first; and the lines common to both input files). Older shells allow only a linear pipe syntax, and so expect their standard input on file descriptor 0 and produce their standard output on file descriptor 1. This scheme could not handle 2-input/3-output commands and so "comm" expects the names of its two inputs on the command line and maps all of its outputs to a single output stream.

With a non-linear representation, we can handle commands with more than one input and more than one output. However, two conventions will be needed: (1) how to attach the inputs and outputs, and (2) a way to describe the command. To continue the basic pipe structure, the command programmer would expect its inputs and outputs to be attached to file descriptors. By default, we would expect one input (on file descriptor 0) and one output (on file descriptor 1). A more complete description would be able to override these defaults.

A complete command description would include the following fields:

For example, a graphical "comm" command might be described by:

The graphical shell would use the command description to generate a display shape with appropriate notches for the inputs and outputs. In the "comm" case, it would create a rectangle with two inputs and three outputs. Either a horizontal or vertical representation might be allowed. Inputs and outputs might be graphically distinguished, such as:

       ------------------------------
       |                            |
       |			    --
       |		   file1 only >
       --			    --
         > file1		    |
       -- 			    --
       |	     comm      common >
       --			    --
         > file2		    |
       --			    --
       |		   file2 only >
       |			    --
       |			    |
       ------------------------------

(Obviously this can be drawn much better with a bit-mapped display, but the ideas should be obvious: input and output ports differ in a way that suggest input and output; each port is labeled to indicate the semantics of its use; each box is labeled with its command name.)

A common system-wide table of these descriptions would be kept for the system provided commands (like sort, comm, uniq, diff, sed, lex, yacc, and so on). In addition, a user-specific table would provide a description for commands written by a particular user, and the user would be able to provide table entries on the fly as necessary, with a pop-up dialog box to fill in the field values. These various tables would be read into the graphical shell and kept to determine the display and execution model for each command.

After a structure was created by the user selecting commands and linking them together with pipes, the user could ask that the composite be executed. The editor would first check that all inputs and outputs of all commands are properly defined, and that all named input files exist. Named output files would be created. A special source and sink icon (possibly the electrical "ground" symbol) would be used to indicate outputs that are to be discarded or inputs that are not provided -- these would be mapped to /dev/null.

For operating systems with large numbers of allowed file descriptors, the graphical shell would then create a large number of pipes, one for each pipe in the graph structure. It would then spawn subprocesses, one for each command. Each subprocess would first remap the ends of the pipes that it needs to the file descriptors from the command table description entry for that command (using the 'dup2' system call), then invoke the command (using the 'exec' system call). If only a limited number of file descriptors can be open at a time (and the total number of pipes/files exceeds this limit), then it would be necessary to intermix the creation of pipes and forking of subprocesses to stay within the system limit on file descriptors.

During execution, various feedback mechanisms could be used by the graphical shell to keep the user informed of progress. At the minimum, each node could be shaded as it finished execution. Alternatively, statistics could be displayed about the usage of each pipe: number of bytes read, written, and remaining in the pipe. Output to standard error (file descriptor 2 by default), could be directed to a pop-up window near the command generating the output. Exit status codes could be attached to each command to show error exits, possibly flagged with a red color.

A special "spy" command would be provided which could be easily inserted or deleted into any pipe. The "spy" command would pass all input to its output, hence its use would be transparent to the commands on either end of the pipe, but it would also create a pop-up window whose contents would provide a scrollable trace of the bytes flowing in the pipe. This would allow graphical commands to be debugged, by examining the flow of data from one command to another.

Similarly a type-in command would allow the user to input data to a command and a display command would create a scrollable window displaying output from a command.

5.0 Graphical Shell Language

The representation of a non-linear pipe system can be easily and naturally allowed on a bit-mapped windowed display. It is also possible to store the data structures representing such a system in a file. This file can be taken as a "shell script" for the graphical shell. For example, if the graphical shell editor/interpreter is called "/bin/gsh", the first line of a graphical shell script would be:

    #!/bin/gsh

Standard Unix loader convention then means that an invocation of a graphical shell script, as an executable file, will invoke the graphical shell. The graphical shell would then create appropriate internal data structure to represent the program, and begin execution.