Co-Processing - controlling other programs

← Chapter_15:_Putting_It_All_TogetherHomeFile_system_backup_scripts →

Introduction to Co-processing

Shell scripts sometimes have to control a program that normally requires or enforces some type of interaction, typically directly with a user. Such programs are generally pre-existing commands that do the job that is required but which are far to complex to use in other ways.

Examples include, running command on remote servers, making database requests, programs that deal with complex data, controlling devices, communicating with network daemons.

So you end up having some 'command' that you want to run from some 'wrapping' script to do a specific task, or combining the command with some other service. And that command is the simplest method of control you have.

Controlling another program and be tricky. Not only do you need to send information to that command, but you also need to get the commands results. That is you need to wait for prompts, results, and perhaps modify your future requests to that command. You also need to watch for errors, end-of-data, end-of-file, and other conditions that could make your wrapping script 'flaky', or 'lockup'.

This technique of running and controlling a sub-daemon is also known as a Co-Processing. It simply means a background running process with its input and output connected to the shell script, or some other program, to control it.

Co-Processors in BASH...

Bash 4.0 has a formal way of starting a background co-process, using the "coproc" keyword See

There are other ways of doing this that are far less formal. Specifically the use of file descriptors, and named pipes (removed once the pipeline is opened)

Network Server Communications...

The action to talking to a very complex network server, such as a NNTP usenet news server, or a local FreeNet Node, (which I have worked on in the past) is also practically identical in complexity, to talking with a background command. A network server however is typically better defined and documented then most CLI interfaces to a library. It usually defines exactly how results and data is returned, typically in the form of a RFC (internet Request for Comments).

This includes

  • Remote control of a terminal session (via SSH, RSH, or telnet)
  • Retrieving files from some file service (world-wide-web, FTP)
  • Uploading files to some service (Flicker, Dropbox, Twitter)
  • Talking to some network server (NNTP news server, SMTP mail server)

and so on.

Two-Stage Programming...

Is a fancy way of saying, writing programs to control other programming. It is often the term used in theoretical computer science papers, where as co-processing is the programming term.

This includes things like

  • Communicating with Databases (mysql, oracle, etc)
  • Backup programs and reacting appropriately (dump, tar)
  • using control interfaces (scientific instruments, Arduino)
  • Changing Passwords

Modem 'Chat' scripts...

This is also exactly the same problem. Setting up a modem initialization, dialling and automatic login using an old modem connection. Typically after being configured and logged-in, the 'chat' script will pass the pipelines to a terminal program for the user to interact with.

So if you are old enough to have dealt with modems, you have already done some co-processing!

In summery...

All these things are essentially the same, and share the same problems. The critical aspect is that it is a two way communication, with FEEDBACK.

That is you are not just using a simple 'pipeline' such as...

  ---> Format_Requests --> 'Run Command Once' --> Parse_Results --->

but included data feedback, using the output to controls the input into the background process.

  ---> Format_Requests --> 'Background Process' --> Parse_Results --->
             ^                                           |
             `---------<------ feedback  --------<-------'

That is the results will effect the future requests that is made. Either in simply timing when those requests can be sent (synchronisation), or modifying the request based on the data returned (feedback).

A simple example of such a co-process is a web page download script, which not only downloads web pages, but also needs to login to access the web page that is wanted. Of course there are so many alternatives for this you can avoid the use of a co-process, but you may not be so lucky with other situations.

The above is the start of a larger document which can be found at [1]

That is a guide that summaries what this chapter is really about! Or perhaps is the start of a new chapter, which contains some of the parts that is currently suggested in chapter 15!