7.4.1 Looping in a Shell Script

If you don't already know this, it's about time you learned that the shell is a programming language as well as a command-line interpreter. To invoke ex on a number of files, we use a simple type of shell script command called the for loop. A for loop allows you to apply a sequence of commands for each argument given to the script. (The for loop is probably the single most useful piece of shell programming for beginners. You'll want to remember it even if you don't write any other shell programs.)

Here's the syntax of a for loop:

for variable in list
do
    command(s)
done

For example:

for file in $*
do
    ex - $file < exscript
done

(The command doesn't need to be indented; we indented it for clarity.) After we create this shell script, we save it in a file called correct and make it executable with the chmod command. (If you aren't familiar with the chmod command and the procedures for adding a command to your UNIX search path, see the Nutshell Handbook Learning the UNIX Operating System.) Now we can simply type:

$ correct sect1 sect2

The for loop in correct will assign each argument (each file in the list specified by $*, which stands for all arguments) to the variable file and execute the ex script on the contents of that variable.

It may be easier to grasp how the for loop works with an example whose output is more visible. Let's look at a script to rename files:

for file in $*
do
    mv $file $file.x
done

Assuming this script is in an executable file called move, here's what we can do:

$ ls
ch01 ch02 ch03 move
$ move ch??
$ ls
ch01.x ch02.x ch03.x move

With a little creativity, you could rewrite the script to rename the files more specifically:

for nn in $*
do
    mv ch$nn sect$nn
done

With the script written this way, you'd specify numbers instead of filenames on the command line:

$ ls
ch01 ch02 ch03 move
$ move 01 02 03
$ ls
sect01 sect02 sect03 move

The for loop need not take $* (all arguments) as the list of values to be substituted. You can specify an explicit list as well; for example:

for variable in a b c d

will assign variable to a, b, c, and d in turn. Or you can substitute the output of a command; for example:

for variable in `grep -l "Alcuin"`

will assign variable in turn to the name of each file in which grep finds the string Alcuin.

If no list is specified:

for variable

the variable will be assigned to each command-line argument in turn, much as it was in our initial example. This is actually not equivalent to:

for variable in $*

but to:

for variable in $@

which has a slightly different meaning. The symbol $* expands to $1, $2, $3, etc., but $@ expands to "$1", "$2", "$3", etc. Quotation marks prevent further interpretation of special characters.

Let's return to our main point and our original script:

for file in $*
do
    ex - $file < exscript
done

It may seem a little inelegant to have to use two scripts - the shell script and the ex script. And in fact, the shell does provide a way to include an editing script inside a shell script.

7.4.2 Here Documents

In a shell script, the operator << means to take the following lines, up to a specified string, as input to a command. (This is often called a here document.) Using this syntax, we could include our editing commands in correct like this:

for file in $*
do 
ex - $file << end-of-script
g/thier/s//their/g
g/writeable/s//writable/g
wq
end-of-script
done

The string end-of-script is entirely arbitrary - it just needs to be a string that won't otherwise appear in the input and can be used by the shell to recognize when the here document is finished. By convention, many users specify the end of a here document with the string EOF, or E_O_F, to indicate the end of the file.

There are advantages and disadvantages to each approach shown. If you want to make a one-time series of edits and don't mind rewriting the script each time, the here document provides an effective way to do the job.

However, it's more flexible to write the editing commands in a separate file from the shell script. For example, you could establish the convention that you will always put editing commands in a file called exscript. Then you only need to write the correct script once. You can store it away in your personal "tools" directory (which you've added to your search path) and use it whenever you like.

7.4.3 Sorting Text Blocks: A Sample ex Script

Suppose you want to alphabetize a file of nroff-encoded glossary definitions. Each term begins with an .IP macro. In addition, each entry is surrounded by the .KS/.KE macro pair. (This ensures that the term and its definition will print as a block and will not be split across a new page.) The glossary file looks something like this:

.KS
.IP "TTY_ARGV" 2n
The command, specified as an argument vector, 
that the TTY subwindow executes.
.KE
.KS
.IP "ICON_IMAGE" 2n
Sets or gets the remote image for icon's image.  
.KE
.KS
.IP "XV_LABEL" 2n
Specifies a frame's header or an icon's label.  
.KE
.KS
.IP "SERVER_SYNC" 2n
Synchronizes with the server once. 
Does not set synchronous mode.
.KE

You can alphabetize a file by running the lines through the UNIX sort command, but you don't really want to sort every line. You want to sort only the glossary terms, moving each definition - untouched - along with its corresponding term. As it turns out, you can treat each text block as a unit by joining the block into one line. Here's the first version of your ex script:

g/\.KS/,/\.KE/j
%!sort

Each glossary entry is found between a .KS and .KE macro. j is the ex command to join a line (the equivalent in vi is J). So, the first command joins every glossary entry into one "line." The second command then sorts the file, producing lines like this:

.KS .IP  "ICON_IMAGE" 2n Sets or gets ... image.    .KE
.KS .IP  "SERVER_SYNC" 2n Synchronizes with ... mode.    .KE
.KS .IP  "TTY_ARGV" 2n The command, ... executes.    .KE
.KS .IP  "XV_LABEL" 2n Specifies a ... icon's label.    .KE

The lines are now sorted by glossary entry; unfortunately, each line also has macros and text mixed in (we've used ellipses (...) to show omitted text). Somehow, you need to insert carriage returns to "un-join" the lines. You can do this by modifying your ex script: mark the joining points of the text blocks before you join them, and then replace the markers with carriage returns. Here's the expanded ex script:

g/\.KS/,/\.KE/-1s/$/@@/
g/\.KS/,/\.KE/j
%!sort
%s/@@ /^M/g

The first three commands produce lines like this:

.KS@@ .IP "ICON_IMAGE" 2nn@@ Sets or gets ... image. @@ .KE
.KS@@ .IP "SERVER_SYNC" 2nn@@ Synchronizes with ... mode. @@ .KE
.KS@@ .IP "TTY_ARGV" 2nn@@ The ... vector, @@ that ... .@@ .KE
.KS@@ .IP "XV_LABEL" 2nn@@ Specifies a ... icon's
label. @@ .KE

Note the extra space following the @@. The spaces result from the j command, because it converts each carriage return into a space.

The first command marks the original line breaks with @@. You don't need to mark the end of the block (after the .KE), so the first command uses a -1 to move back up one line at the end of each block. The fourth command restores the line breaks by replacing the markers (plus the extra space) with carriage returns. Now, your file is sorted by blocks.

7.4.4 Comments in ex Scripts

You may want to reuse such a script, adapting it to a new situation. With a complex script like this, it is wise to add comments so that it's easier for someone else (or even yourself!) to reconstruct how it works. In ex scripts, anything following a double quote is ignored during execution, so a double quote can mark the beginning of a comment. Comments can go on their own line. They can also go at the end of any command that doesn't interpret a quote as part of the command. (For example, a quote has meaning to map commands and shell escapes, so you can't end such lines with a comment.)

Besides using comments, you can specify a command by its full name, something that would ordinarily be too time consuming from within vi. Finally, if you add spaces, the ex script above becomes this more readable one:

" Mark lines between each KS/KE block 
global /\.KS/,/\.KE/-1 substitute /$/@@/
" Now join the blocks into one line
global /\.KS/,/\.KE/ join
" Sort each block--now really one line each 
%!sort
" Restore the joined lines to original blocks
% substitute /@@ /^M/g

7.4.5 ex Scripts Built by diff

A further example of the use of ex scripts is built into a UNIX program called diff, which compares two files and prints out lines that differ. The -e option of diff produces an editing script usable with either ed or ex, instead of the usual output. This script consists of a sequence of a (add), c (change), and d (delete) commands necessary to recreate file1 from file2 (the first and second files specified on the diff command line).

Obviously there is no need to completely recreate the first file from the second, because you could do that easily with cp. However, by editing the script produced by diff, you can come up with some desired combination of the two versions.

It might take you a moment to think of a case in which you might have use for this feature. Consider this one: two people have unknowingly made edits to different copies of a file, and you need the two versions merged. (This can happen especially easily in a networked environment, in which people copy files between machines. Poor coordination can easily result in this kind of problem.)

To make this situation concrete, let's take a look at two versions of the same paragraph, which we want to combine:

Version 1:
The Book of Kells,
now one of the treasures of the Trinity College Library 
in Dublin,
was found in the ancient monastery at Ceannanus Mor, 
now called Kells.
It is a beautifully illustrated manuscript of the Latin 
Gospels, and also contains notes on local history.  
It was written in the eighth century.  
The manuscript is generally regarded as the finest 
example of Celtic illumination.

Version 2:
The Book of Kells
was found in the ancient monastery at Ceannanus Mor, 
now called Kells.
It is a beautifully illustrated manuscript of the Latin 
Gospels, and also contains notes on local history.  
It is believed to have been written in the eighth century.  
The manuscript is generally regarded as the finest 
example of Celtic illumination.

As you can see, there is one additional phrase in each of the two files. We can merge them into one file that incorporates both edits. Typing:

$ diff -e version1 version2 > exscript

will yield the following output in the file exscript:

8c
It is believed to have been written in the eighth century.  
.
1,3c
The Book of Kells
.

You'll notice that the script appears in reverse order, with the changes later in the file appearing first. This is essential whenever you're making changes based on line numbers; otherwise, changes made earlier in the file may change the numbering, rendering the later parts of the script ineffective. You'll also notice that, as mentioned, this script will simply recreate version1, which is not what we want. We want the change to line 8, but not the change to lines 1 through 3. We want to edit the script so that it looks like this:

8c
It is believed to have been written in the eighth century.  
.
w

(Notice that we had to add the w command to write the results of the edit back into the file.) Now we can type:

$ ex - version1 < exscript

to get the resulting merged file:

The Book of Kells,
now one of the treasures of the Trinity College Library 
in Dublin,
was found in the ancient monastery at Ceannanus Mor, 
now called Kells.
It is a beautifully illustrated manuscript of the Latin 
Gospels, and also contains notes on local history.  
It is believed to have been written in the eighth century.  
The manuscript is generally regarded as the finest 
example of Celtic illumination.

Using diff like this can get confusing, especially when there are many changes. It is easy to get the direction of changes confused or to make the wrong edits. Just remember to do the following:

• Specify the file that is closest in content to your eventual target as the first file on the diff command line. This will minimize the size of the editing script that is produced.

• After you have corrected the editing script so that it makes only the changes that you want, apply it to that same file (the first file).

Nonetheless, because there is so much room for error, it is better not to have your script write the changes back directly into one of your source files. Instead of adding a w command at the end of the script, add the command %p (or 1,$p) to write the results to standard output. This is almost always preferable when you are using a complex editing script.

If we use this command in the editing script, the command line to actually make the edits would look like this:

$ ex - version1 < exscript > version3

Writers often find themselves making extensive changes and then wishing they could go back and recover some part of an earlier version. Obviously, frequent backups will help. However, if backup storage space is at a premium, it is possible (though a little awkward) to save only some older version of a file, and then keep incremental diff -e scripts to mark the differences between each successive version.

To apply multiple scripts to a single file, you can simply pipe them to ex rather than redirecting input:

$ cat script1 script2 script3 | ex - oldfile

But wait! How do you get your w (or %p) command into the pipeline? You could edit the last script to include one of these commands. But there's another trick that we ought to look at because it illustrates another useful feature of the shell that many people are unaware of. If you enclose a semicolon-separated list of commands in parentheses, the standard output of all of the commands are combined, and can be redirected together. The immediate application is that, if you type:

$ cat script1 script2 script3; echo '%p' | ex - oldfile

the results of the cat command will be sent, as usual, to standard output, and only the results of echo will be piped to ex. But if you type:

$ (cat script1 script2 script3; echo '%p') | ex -oldfile

the output of the entire sequence will make it into the pipeline, which is what we want.

7.4.6 Where to Go from Here

If this discussion has whetted your appetite for even more editing power, you should be aware that UNIX provides editors even more powerful than ex: the sed stream editor and the awk data manipulation language. For information on these programs, see the Nutshell Handbook Sed & Awk.