// Patrick Louis

Tools, glue, scripts, and automation on Unix

Tools, glue, scripts, and automation on Unix

https://nixers.net/showthread.php?tid=2191

Introduction

Today we take for granted the concept of software as a tool but it didn’t always exist. Mini-scripts, the interoperable programs, the small utilities for specific tasks, etc.

This is what we’re going to discuss, where do they come from, the history, and a bit more.

What are tools

There is a type of program that we write for a special run-time environment that automates the execution of tasks that could alternatively be executed one-by-one manually.

This is what we call scripts.

There are other definitions of what scripts are but let’s stick to this one for this podcast.

Usually scripts are interpreted, like we said “run-time”, rather than compiled.

For example, the shell is a scripting environment. Perl also, and other dynamic high level languages, could be considered scripting environments.

Perl might be a bad example since it actually uses a bytecode compiler.

The usual idea of scripting is to combine smaller repetitive tasks to create more complex tasks and automate them, script them.

However, one problem that emerges from this technique is that programs may still need to communicate with one another. It becomes mandatory to insert a middle-man that will reshape the output of one program so that it’s compatible with the input of another.

This code or software that is added is the glue code or glue language and its sole purpose is to connect software components together.

The concept of glue code is seen everywhere, from wrappers like decorators in some languages, to adapter patterns in object-oriented programming, to the pipelines on the shell.

Glue code is useful for a lot of reasons: It’s quick to write, it keeps the parts on both sides separate and thus makes them easier to maintain, it creates interoperability between components, it’s simpler than to break the separate pieces or dig in their code and learn how they work, it’s even valid when you don’t have access to the source of the softwares.

On the other hand, it’s considered somewhat of a sort of duct tape programming. A quick fix that might not last. It also has the cost of performance penalties for the adaptation code and the transfer of this info, which is usually some sort of input/output mechanism.

You get the gist of it.

Let’s give some notable examples, be them text processing, or macro languages, preprocessors, etc. Mostly text processing utilities.

  • roff/nroff/troff/ditroff/groff
  • dc/bc
  • ed/ex
  • g/re/p
  • sed
  • diff
  • The C preprocessor
  • m4
  • vi
  • Makefile
  • Awk

And much more.

You’ve definitely heard of those names before, they’re very Unixy. All of them are used on the shell, in scripts to automate tasks. But it wasn’t always that way, we may take for granted those softwares, we may take for granted the idea of softwares as tools.

Where did the idea come from?

Pipes were first (second attempt) implemented in UNIX in 1972 by Doug McIlroy. The idea is simple but powerful: to allow the standard output of the program on the left side of the pipe to be the standard input of the program on the right side of it.

That was the second attempt of implementing pipes. Their first approach — which only lasted a couple of months — used “<” and “>” to define the direction.

This sounds more practical than to use other means of inter-process communication such as databases, sockets, or simply files.

After pipes were invented this lead to some room for new ways of thinking but this didn’t really happen before some event.

Doug McIlroy, the creator of pipes, still warm and happy about them, was working on a text-to-speech program and wanted to manipulate large dictionaries. The ed editor didn’t fill the task properly, it was too cumbersome. With the fresh idea of pipes in mind he then asks Ken Thompson if he could extract the regex feature from ed and make it standalone, capable of accepting input and output so that he could use it in a pipeline, it was the creation of grep.

It became obvious to McIlroy, after this small but meaningful event, that there was a useful pattern to be extracted, the one of “software tool”. An idea that was later better articulated in the book “Software Tools” by Kernighan and Plaugher.

This insightful idea became a guiding principle to build programming environments.

The software methodology that followed is the well known one, the so often heard “each tool/utility achieving its end and role internally”, the “do one thing and do it well”, the “single functionality program”, the “program as generic as possible, accepting stdin/stdout”.

This is where the concept of softwares as tools come from. This was truly a “new style of computing and thinking on how to attack problem”, from a bottom up approach. With a bunch of tools we link/combine them together to create a software. Small parts coupled to build a whole and not a big monolithic block of software.

If a feature is part of a particular environment but is useful to so many other developers then why not separate it in as its own utility. Making it fun and helpful to use by other programmers.

Not only that, there were now tools to facilitate the creation of tools, such as yacc and lex.

This general concept is a trademark features of Unix.

Not anymore: There are blown-up Unix tools (tried Solaris recently?) today while the “KISS” thing had been ported to VMS and other operating systems soon after its invention.

So let’s dive in the history and concept of some of the most popular of those tools that are used as glue code in scripts or more.

roff/nroff/troff/ditroff/groff

A utility without a manual is of no utility at all.

roff apparently was the first Unix text-formatting computer program, and the first application to run on the first machine specifically purchased to run UNIX.

But it has many predecessors, to understand it we have to go back in history.

It was a Unix version of the runoff text-formatting program from Multics, which was a descendant of RUNOFF for CTSS (Compatible Time Sharing System, a project of MIT, the Massachusetts Institute of Technology) in 1964.

The RUNOFF of the CTSS was one of the first, if not, computer text formatting program.

In the modern sense of the word, yes.

How did RUNOFF work and what did it do?

It was composed of two programs, TYPSET, a document editor, and RUNOFF, the output processor.

The role of a typesetter is to do typesetting, which is to arrange text on a page, change the font, the colors, etc. Kind of like a word processor does. TeX and LaTeX are typesetters too, markdown includes some features of typesetting though it lacks a lot of them like paginations, spaces, and text modification.

Generically you have a specific syntax or way of doing thing that applies something over the text, like spacing, bullets, colors, style, slant, size, aspect, anything.

That’s the idea.

RUNOFF supported pagination, headers, and text justification.

The name is said to have come from the phrase “I’ll run off a copy” or “run off a document” which means to print it out.

This explains why it’s such an important software, remember the move from physical typewriters to typewriters connected to computers, to glass teletypes, to raster graphics.

Typewriters are made to edit text and people wanted to keep that initial feature. Right?

Multics had its own version, runoff, that was the successor of RUNOFF for CTSS. roff (roff already existed, ported to Unix) and nroff are the successor of the Multics version.

roff already existed in a Multics version. (see history tree in the notes)

One of its important usage was for manual pages in Unix.

However, the pages were only written in runoff (the troff version) since the 4th Unix edition, though according to some account, that it’s from version 1 through 3. Some places say that the first 2 years Unix didn’t have any documentation though.

According to jkl:

Not one single manpage on Unix was written in runoff as runoff never existed on Unix. There was not even a roff command on PDP-7 UNIX. Before he did troff, Joe Ossanna (“jfo”) — double-N — implemented his own version of McIlroy’s BCPL version of Robert Morris’s implementation of RUNOFF named “roff” into V1 UNIX: man(1). http://www.tuhs.org/cgi-bin/utree.pl?fil…an1/roff.1

In all cases, they started using runoff documenation because Doug McIlroy insisted on it. And to this day this is regarded as one of the great advancement, to be able to have a manual for everything on the system. The lack of documentation on Unix is even seen as a lack of quality.

nroff -man Documents/programming/2bwm/2bwm.man -Tascii | less

The Unix version, in 1970, was a port of the BCPL version of runoff made for Multics into the PDP-7 assembly language.

When roff started being used for manpages it instantly became popular within Bell Labs patent department. It was the first Unix application used by people other than the developers themselves, real usage.

It was flexible, easily modifiable, and with real world usage. So providing features to it was an important factor to the adoption of Unix, it filled the word processing needs (of the patent department at Bell labs here). Altogether this gave credibility to the project and secured funding for the purchase of a PDP-11/45s.

When they got the PDP-11, in the late 1970, they started transliterating it to the pdp-11 assembly, this version was done in 1971.

Some say that runoff was the hook that justified the cost of getting the PDP-11.

There’s a long history of passing the code/torch, adding more and more features to runoff, polishing the code, making it more interoperable, making it work on all machines, all output types, etc.: I’ve linked in the show note a tree hierarchy of the oh so many runoff-like softwares: https://manpages.bsd.lv/history.html.

This is a noble piece of software.

In 1972 for example, Joseph Ossanna took over the PDP-11 roff and build a version for the Graphic Systems CAT Phototypesetter, which the lab just acquired. He called it troff, t-roff, for “Times roff” (Times font family was the most used), or other say for “typesetter roff”. A typesetter is sort of the ancestor of the modern printer, don’t sweat the definition, it takes some specific programs and typeset the text and print it.

troff was basically an add-on over nroff, with some #ifdefs to produce and remove specific features for the new typesetter.

Preprocessing, transforming part of the document so that they are compatible with it.

Ossanna’s troff was not so extensible, written in PDP-11 assembly and produced specific output for the CAT phototypesetter only. He started rewriting it in C to make it support multiple vendors however he died of a heart attack before finishing the work (1977).

Brian Kernighan took over the task and called it ditroff, for device independant troff. This was in 1979.

On and on, several new pre/post/processors appeared. For graphs, pics, references, etc.

This went on.

In 1989, James Clark implemented a GNU version of ditroff, called groff, which was released in 1990.

It included many features, postprocessors for character devices, postscript, TeX DVI, X Windows (as in a frontend to facilitate the construction of pipelines), etc.

runoff/roff was one important piece in the Unix arsenal even though it might not look like a software that would be so important to us now though.

dc/bc

Let’s move to other well known tools, dc and bc.

dc stands for desk calculator.

It’s the oldest surviving Unix language. It was the first program to be run on the newly acquired PDP-11, the one that they secured funding for because of the runoff success.

Depends on how you define a “language”. Wouldn’t that be B? dc is the oldest surviving interpreted language though.

According to McIlroy’s notes (p. 10), it was the first language to be run there. I could not find a source about whether it was the first program, but Ritchie suggested that it was, in fact, “one of the earliest programs to run on the PDP-11”. As they had received their PDP-11/20 in 1970 and Unix ran on it in the same year, I would want to assume that dc predates 1971. Ken Thompson said that they already had a version of it on the PDP-7. http://www.cs.dartmouth.edu/~doug/reader.pdf http://webarchive.loc.gov/all/2010050623…/hist.html

The first version was written in B, predating the C language, by Robert Morris and Lorinda Cherry, which are also the author of bc which we’ll tackle in a bit. I’m not sure at what time it was created but it was present in Unix 1st edition, so 1971 or before.

So what is dc all about?

It’s a calculator language that uses reverse-polish notation and has arbitrary precision arithmetic.

And what does that mean?

dc is first a mini-language, it has storage, registers usually single letters variables, operations, it has basic conditional expressions, and even some versions have support for macros.

It’s not your average general purpose programming language but it’s a domain specific language, it’s made specifically as a calculator.

A calculator of arbitrary precision.

What does arbitrary precision means?

It’s something we take for granted today, the way you can control the number of fractional digits, the scale aka the number of digits following the decimal point.

However, by default it’s set to 0, so setting this at the start of the program is a good idea.

So, the last part, the one we didn’t discuss yet, is the particularity of this calculator: that it uses RPN, the reverse-polish notation. What is it?

RPN, reverse polish notation, is a method of expressing a mathematical expression by putting the operands before the operator.

As in: 4 5 + would mean “keep 4 and 5 on the side then apply + to them.

This is also called a postfix notation in contrast with the infix notation, the one we commonly use where the operators are in between the operands.

The advantage is that programmatically it is easier to implement by using stacks. We push operands on the stack and when we see operators we take as many operands from the top of the stack as is needed, compute the result, and push it back on the top of the stack.

However, as you would’ve guessed, it’s not as intuitive to use.

So here comes bc.

bc stands for basic or bench calculator and it’s the infix-notation version of dc. It is more user-friendly, with a syntax resembling the C programming language.

It was introduced in 1975, in Unix V6, also by Robert Morris and Lorinda Cherry, the same author of dc. It was implemented as a wrapper/frontend over dc. A simple, hundred line compiler in Yacc, converted the bc like into dc’s postfix notation and then piped the results through dc.

The conditions and variables, and all the stuffs are easier to write in bc.

Later on, in 1991, POSIX standardized bc, there exist today 2 implementations of the standard: the front-end version implemented in Plan 9 and the GNU bc version, though, I’m sure there are others.

The GNU bc version, released in 1991, is no longer being a front-end to dc and has many extensions and new features, like not being limited to having variables, arrays and function names limited to one character only.

So let’s discuss exactly that, the usage of both.

dc is quite straight forward when it comes to usage.

You type the RPN directly and the terms gets added to a stack in the program. To print the result, aka the value on the top of the stack, you can type ‘p’, to quit you can use ‘q’. The interface is ed-like.

dc also offers one letter variables called registers. They are a second place of storage other than the stack of the RPN. To store the value that is on top of the stack in the ‘c’ register you can use ‘sc’ and to retrieve it, to put it on the top of the stack, you can use ‘lc’.

As with bc, if you start it, it’ll be in interactive mode like dc, you type like you’d normally type your usual calculation in C: (1 + 4) * 2 and it’ll output 10 directly. No need for a ‘p’ command if you don’t assign the value to a variable.

bc has conditions and loops too, with a similar syntax as in C, dc has conditions, but they’re kind of ugly to write once the expression becomes big enough.

As we said, the precision of both calculators is by default 0. To set it in dc you push the scale you want into the stack and pop it using the ‘k’ command which will set it as the scale. For bc you assign the scale to the scale variable or you can start bc with the ‘-l’ flag which will set the number of decimal point to 20 and will also enable some mathematical functions in the language. sqrt(), square root is POSIX bc’s only built-in math function otherwise.

With -l you get:

s(x) sine c(x) cosine a(x) arctangent l(x) natural logarithm e(x) exponential function j(n,x) bessel function

Example:

$ bc -lq
scale=10000
4*a(1) # The atan of 1 is 45 degrees, which is pi/4 in radians.
# This may take several minutes to calculate.

There’s a lot more features and you can dig more into those languages. Let’s just mention that there are some subtle difference between C syntax and bc.

()[]{}>=< etc.. Act exactly like the C operators. % like C but depends on scale variable. ^ ^= resembling C XOR but is in fact exponent.

Now there’s an interesting backstory I’d like to mention…

Robert Morris is also an expert cryptographer who later worked for the NSA. His son Robert Tappan Morris is the author of the Morris worm, the first worm on the internet and the first person convicted under the then-new Computer Fraud and Abuse Act. He also founded a company with Paul Graham, of Y Combinator, which you might have heard through The Hackernews.

The world is small isn’t it, and I find this ironic considering that for some time dc was used as a memento for RSA encryption and the controversy against crypto export in the USA and Canada. You can find some links about it in the show notes.

Compute: g ^ e % m
gen ^ exp % mod

#!/usr/bin/perl -- -export-a-crypto-system-sig Diffie-Hellman-2-lines
($g,$e,$m)=@ARGV,$m||die"$0 gen exp mod\n";
print`echo "16dio1[d2%Sa2/d0<X+d*La1=z\U$m%0]SX$e"[$g*]\EszlXx+p|dc`

ed/ex

Another classic is ed.

ed is a line editor aka the “standard editor”. It was one of the first part of the Unix OS that was developed in August 1969. It is also part of the POSIX standards as one of the essential tools.

It was developed early on by Ken Thompson on the PDP-7 as a mandatory element of the OS, it being the editor that goes along with the assembler and the shell.

Many features came from qed, a piece of software that originated from Berkeley university, which is the university Thompson studied at which made him very familiar with it. He even reimplemented it on the CTSS and also on Multics with a BCPL version.

His big addition to the mix was regular expressions, which is also a big feature of ed.

QED stands for “quick editor” and it is also a line-oriented text editor. It was developed at Berkely in 1965 for the Timesharing System running on the SDS 940.

QED was not only influential for Thompson alone, it also influenced Dennis Ritchie and Brian Kernighan and the whole team, they wrote the QED manuals used at Bell Labs.

So they were well familiar with qed.

Dennis Ritchie later on produces a version of ed that Doug McIlroy described as the “definitive” ed. I can’t find much about it but I think it was a C rewriting.

There were quite some “eds” involved. http://web.mit.edu/kolya/misc/txt/editors

So why the line-oriented text editor, what does that mean?

Line-oriented, a line by line editor, actions over lines, very succinct in the output, is what is meant by line-oriented, which is contrasted with editor such as vim or emacs that are more visual.

To understand this decision we have to realize the type of machines they were using to interact with the OS: typewriters. If you’ve listened to the podcast about terminals you might have a clue to what they are, they are very simple terminals that use a typewriter, a keyboard and paper that is printed on.

This kind of interaction is slow and so interactive editors are not especially good with them. But it is good for line by line editing and scripting.

So, ed usage is very linear, as you would’ve imagined but what does it actually look like.

You specify a file to ed on the command line so that it opens it and then you’re faced with a prompt, like the shell, you give it commands to execute on lines and some of the commands return outputs on the screen directly.

You don’t see the entire file as you are editing it. You specify a range of lines, comma separated and an operation to perform over them. For example “1,5l” will display lines 1 through 5, and “,l” will print the whole file.

If you are familiar with vim then this shouldn’t be surprising because most of it is included in it through vi and ex inspiration, we’ll talk about those in another part.

Here are some commands:

l to print, range before 1,3 or just , for all
i for insert
a for append
w for write
q for quit
g to go to the top
G to go to the end of the file
d to delete
g/re/something/
s/something/somethingelse/g

As you could’ve guessed ed has become famous or more infamous for its terseness, it has even been called the “most user-hostile editor ever created”. Probably because according to today’s standard the lack of visual feedbacks is regarded as backward-minded.

For example, the message that ed will produce in case of error, or when it wants to make sure the user wishes to quit without saving, is “?”. Some older versions didn’t even ask for confirmation when quitting.

ed doesn’t do anything unless it’s requested to do it or display it.

So why should you use ed, and where, today in a world of editors with visual feedback?

  • Available on essentially all Unix systems (and mandatory on systems conforming to the Single Unix Specification).

  • Inside a modal editor supporting command mode, text mode and viewing mode support for regular expressions

  • Powerful automation can be achieved by feeding commands from standard input. In scripts for example.

The ED style of interface, the “ed pattern”, or maybe we have to put it as the QED pattern, has inspired many more descendants.

Many other programs use this type of interface, for example gdb.

The pattern sometimes comes with a downside, it’s not always as easily scriptable.

ex, the Extended ed, is also a line editor, written by Bill Joy in 1976 as a more friendly version of ed, or more precisely a modified version of George Coulouris improved ed called ‘em’, that took advantages of video terminals, which the Bell Labs didn’t have.

Coulouris considered ed to be way too hard to use, only suitable for “immortals” as he put it, thus he modified it and created the ‘em’ editor, which was specifically designed for display terminals and was also single-line-at-a-time visual editor. It was arguably one of the first program to use extensively the “raw terminal input mode” bypassing the terminal line-discipline, check the podcast about terminals to know more about this topic. Anyway, Bill Joy liked it and modified his version further to create ex.

Then a full-screen visual interface was added to ex which turned it into the vi text editor. Thus, ex is a part or mode of vi.

ed also influenced the creation of grep and sed, we’ll talk about them separately later.

So it is more or less true that ed doesn’t have much practical use today as an interactive program, that most of its cool features are present in other places and not on its own. Still, it’s nice to learn how it’s an integral part of many other tools.

vi

Let’s specifically talk about vi now.

vi, unlike ed, is a screen oriented text editor. It was written by Bill Joy in 1976, as we said, and it’s part of POSIX.

It was made as the visual mode for the ex line editor, and so the first few releases within BSD were also named ex and in May 1979 the editor was installed under the name “vi”, short for visual, and took the user directly in the visual mode of ex.

However, the vi we use today isn’t really a direct descendant of this one, there’s way more history behind it.

As we said, Bill Joy and friends were inspired by Coulouris’ ‘em’ and extended it to create ‘ex’ then added a full-screen visual mode to ex and there was ‘vi’.

As with all the work that was done, it was all a superposition of ideas and code, one over the other. The same goes for the visual mode, it was inspired by the Bravo editor, which is said to be the first WYSIWYG editor. Not surprisingly Bravo was made at Xerox PARC in 1974.

Many ideas were taken from Bravo, for example the dot command is the double-escape from Bravo, and for those not in the know it’s a redo command.

So what kicked off vi?

Joy created the BSD Unix in March 1978, which gave him the option to ship freely his new editor, ex and then vi as we said, within it. It gave it exposure, and because at the time there weren’t that many editors that came with Unix, basically only ed, it gave rise to the success of vi.

On the other side Emacs, which the version at the time wasn’t a free as in free beer, would cost hundreds of dollars, and to be clear we’re not talking about GNU Emacs, which didn’t appear before 1985, but an earlier version also called Gosling Emacs or Unipress Emacs.

Gosling Emacs “didn’t appear” before 1981, so, in fact, the expensive versions of Emacs were not a thing when vi was published. 1978 already had (versions of) TECO, Emacs’s birthplace, as well as Stallman’s original MIT Emacs itself which was available at no cost either - we had the topic “everything was free software once” in a previous discussion. Here is a list of free Emacsen. http://www.finseth.com/emacs.html#25

I digress, it was free, at least within Unix, as we’ll see in a bit.

We mentioned that earlier that vi was made as an update to ex, but it was actually a hardlink to ex’s visual mode (ex 2.0) that started to be shipped within BSD2 because everyone that was using ex was using its visual mode more than anything.

Which brings the question: is vi ex? Yes in a sense until it got upgraded specifically for its visual mode only, with that new way of thinking first visual. Same goes for all the previous projects we talked about in this podcast, they are all work upon work upon work. As people learned new techniques, even though those were not always clean and elegant and pompous, people just tried implementing new things over the work of others.

Bill Joy implemented the visual mode because he thought it was primarily useful for others. These days the attraction to vi stems from how lightweight and efficient it is, however in its first iteration vi was a very large program it could barely fit in the memory of the PDP-11/70. Even the version 3.1 shipped in BSD3 in 1979 couldn’t fit any more within the memory of a PDP-11.

Now back to what I said “it was free, at least within Unix”.

That is because it was only available through commercial Unix vendors such as Sun, HP, DEC, and IBM which included the code until vi 3.7 with some of their own customizations in their OS, namely Solaris, HP-UX, Tru64 UNIX, and AIX.

The version included in those OS descends directly from vi, though with modifications.

I mention this because this isn’t the case everywhere.

ex and vi being based on ed, were burdened by the AT&T licensing, and thus people looking for a free editor would have to use something else.

For example Minix created (published under it) a vi clone called Elvis.

Tanenbaum added the third-party Elvis editor to Minix — it was not created “by Minix” although its creator published it to the Minix newsgroup first. https://groups.google.com/forum/#!origin…lQiUR7aeYJ

Remember the whole war that AT&T lead against the BSDs. It was not until BSD Unix started relicensing its code that they realized the issue, first in 1992 in their 386BSD they used Elvis as a replacement for vi, then in BSD 4.4 Lite, in 1994, Elvis was used as a starting point to create nvi a more one to one correspondence with vi. This is the version of vi that is used today in FreeBSD and NetBSD.

It’s only recently in 2002 that the original version of vi were allowed to be distributed legally. Which is sort of ironic that the BSD, which were created by Bill Joy don’t include the initial version of vi that he wrote.

So what’s so special in vi usage that separate it from ed.

First, vi is modal, that means it can work in different modes, there’s a mode to access the ex features, a mode for applying commands, normal mode, there’s a mode for direct editing, insert mode, and others.

It also has the idea of combining single letters keys to create a full command to be applied over text.

For example, in vi, you can type cw<replacement text>Esc and it would mean “change a word by “replacement text”. And the operation can be repeated by the redo command we mentioned, the dot.

There are many critics of this way of interaction. That there are too many single characters commands and that they are hard to remember.

Though arguably those were single characters because the connection of the terminal was slow and it needed to be efficient rendering wise, but not memory wise which is fun to think about.

Another critic is that it is lacking mouse interaction, feedback when switching mode, and multiple undo levels. Which is all because internally vi is just ex with a nicer UI.

The last thing to tackle with vi is the origin of the iconic HJKL and the usage of escape. Those keys were chosen because on the ADM-3A terminal, the one Joy used, the escape key was at the location where tab is these days, which is a good location, and the hjkl keys served as cursor movements because on the keyboard those were printed with arrow keys on them and the ADM-3A had no cursor keys to begin with.

A nota bene here is that you can use control sequences to simulate escape, ctrl-[ does the trick.

g/re/p

Back in time again, with grep. We talked a bit about its history earlier so let’s get back to it.

grep is simple, it’s one of the most commonly used command on Unix. It’s a command that searches the input file for a pattern, using regular expressions, and prints the lines that contains it.

This is it, end of the line, nothing more.

Grep, as we said was created by Ken Thompson in March 1973, soon after the implementation of pipes, as a standalone application based on the request from McIlroy so that he would get ed matching facilities outside of ed.

The name is literally the command that were initially used inside ed, g/re/p, globally search a regular expression and print it.

g is for global, the /regular expression for the search/ , and p to print the results.

If you’re familiar with vi it is the same command, as we mentioned earlier. For example s/test/test2/g would replace all global occurrence of the word “test”.

With ex for example you would do:

ex -c 'g/test/p' file

And with grep simply grep 'test' file.

However, grep didn’t directly get included as a tool for Unix. According to McIlroy he kept it as a personal tool for a while, until he finally decided to make it public.

From what I could see the tool official release was only 7 to 8 months in November after its creation in March. However, I’m not sure of my sources, and I’m not sure if it was released internally way earlier.

He was parsing a dictionary in a horrible inefficient way for a project and once he got his hand on this grep it worked like a charm in a chain of piped programs.

It became such as useful tool and lead to the start of the understanding of the concept of software as tools. It is often stated as the “prototypical software tool” that is credited with “irrevocably ingraining” Thompson’s tools philosophy in Unix.

As with others, the software morphed and diverged over time.

Al Aho also a researcher at Bell Labs and the co-author of AWK wrote egrep and fgrep one weekend in 1975, and they got introduced in Unix V7.

egrep for extended regex using Alfred Aho. fgrep for a list of fixed strings using Aho-Corasick string matching algo.

Historically, grep and egrep took turns being faster than the other.

The modern versions within GNU and BSD have deprecated egrep and fgrep, they are now aliased to grep. According to POSIX grep should have the -E and -F switches for compatibility.

Back to the topic. grep is so iconic that it is used by everyone as a word to indicate searching for something.

So much that in December 2003, the Oxford English Dictionary Online added draft entries for “grep” as both a noun and a verb. I’m not sure if it has now been officially included, it seems like it. There’s nothing indicating that it’s a draft.

Last thing to mention is also related to the massive influence it got. As an example there’s a lot of tools with the word grep in them, the pgrep utility that displays the processes whose names match a given regular expression.

It got something started within the community, the creation of mini tools that did one thing. The next one is exactly that, we’re going to tackle sed.

sed

It wasn’t long before the creation of grep lead to the invention of other special purpose tools.

The next ideas were kind of obvious, a grep-like tool but for substitution, a so called gres, a gred for deletion, a grea for appending, and so on… There seemed to be no end to this program as tool thing. And that’s when Lee McMahon saw that, in 1973, same year as before, he decided to merge all those concepts into one single tool, a sort of ed but that could be used on the pipeline, he called it sed for “stream editor”, to edit right in the pipeline, in the stream.

sed is more or less a literal ed in the pipeline, thus its language is very similar to the one of ed. ed was kind of universal at the time, so it was common knowledge and intuitive to use sed.

The power of sed comes from how it allows to edit in the middle of a pipeline passthrough and the flexibility and complex kind of editing and pattern matching it allows. This is the epitome of the middle-man, glue scripting, we talked about, it can easily adapt its input to fit the output and it does that really fast. It’s on par with languages like awk or perl for that matter. It’s even Turing complete.

It was a dream come true for programmers that were using slow teletype terminals and had to do basic editing. And it still is a key tool today to edit huge files in a single pass through instead of using an interactive editor.

It does that by reading the file line by line into an internal buffer called the pattern space, then apply operations on this line, outputs the modified line if it needs to, and continue the cycle. It is quite efficient.

However, you’ll rarely see sed today being used to its full potential, all you’ll see is its simple replace feature. While in fact there are more than 20 other commands available in sed language.

It’s a full Turing complete language with two variables (only), the pattern space we talked about, and the hold space, a sort of storage, and it has conditions via a sort of GOTO-like branching (using label, colon + string, and using the b command for branching to this label).

So how is it used.

You call it on the command line ‘sed’, followed by the sed script file passed as argument (with -f file or without the -e expression option) or the script written directly on the pipeline. It can accept a file as input or a stream, as we said, as input.

The important part is the expression syntax. I won’t list all the commands here but I can mention some.

For example, s is for substitution, it is followed by a slash separating the pattern match (regex) and the string to substitute the match with. d is used for deletion. i for insertion. a for appending. r to append the text from a specific file to the current one. y to transliterate from one character set to the other, and much more.

About sed’s ed compatibility: ed does not limit you to the slash character: sxaxe or swawe, whatever works for you, replaces “a” by “e”. sed does not support that.

Commands in sed may also take optional address so that you can specify where to apply them, it could be a line number or a regular expression.

For example 2d would delete only the second input line and print all the others, while /^ /d would delete all lines beginning with a space.

You can see why for a long time no one saw a utility for the ‘head’ tool and instead intuitively used sed 10q instead, which print the first 10 lines and then apply the ‘q’ for quit command.

Today we even have more powerful sed versions available, arguably and probably a bit slower though.

For instance, GNU sed added several new features, including in-place editing of files.

The original sed is sort of lost in history, though Eric Raymond reverse engineered it, probably in 1995, and released it as minised. Which was the default sed for a while instead of GNU sed, and is also the default sed in Minix.

sed was the big inspiration for the next mini-language we’re going to discuss, AWK. Which in itself was the inspiration for Perl. Everyone influences everyone else.

Awk

AWK stands for the initials of its creators: Alfred Aho, if you remember it’s the guy who implemented the extended regex grep version “egrep”, Peter Weinberger, and Brian Kernighan.

It was made in 1977, 4 years after the previous tools.

In 1985, a new version, called nawk or new-awk, made the programming language more powerful, now letting the user define his own functions, adding multiple input streams, and computed regular expressions. This version was popularized within Unix System V released in 1987.

As with the other tools, AWK is in the POSIX specifications standards.

AWK, like sed, is a programming language designed for text editing. However, unlike sed it isn’t mainly for editing streams, it specializes in a more data-driven approach, it is more for extraction and transformation than editing.

As with sed it also both act on input streams and files.

To understand AWK you have to get the generic usage. It takes a pattern (regex) for matching and an action to extract and transform the data from the lines matching the pattern. But you can omit the pattern or the action, and leave only one of both, or the pattern alone or the action.

awk /pattern/{action}

The extraction mechanism automatically understand separators and format. By default, it split the input by columns which you can act upon via variables that get assigned directly.

The action part of AWK is a full language, it includes strings and associative arrays. It’s actually the language that first popularized the usage of associative arrays, arrays in which the index keys are strings.

The associative array allows to easily generate and report information over huge amount of data, accumulating information on each stream parsing.

The language itself is easier to read than sed as it’s C-like.

As with other tools there are newer versions of them.

GAWK was written between 1985-1988 by Paul Rubin with advice from Richard Stallman. If you remember NAWK was released 1985-1987, so the releases were close in time.

GAWK was, like any other tool, extended by even more people afterward.

In May 1997 it got even more extended by Jurgen Kahrs, which addded network access features to gawk.

Also, as with sed, vi, and others, the original version of the awk tool was kept closed behind doors until Brian Kernighan’s nawk (New AWK) source was publicly released in the late 1990s; This version is used by many BSD systems to avoid the GPL license.

Makefile

The last tool we’re going to talk about is make and the makefile. It isn’t really a glue tool, but it’s an important one for scripting.

Make is a build automation tool, more often but not limited to create executable programs and libraries from source code by reading a file named Makefile.

The make utility is used to interpret the Makefile.

More precisely it’s a dependency-tracking build utility. It knows when the timestamp of all the dependencies changes and act accordingly, doing the minimum amount of work necessary, rebuilding only what depends on those updated files.

Historically it was created in April 1976 by Stuart Feldman at Bell labs, and included in Unix starting with the first edition of PWB/UNIX (programmer’s workbench).

Feldman got the idea of make when one of his coworker was having trouble debugging a program but didn’t realize that the executable wasn’t updated with the changes he made to the code, thus rendering the changes useless, and wasting time chasing invisible bugs.

This person that was debugging was Steve Johnson, the author of YACC, a parser generator, that was used for a bunch of other tools that we mentioned.

Thus, Feldman thought of building a dependency analyzer but came up with something much simpler, and created the make utility over a weekend.

The makefile language parsing was made using Johnson’s YACC.

This was great because before make they used separate shell scripts accompanying source code to build projects. Including all the abstract target dependencies and tracking them in a single file was a big move forward.

So this makefile list the dependencies but what’s in a makefile.

Makefiles are relatively simple. They are a set of directives, targets, dependencies, and command accompanying them.

Concretely it contains five kinds of things: explicit rules, implicit rules, variable definitions, directives, and comments.

A comment is anything that is after the ‘#’ character.

An explicit rule is a label, a target, it lists other files or targets that depends on it, that are prerequisites to it. Make will rebuild this target if any of the dependencies changes.

The implicit rule is similar but is for a class of files based on their names, for example you could create a rule for any c header file.

The variable definition is simple assignment, like in the shell. It can be used and substituted later.

Building, or the directive, means executing the command or instruction that is under the label. But those directives should be tab indented.

There are two critics to this. One each command is executed in a separate shell and this is system dependant. Thus, not so portable. Secondly, the tab is a whitespace character, and thus could mislead the users. Making the tab character visible solves this.

The tab was used because yacc was new and the author wasn’t accustomed to it, he was just learning to create something useful out of it. It worked and spread too fast to revert the changes.

Anyway, this is as far as it goes, you got targets, dependencies for it, and commands to execute after the dependencies/prerequisites have been fulfilled.

target: dependencies
system command(s)

Or separated by semicolon

targets : prerequisites ; command

You can see that this is not limited to C programming, contrary to what some may think. It’s a valid build system for anything that has dependencies or that needs changes to be tracked. They’re a good way to bundle up little procedures together, to script, for example when making a distribution package.

Usage wise make is invoked on the command line with make target. It uses the Makefile named Makefile by default but it could have any other name.

Without argument, it’ll build the first target that appears in the file. And one thing to note here is that the targets don’t need to be in any specific order in the Makefile, make will understand the dependencies between them.

Today there are ton of other build systems, which I won’t cover. Imake, autoconf, cmake, automake, gnu make, etc..

All have their advantages and disadvantages.

Conclusion

You can probably guess what kind of conclusion I’m going to give to this podcast. You realized after two of those tools. They’re all linked through history, they all learned from the one before. And this is probably totally unrelated to the topic but it’s the most apparent relation, everything is a remix, it’s great. It made me respect those humble tools.

And there are much more that I didn’t mention.

After learning most of this history I think the concept of the tool as a glue code is less about gluing code and more about solving problems in efficient and human ways. The pipeline has its place there but probably not the skewed extremist view of the Unix philosophy as minimalist radicalism we kind of see in some places today.

So get your pipeline ready, get your scripts ready, think of how to improve them, try all the tools mentioned and others, and more importantly have fun.

References

Scripting or Glue-coding:

roff:

bc/dc:

ed/ex:

vi:

g/re/p:

sed:

awk:

Make:

Music

Catmosphere - Candy-Coloured Sky: https://soundcloud.com/argofox/catmosphere-candy-coloured-sky?in=photochic16/sets/creative-commons-license




If you want to have a more in depth discussion I'm always available by email or irc. We can discuss and argue about what you like and dislike, about new ideas to consider, opinions, etc..
If you don't feel like "having a discussion" or are intimidated by emails then you can simply say something small in the comment sections below and/or share it with your friends.