Exercise 9 of 10 · Pipes and filters

Birdwatching Tally

What you will make

A summary of a birdwatching log: total sightings, number of species, number of robins, and a tally of every bird from most to least seen, each answered by a short pipeline.

The one new idea: Pipes chain sort, uniq -c, grep -c and wc -l into a tally

This is the shell's signature move: small tools that each do one thing, joined by pipes into an answer none of them could give alone. sort | uniq -c is the classic tally idiom, grep -c and wc -l are the everyday counters, and the difference between grep and grep -c, or between uniq on sorted and unsorted input, is the kind of thing you only learn by seeing it go wrong.

Go straight to the code ↓

Chaining small tools

A pipe, |, sends the output of one command into the next as its input. Each tool does one small job, and the pipe lets you line up several to answer a question none of them could answer alone.

The tools in this exercise:

sort puts lines in order. With -u it also drops duplicates; with -rn it sorts as numbers, largest first.
uniq -c collapses runs of identical adjacent lines and prefixes each with a count. Adjacent is the key word — uniq only looks at neighbours, so unsorted input gives a wrong tally. Always sort first.
grep -c word counts the lines containing word. Without -c, grep prints the matching lines themselves.
wc -l counts lines.

The while read loop from the previous exercise works on a pipe just as well as on a file: put it after the last | and it receives each line. Because read splits a line into words, read -r count bird pulls the count and the name apart for you.

A worked example

Shell

cat > fruit.txt <<'END'
pear
apple
pear
END
echo "Kinds: $(sort -u fruit.txt | wc -l)"
echo "Pears: $(grep -c pear fruit.txt)"
sort fruit.txt | uniq -c | while read -r n f; do
  echo "$f: $n"
done

Output

Kinds: 2
Pears: 2
apple: 1
pear: 2

The cat > fruit.txt <<'END' block is a here-document: everything up to the line END is written into the file. Then three pipelines answer three questions.

Your turn

The editor holds a birdwatching log with six sightings, and four summaries: how many sightings, how many species, how many robins, and a tally of each bird from most to least seen.

Three lines are marked. The species count uses sort without -u, so duplicates are still counted. The robin count uses grep without -c, so it prints the matching lines instead of a number. And the tally runs uniq -c straight on the unsorted file, so the same bird shows up several times with a count of 1 each.

Add -u, add -c, and put a sort in front of uniq -c, then run it.

If something goes wrong

If "Robins:" is followed by the word robin on several lines, grep is printing matches rather than counting them. Add -c.

If the tally lists the same bird more than once, uniq is receiving unsorted lines. The pipeline should be sort file | uniq -c | sort -rn | while ....

If the species count equals the sightings count, sort needs its -u so that each species is counted once.

Write your code

Runs in your browser. Press Run (or Ctrl/Cmd+Enter) and the output is checked for you.

Ctrl/Cmd+Enter to run

Press Esc then Tab to move keyboard focus out of the code editor.

Ready

Output will appear here after you run your code.

The runtime is starting in the background. You can type now — it will be ready before you are.

The answer appears here once you have run your code at least once.

Things that often go wrong here

Running uniq on unsorted input: uniq compares each line only with the one before it. Unless identical lines are adjacent, they are not merged, so the tally repeats birds with a count of 1. sort first, then uniq -c.
Using grep instead of grep -c to count: Plain grep prints every matching line. Inside $(...) those lines all end up in the echo, spread over several output lines. -c makes grep print a single number, the count of matching lines.
Counting species with sort | wc -l instead of sort -u | wc -l: Sorting does not remove anything, so the count stays at six. The -u flag is what keeps just one copy of each line before wc counts them.
Piping into the loop and then also adding < sightings.txt after done: A loop can take its input from a pipe or from a file redirect, not both. If you add < sightings.txt after done, the pipe is ignored and the loop reads the raw file, splitting each bird name into count with an empty bird.

Want a blank editor instead? Open the Bash & the Shell playground.