Retired Techie

Getting older, not necessarily wiser!

Streams, Redirects, and Pipes

Published on September 5, 2021 at 7:13 am by LEW

Introduction

I was contemplating a post about terminal use when I realized streams, redirects, and pipes were an underlying feature that I was taking for granted. That Operating System (OS) use standard data streams for inputs and outputs with various commands. These streams can be Piped or Redirected to different inputs and outputs. Available in most OS’s (yes, even Windows), in this article I will focus on the Linux command line.

Depending on what you are doing with the command line, you may not need to know this. If you are a casual terminal user, then the odds are good you will not need this. On the other hand, if you work with the command line on a regular basis, then having an understanding of standard streams, redirects, and pipes can have a marked effect on your experience and efficiency.

Overview

Most commands used in the terminal need two items to do something. A third item is not required for basic functionality, but is really really nice to have.

• Most commands do not exist in a vacuum, so they need to get information from somewhere to perform their function. This is referred to as Standard In (abbreviated stdin). In most cases this will default to the keyboard.
• Most commands also need a way to communicate results back to the user while performing their function. This is referred to as Standard Out (abbreviated stdout). In most cases this will default to the display
• It is really helpful if a command has a way to alert the user if there is an issue or problem encountered while executing their function. This is referred to as Standard Error (abbreviated stderr). In most cases this will also default to the display.

Note that this is an expected behavior, not a requirement. Commands in Linux are written by many different people with their own ideas about how terminal command should function. And there are also some use cases where the above standard may not be totally applicable, or even make sense. So while many terminal commands you use will conform, don’t be surprised if some don’t.

Using stdin, stdout, and stderr

Back when I started using computers, stdin could be a paper punch tape, punch cards, or a teletype keyboard, and stdout and stderr was the paper roll on a teletype. On most modern systems running a terminal, stdin is the keyboard and stdout/stderr is the screen.

For example, the cat (short for concatenate) is used to send output to the screen (stdout). If we type it into a terminal by itself and hit return, we are staring at a blank line. If we type “test” and press return, we imminently see it echoed back to the screen. Since the keyboard is stdin, the cat command will echo anything we type back to the screen. (stdin → stdout). You can type control + c (ctrl c) simultaneously to exit the command.

LW@D11:~$ cat
test
test
^C
LW@D11:~$

Another example using ls (list) command. Typing ls by itself will list the content of your current directory to the screen. Using stdin we can type ls /usr/share to list the content of the /usr/share (stdout) directory. If we type ls /usr/foobar, where /usr/foobar is a non existent directory (stdin), we do not see any stdout on the screen. Instead we see an error message (stderr) about “no such file or directory”.

LW@D11:~$ ls /usr/foobar
ls: cannot access '/usr/foobar': No such file or directory
LW@D11:~$

Using Redirects

It is all well and good to get stdin, stdout, and stderr from the keyboard and screen. However there are times when we want to use different inputs and outputs. This process is called redirection.

Redirection can get quite complicated. But for most common needs, only a few specifics will normally be used. Basically redirection uses the “<” and “>” symbols to change default targets. When using redirects, stdin and stdout are automatically assumed. The stderr stream needs to be specified. However, if desired, you can specify both stdin and stdout.

0: stdin  0< or <
1: stdout 1> or >
2: stderr 2>

My intent was to provide an example of redirecting stdin here. Try as I might I could not come up with a simple non trivial example for use. All the cases where I have actually used stdin redirect were somewhat complex. If anyone has a good simple and practical example, please send it to me.

Basically the form would be wither “command < source” or “command 0< source”. Where command is the shell command and source is where the input for that command should come from. This will maybe make a little more sense once we talk about stdout redirects.

Lets use the cat (short for concatenate) command to demonstrate redirecting stdout. Instead of writing the output to the screen, you might want to write output to a file for later review. In this case we will redirect the output from the screen to a file. To do this we can implicitly use cat test1 > testresult.txt. Or we could also  explicitly do it with cat test1 1> testresult.txt.

LW@D11:~$ cat test1 > testresult.txt
LW@D11:~$

Lets say we have multiple files generated by a program we are testing. They are named test1, test2, test3, and test4. The program will overwrite these files when run again. So we want to save them to a single file marked with a date. Using cat test1 test3 test4 will write the content of each file to the screen. If we want to save the output, we can redirect it to a file instead by using cat test1 test3 test4 > testresult.txt. Now all the data is in testresult,txt file.

In case of errors, we may want to redirect stderr also. Lets assume that the file test2 does not exist. We would type cat test1 test2 test3 test4 > testresult.txt 2>testerror.txt. Remember stderr must be explicitly called. If you look at the newly created file testerror.txt, it will contain an error message about test2 not existing.

One additional thing I want to cover for redirects. The cases we have looked at will either create a new file or overwrite an existing file. If we do not want to overwrite a file we can append to a file instead. If we change the stdout redirect in this command cat testfile1 > testresult.txt to look like cat testfile >> testresult.txt (using >> instead of >) we will append to our file rather than overwriting it.

Using Pipes

The pipe in Linux is a type of redirect shortcut between commands. It uses the “|“character in the command line.It basically takes the stdout from one command and directs (pipes) it to the stdin of the next command. A example of this is using the ls command when there more entries in a directory than can be seen on the screen, and you have some idea of what you are looking for.

For example, entering ls /dev will list the content of the /dev directory where device files live. Unless you have a large screen, this will scroll. If we are looking for a hard drive/ USB drive entry, we know they are all sd?? (the first drive is sda, and the first s partition of the first hard drive is sda1). To clear the clutter we will pipe the output of the ls command output into the grep command (line filter) input with the filter parameter “sd”. This should filter out everything but the hard drive entries.

LW@D11:~$ ls /dev | grep sd
sda
sda1
sda2
sda5
LW@D11:~$

You can also redirect the grep output from the screen to a file with a redirect.

LW@D11:~$ ls /dev | grep sd > DrivesList.txt
LW@D11:~$

Conclusion

This has been a brief high level look at streams, redirects, and pipes. The intent was to provide enough information for the average user to both understand basic usage, and recognize them when they see them in examples. It was not intended to be an in depth discussion. There are other sites more suited to that.

What we covered;

• The terms stdin, stdout, and stderr
• Usage of the redirect symbols “<”, “>”, “2>”, and “>>”
• The Pipe symbol “|”

You should now be able to recognize them in examples, and stat using them yourself to improve your command line experience.

Copyright © 2021-2024 Retired Techie All rights reserved.