Linux/UNIX Pipes File Descriptors, and Process Substitution
- POSIX Shell – A logical container providing an environment and a set of commands to interact with a UNIX and/or UNIX-like system.
Examples include: sh(Bourne shell), bash, ash, csh, etc. - Pipe – a unidirectional connection between two processes utilized for inter-process communications.
- File Descriptor (fd) – A dynamically created, locally significant and non-persistent file representation of a FIFO buffer which can be connected to a pipe, or by default to the parent process’ standard input, output, and error buffers. As an example, a (virtual) terminal’s input device (keyboard, mouse, etc.) and output device (display).
- Process substitution – Allows the POSIX shell user the ability create scripted constructs that mimic functional and/or imperative programming syntax in-line.
Every interactive process on startup within a POSIX shell is attached dynamically to three pipes local to its memory space. Standard Input(0), Standard Output(1), and Standard Error(2). If the POSIX system supports file descriptors they are then created and attached to the pipes at ‘/dev/fd/’.
Name | Abbrev. | File | Symbols |
standard input | stdin | /dev/fd/0 | < or << |
standard output | stdout | /dev/fd/1 | 1> or 1>> |
standard error | stderr | /dev/fd/2 | 2> or 2>> |
POSIX Shell Plumbing
The POSIX (Bourne) shell and its derivatives provides the ability to utilize various symbols to plumb processes together by their file descriptors.
Symbol Name | _Symbol_Syntax_ | ________Function________ |
pipe | { block } | { block } | attach stdout of left script block to stdin of right script block |
success exec |
cmd && { block } | if cmd exits with success (0) then execute block |
failure exec |
cmd || { block } | if cmd exits with error (not 0) then execute block |
write file |
{ block } #>file | attach /dev/fd/# of left script block to file and overwrite with bytestream |
write fd |
{ block } #>&? | attach /dev/fd/# of left script block to /dev/fd/? and overwrite with bytestream |
append | { block } #>>file | attach /dev/fd/# of left script block to file and append bytestream |
close stdout |
#>&- | close /dev/fd/# (no # specified defaults to stdout) |
read file |
{ block } <file | attach stdin of left script block to file and read bytestream |
read until |
{ block } <&? | attach stdin of left script block to /dev/fd/? and read bytstream |
read until |
{ block } <<word | attach stdin of left script block to stdin of proc and read until word |
close stdin | <&- | close stdin |
Process Substitution
The POSIX shell also provides the ability to perform inline substitution utilizing two methods:
_Symbol_ Name |
Symbol_Syntax | ________Function________ |
sub-shell exec | ( block ) | execute process proc without environment persistance |
sub-block exec | { block; } | execute proc resulting inline at location within script with evnironment persistance |
…for examples see below
Examples
Basic Pipeline
find / | grep '^\/bin' | sed -r 's/^/\/usr/g;'
This is a very basic example showing the connection of stdout(1) of the find utility to the stdin(0) of the grep utility to filter all paths that start with ‘/bin’ and prepending the paths with ‘/usr’. The intent is to translate the output of find with all ‘/bin’ paths to ‘/usr/bin’.
Dynamic File Descriptors
$ FOO=who # SET VARIABLE FOO TO who $ cat <( # READ FD GENERATED BY <( ... ) FOO=bar ; # SET LOCAL VARIABLE FOO TO bar echo $FOO ; # PRINT LOCAL VARIABLE FOO ) # DESTROY ENVIRONMENT bar $ echo $FOO # PRINT VARIABLE IN OUTER CONTEXT who? $ echo <( FOO=bar ; echo $FOO ) # PRINT FD AS FILE PATH /dev/fd/63
The above is a great example that highlights what is returned by the <( … ) is actually a file path and not the contents. It is the responsibility of the script block, command, or executable to the left to read the contents of this file descriptor. It also shows how the contents within ( … ) are executed in a completely different context preventing access to variables in the executing script and vice versa.
Local execution of a Script generated on a Remote Server
$ /bin/sh <( curl http://www.example.com/getScriptToRun.sh )
Executes cURL retrieving contents generated by ‘getScriptToRun.sh’ on web-server hosted at http://www.example.com and runs the script locally without saving the file to disk first. As an added inspection into whats happening, you can see the dynamic creation of a file descriptor by utilizing the following:
Pipemill
{ { { tee /dev/fd/3 /dev/fd/4 | command1 >&9; } 3>&1 | command2 >&9; } 4>&1 | command3 >&9; } 9>&1
‘tee /dev/fd/3 /dev/fd/4‘ – takes stdin (/dev/fd/0) and duplicates it into three separate piped paths:
- stdout /dev/fd/1 -> command1 -> /dev/fd/9 -> stdout (/dev/fd/1)
- /dev/fd/3 -> command2 -> /dev/fd/9 -> stdout (/dev/fd/1)
- /dev/fd/4 -> command3 -> /dev/fd/9 -> stdout (/dev/fd/1)
NOTE: stderr (/dev/fd/2) is maintained end to end without redirection.
Other Interesting tidbits
# CREATE SCRIPT $ echo '#!/bin/sh echo import.sh[1]: first line of imported script echo import.sh[2]: bla bla...' > import.sh $ echo '#!/bin/sh echo main.sh[1]: this is the first line of the main script echo main.sh[2]: lets inline import import.sh . import.sh echo main.sh[4]: this is the fourth line of the main script' > main.sh # EXECUTE SCRIPT $ /bin/sh ./main.sh
. filename reads a filename as if it had occurred at that place in the script..