FILTERS
A filters is a program that takes its input from the standard input file, processes (or filters) it, and sends its output to the standard output file. Linux has a rich set of filters that can be use to work on data in an effective way. Some examples of filters are cat, grep, wc, tr, and cut.
The grep Filter
The grep filter searches a file for a particular pattern of characters and displays all the lines that contain that pattern. The pattern that is searched for is referred to as a regular expression. The grep filter cannot be use without specifying a regular expression.
Syntax
grep regular_ expression [filename]
The file name is optional in the grep command. Without a file name, grep expects standard input (i.e. inputs from the keyboard). As a line is entered, grep searches the line for the regular expression and displays the line if it contains the expression. Execution stop when a user indicates the end of input by pressing Ctrl + d.
Example:
grep job
Work
Job
Job
Task
<Ctrl>d
In this example, grep searches for the word the user types in the work, job, and task. When it finds the word job it displays the word job again on the standard output. That is why you can see job written twice in the output.
Example:
grep “root” /etc/passwd
root:x:0:0:root:/root:/bin/bash
In the above example, the lines containing the expression “root” are displayed.
Note
When specifying the pattern “root” in the above example, the double quotes (“) are optional. The above command can also be given as:
grep root /etc/passwd
Using the characters summarized in the following table, you can specify complex regular expressions.
Character | Use | Example | Description |
[ ] | Matches any one of a set of characters | Grep “New [abc] “ filename | It specifies the search pattern as ‘Newa; ‘Newb; or ‘Newc’. |
[ ] with hyphen | Matches any one of a range of characters | Grep “New[a-c] “ filename | It specifies the search pattern as ‘Newa; ‘Newb; or ‘Newc’. |
^ | Matches the pattern following it if the pattern occurs at the beginning of a line | Grep “New [abc]”filename | It specifies the search pattern as ‘Newa; ‘Newb; or ‘Newc; but this must occur at the beginning of the line. |
^ within [ ] | Matches the pattern that does not contain any character in the specified set | Grep “New [^a-c] “ filename | It specifies a pattern containing the word ‘New’ followed by any character other than ‘a’, ‘b’, or ‘c’. |
$ | Matches the pattern preceding it if the pattern occurs at the end of each line | Grep “New [abc] $ “ filename | It specifies the search pattern as ‘Newa’, ‘Newb’, or ‘Newc’, but this must occur at the end of the line. |
.(dot) | Matches any one character | Grep “New [abc] “ filename | It specifies a pattern containing the word ‘New’ following by any character, followed by ‘a’, ‘b’, or ‘c’. |
/ (backslash) | Ignores the special meaning of the character following it | Grep ” New\.\ [abc] “ filename | It specifies a search pattern, New.[abc] , in which the dot signifies the dot character itself and [abc] signifies of the characters dot, [and] is ignored. |
Options of the grep Filter
The grep filter also has options that alter the output of the command. These are:
-n :This prints each line matching the pattern, along with its line number. The number is printed at the beginning of the line.
-c :This prints only a count of the lines that match a pattern.
-v :This prints all the lines that do not match the pattern specified by the regular expression.
Options must be specified before the regular expression. Option can also be combined. (For example, -n and –v can be used together as –nv .)
The wc Filter
The wc filter is used to count the number of line, words, and characters in a disk file or in the standard input.
Syntax:
wc [options] [filename/s]
Example
wc sample
2 7 29
The file sample has two lines, seven words, and 29 characters.
The following table summarizes the options of the wc filter.
Options:
-l : displays the number of lines
-w : diplays the number of words.
-c : displays the number of characters
Since wc is a filter, it use the standard input if no file name is provided, as depicted below:
wc
wc is an example of a filter
<Press Ctrl and d>
1 7 29
The cut Filter
The cut filter is useful when specific columns from the output of certain commands (such as Is -1, who) need to be extracted. It can also be use to extract specific columns from files.
Syntax
cut [options] [filename/s]
Refer to the following for the options of the cut filter:
-f : displays the specified columns
-c : displays the specified characters.
-d : specifies the columns delimiter (or you could say seperator).
Example
cut -d ‘:’ -f1 /etc/passwd
root
daemon
bin
sys
sync
games
www-data
…
alexander
Here, the cut command has been used to extract only the names of the users in the /etc/passwd file. The field separator is “:”.
Example
cut -c1-5 /etc/passwd
This command will display the first five characters from the file /etc/passwd.
Example
cut -d ‘:’ -f1,6 /etc/passwd
root:/root
daemon:/usr/sbin
bin:/bin
sys:/dev
sync:/bin
games:/usr/games
…
alexander:/home/alexander
The above command will display the first and the sixth column from the file /etc/passwd.
Example
cut -d ‘:’ -f1-4 /etc/passwd
The above command will display the first, second, third, and fourth column from the file /etc/passwd.
The tr Filter
The tr filter can be used to translate one set of characters to another. The tr filter translates the characters in the output only and does not affect the input file.
Example
tr ‘:’ ”< /etc/passwd
This command replaces all the occurrences of the ‘:’ character with a space for the file /etc/passwd, and displays the output on the standard output.
This filer can also be used to squeeze (reduce) repeated occurrences of a character into one. Several commands have a multicolumn output and the gap between columns is more than one space. In such cases, users cannot use cut if they want to extract a column, since two columns are separated by more than one space. Therefore, the solution is to squeeze the multiple space between columns into a single space and then use the cut command to extract the desired columns.
The –s option is used to squeeze several occurrences of a character into one character, as illustration below.
who
alexander :0 2016-06-12 21:40 (:0)
alexander pts/0 2016-06-14 12:07 (:0)
To make the column separator a single space, tr – s has to be use, as shown below:
who > temp_file
tr -s “ ” < temp_file
alexander :0 2016-06-12 21:40 (:0)
alexander pts/0 2016-06-14 12:07 (:0)
Here, the output of the who command is first redirected to a file name ‘temp_file’. And then the –s option of tr works on every record, squeeze (reducing) the repeated spaces into a single space.
Another common use of the tr filter is case-conversion.
Example
tr “[a-z]” “[A-Z]”
The cat jumped over the dog quickly
THE CAT JUMPED OVER THE DOG QUICKLY
The above command converts all lowercase letters to uppercase.
The content of a file could also be translated on the command line. Let us translate the temp_file we created earlier to all uppercase;
tr “[a-z]” “[A-Z]” < temp_file
ALEXANDER :0 2016-06-12 21:40 (:0)
ALEXANDER PTS/0 2016-06-14 12:07 (:0)
PIPES
In Linux, filers and other commands can be combined such that the standard output of one filter or command can be sent as standard input to another filter or command.
For example, to display all the contents of the current directory at a time, you can type the following commands:
ls > temp_file
more temp_file
deja-dup
Desktop
Documents
Downloads
examples.desktop
Music
Pictures
Public
tempfile
Templates
ubuntu
Videos
Here, a listing (using the ‘ls’ command) of the directory is stored in the file temp_file by the first command. This file is then used as input by the more command.
Through the Linux pipe feature, these two steps can be combined and executed as a single command without creating a temporary file, as shown below:
ls | more
deja-dup
Desktop
Documents
Downloads
examples.desktop
Music
Pictures
Public
tempfile
Templates
ubuntu
Videos
The vertical bar (|) is the pipe character, which indicates to the shell that the output of the command before ‘I’ is sent as input to the command after ‘I’.
Another advantage of the pipe feature is that utilities do not have to be rewritten to perform complex tasks. Instead, Linux tools (commands) can be combined. There is no limit to the number of filters or commands in a pipe. Consider the example in the figure below to understand how a pipe works.
Example
The following is a sample output of the ls -1 command:
total 56
drwxrwxr-x 2 alexander alexander 4096 May 24 08:46 deja-dup
drwxr-xr-x 9 alexander alexander 4096 Jul 30 12:08 Desktop
drwxr-xr-x 5 alexander alexander 4096 Jul 29 16:10 Documents
drwxr-xr-x 2 alexander alexander 4096 Jul 30 11:31 Downloads
-rw-r–r– 1 alexander alexander 8980 Apr 28 07:46 examples.desktop
drwxr-xr-x 2 alexander alexander 4096 Apr 28 07:54 Music
drwxr-xr-x 3 alexander alexander 4096 Jul 30 11:31 Pictures
drwxr-xr-x 2 alexander alexander 4096 Apr 28 07:54 Public
-rw-rw-r– 1 alexander alexander 109 Jul 30 13:34 tempfile
drwxr-xr-x 2 alexander alexander 4096 Apr 28 07:54 Templates
drwxrwxr-x 2 alexander alexander 4096 Jun 6 11:02 ubuntu
drwxr-xr-x 2 alexander alexander 4096 Jul 30 11:30 Videos
Some example of the pipe feature, based on the above output, are given below.
1. To display the names of all the files in the current directory, type the following command:
ls -l | grep “^-” | tr -s “ ” | cut -d” ” -f9
Here, the output of the ls -1 command is given as input to the grep command. The Is -1 command gives a detailed list of the files in your current directory. The grep command extracts all the ordinary files. This output is given to the tr command, which replaces multiple spaces with a single. The cut command takes this as input and extracts the ninth field, which is the file name, and displays it.
2. To display the file names in the current directory along with the file size, a screen-full at a time, you need to give the command:
ls -l | tr -s “ ” | cut -d” ” -f5,9 | more
3. Let sale there is a staff data file (lets say ‘staffContact‘) that contains information like this;
mikel,60 male 08066786568
sarah 23 female 09086676768
collins 34 male 08054768769
brymo 44 male 07054646271
gerad 23 Male 08165242516
To scan the file, staffContact, and search for the string, “male” (assuming we want to extract all the staff who are males), we can use this command;
cat staffContact| grep “male” | more
or
grep “error” < staffContact | more
The tee command.
the intermediate output in a pipe is discarded y linux, which means it is not saved on the disk. Sometimes, you may want to redirect the output of a command to another command and also in the process save it on the disk for later use. The tee command takes standard input and writes to standard output and to file(s). if the file to be written to does not exist, the file is created. If the file already exists, its contents are overwritten. The -a (append) option can be used to append contents to an already existing file.
ls -al | tee allFiles
The above command displays all the files in the current directory on the screen and also writes to the file “allFiles”.
cut -d”:” -f1-4 /etc/passwd | tee users-passwords
The above command extracts the first, second, third, an fourth columns from the passwd file, and then passed to the tee command which creates the file users-passwords to store the data (or output) before displaying it on the screen.
Happy Linux’NG
- Time complexity analysis: How to calculate running time - April 1, 2024
- Sovereign Tech Fund Invests €1M In GNOME’s Open-Source Project - November 19, 2023
- Google’s Bard AI: ChatGPT Rival or The Next Frontier in AI and NLP Technology - February 8, 2023