Effortless Text Processing and Task Automation

These topics are covered in today's session by Pranav Jambare

Filters

In Linux, filters refer to commands that process input data and produce output data based on specific criteria.
Used, when we want to see exact data rather than the whole data.
Some of the common filters will be explained below.

Sort

This sort filter is used to sort the content of a file based on the specific format given for sorting.

#To sort the content of file in ascending order
#Also display the output on the terminal

#Syntax:
sort <filename>

#Example:
[viraj@localhost ~]$ vim sortTheFile
apple
mango
banana
grapes
jackfruit
pineapple

[viraj@localhost ~]$ sort sortTheFile
apple
banana
grapes
jackfruit
mango
pineapple

Options for the sort filter

#Syntax:

# Used to sort the content of file in decending order
sort -r <filename>

# Used to sort the numbers present in the file in ascending order
sort -n <filename>

# Used to check whether the file content is sorted or not
sort -c <filename>

# Used to sort the content of file & also remove the duplicate content in file
sort -u <filename>

#used to sort the months in order
sort -M <filename>

Tail

The tail filter is used to display the last few lines of a file or stream of data.

#Syntax:
tail <filename>

#Example:
[viraj@localhost ~]$ tail mySampleFile
This is a sample file
_____________________________________________________________________________

# tail filter with options: 
#To display the last 'n' lines of a file
#Syntax:
tail -n <filename>

#Example:
[viraj@localhost ~]$ vim filterFile
Hello
Welcome
To
My
Blog

[viraj@localhost ~]$ tail -3 filterFile
To
My
Blog
_____________________________________________________________________________

# To merge the content of the files
#Syntax:
tail -q <file1> <file2>

#Example:
#Creating the file1 and file2 and then merging them...
[viraj@localhost ~]$ vim file1
This is my file1 created by me

[viraj@localhost ~]$ vim file2
This is another file2 created by me again!!

[viraj@localhost ~]$ tail -q file1 file2
This is my file1 created by me
This is another file2 created by me again!!
_____________________________________________________________________________

# To get the dynamic (real-time) output on the terminal
#Syntax:
tail -f <filename>

#Example:
# To get the logs output dynamically in myCustom.log file
tail -f myCustom.log

Head

The head filter is used to display the first few lines of a file or a stream of data.
It is the counterpart to the tail command, which shows the last few lines of a file.

#To display first 'n' lines of the file
#Syntax:
head -n <filename>

#Example:
#Creating the file...
[viraj@localhost ~]$ vim myHeadFile
Hello 
Again
Nice 
Meeting
You!!

#First 2 lines are printed...
[viraj@localhost ~]$ head -2 myHeadFile
Hello 
Again

#The other head options are very similar to the tail options mentioned above...

WC

WC is a filter that stands for word count.

#Syntax:
wc <filename>

#Example:
[viraj@localhost ~]$ wc myFile
1 8 37 myFile
____________________________________________________________________________

# wc with options:
#To get the line count in the file
#Syntax:
wc -l <filename>

#Example:
[viraj@localhost ~]$ wc -l myFile
1 myFile
____________________________________________________________________________

# To get the count of characters in the file
#Syntax:
wc -m <filename>

#Example:
[viraj@localhost ~]$ wc -m myFile
37 myFile
____________________________________________________________________________

# To get the count of bytes in the file
#Syntax:
wc -c <filename>

#Example:
[viraj@localhost ~]$ wc -c myFile
37 myFile
____________________________________________________________________________

# To get the count of words in the file
# Syntax:
wc -w <filename>

#Example:
[viraj@localhost ~]$ wc -w myFile
8 myFile

Uniq

The uniq filter in Linux is used to remove duplicate lines from a sorted file or input stream.
Its primary purpose is to identify and display unique lines, removing consecutive duplicates and leaving only one occurrence of each unique line.

#To get count of repetition
#Syntax:
uniq -c <filename>

#Example:
# Creating a file...
[viraj@localhost ~]$ vim myUniqFile
viraj
viraj
amogh
suyash
suyash

[viraj@localhost ~]$ uniq -c myUniqFile
2 viraj
1 amogh
2 suyash
____________________________________________________________________________

#To only print the repeated lines
#Syntax:
uniq -d <filename>

#Example:
[viraj@localhost ~]$ uniq -d myUniqFile
viraj
suyash
____________________________________________________________________________

#To print repeated lines with occurence
#Syntax:
uniq -D <filename>

#Example:
[viraj@localhost ~]$ uniq -D myUniqFile
viraj 
viraj
suyash
suyash
____________________________________________________________________________

#To only print unique lines
#Syntax:
uniq -u <filename>

#Example:
[viraj@localhost ~]$ uniq -u myUniqFile
amogh

Comm

#To show the common content between the two files
#Syntax:
comm <file1> <file2>

#Example:
#Creating two files...
[viraj@localhost ~]$ vim commFile1 
ABC
DEF
GHI
JKL

[viraj@localhost ~]$ vim commFile2
ABC
DEF
GHO
JKM

[viraj@localhost ~]$ comm myFile1 myFile2
            ABC
            DEF
GHI                  
      GHO
JKL   
      JKM

Diff

#To show the different content between the two files
#Syntax:
diff <file1> <file2>

#Example:
[viraj@localhost ~]$ diff myFile1 myFile2
3,4c3,4
< GHI
< JKL
------
> GHO
> JKM

GREP

# To search the exact word
#Syntax:
grep -w <filename>

#Example:
[viraj@localhost ~]$ grep computer myFile
____________________________________________________________________________

# To search the word with case insensitivity
#Syntax:
grep -i -w <filename>

#Example:
[viraj@localhost ~]$ grep -i computer myFile
____________________________________________________________________________

# To display only the matched words (whole words)
#Syntax:
grep -o -w <filename>

#Example:
[viraj@localhost ~]$ grep -o computer myFile
____________________________________________________________________________

# To get count of a word in a file
#Syntax:
grep -c -w <filename>

#Example:
[viraj@localhost ~]$ grep -c computer myFile
____________________________________________________________________________

# To search for multiple words
#Syntax:
grep -e -w -e -w <filename>

#Example:
[viraj@localhost ~]$ grep -e computer -e program myFile
____________________________________________________________________________

# To display the matched line and line number
#Syntax:
grep -n -w <filename>

#Example:
[viraj@localhost ~]$ grep -n computer myFile

AWK

# To print the content of the file
#Syntax:
awk '{print}' <fileName>

#Example:
[viraj@localhost ~]$ awk '{print}' myFile
____________________________________________________________________________

# To print the lines that match the given word in the file
#Syntax:
awk '/word/ {print}' <fileName>

#Example:
[viraj@localhost ~]$ awk '/computer/ {print}' myFile
____________________________________________________________________________

# To print specific words of each line in the file
#Syntax:
awk '{print $<lineNumber>, $<lineNumber>}' <fileName>

#Example:
[viraj@localhost ~]$ awk '{print $2, $6}' myFile
____________________________________________________________________________

# To print the last word of each line in the file
#Syntax:
awk '{print $NF}' <fileName>

#Example:
[viraj@localhost ~]$ awk '{print $NF}' myFile
____________________________________________________________________________

# To print specific words of specific lines in the file
#Syntax:
awk 'NR==<line_number> {print $<lineNumber>, $<lineNumber>}' <fileName>

#Example:
[viraj@localhost ~]$ awk 'NR==2, NR==3 {print $2, $4}' myFile

Cut

#Syntax:
cut -d<delimiter> -f<fieldNumber> <fileName>

#Example:
[viraj@localhost ~]$ cut -d"-" -f3 myFile

Sed

#To replace the word by occurrence
#Syntax:
sed 's/<wordNeedToReplace>/<replaceWith>/<occurrenceNum>' <filename>

#Example:
sed 's/ABC/VIRAJ/2' myFile
_____________________________________________________________________________

#To replace the word globally
#Syntax:
sed 's/<wordNeedToReplace>/<replaceWith>/g' <filename>

#Example:
sed 's/ABC/VIRAJ/g' myFile
_____________________________________________________________________________

#To replace the word on the specific line
#Syntax:
sed '<lineNumber> s/<wordNeedToReplace>/<replaceWith>/' <filename>

#Example:
sed '3 s/program/programs/' myFile
_____________________________________________________________________________

#To delete the last line
#Syntax:
sed '$d' <filename>

#Example:
sed '$d' myFile
_____________________________________________________________________________

#To delete particular lines
#Syntax:
sed '<lineNumber>d' <filename>

#Example:
sed '1,3d' myFile
_____________________________________________________________________________

#To delete lines matching a word
#Syntax:
sed '/word/d' <filename>

#Example:
sed '/computer/d' myFile

Shell Script

A shell script in Linux is a script written in a scripting language that is interpreted by a shell.
A shell script is a series of commands written in a text file and executed as a single unit.
In a shell script, you can use the full range of commands and features available in the shell, including loops, conditionals, functions, variables, and input/output redirection.
Shell scripts allow you to automate tasks, execute multiple commands sequentially, and perform complex operations that would otherwise require manual execution on the command line.
The file extension commonly used for shell scripts is .sh, but it is not mandatory.

Types of Shell Script

Non Interactive :
- Non-interactive shell scripts are designed to run without any user input during their execution.
- They execute a predefined set of commands and do not require any interaction with the user while running.
Interactive :
- Interactive shell scripts are designed to interact with the user during their execution.
- They prompt the user for input and respond accordingly.
Command Line :
- Command-line shell scripts are run from the command line and accept arguments provided by the user when executing the script.

Shebang

The shebang, also known as a hashbang or pound-bang, is the special character sequence #! at the beginning of a script file in Linux.
The shebang indicates the interpreter that should be used to execute the script.

#!/pathTo/interpreter

For example, in a Bash shell script, the shebang would typically be:

#!/bin/bash

Creating and Running a Script in Linux

Create a file with the extension .sh

#Creating and editing the file in vim editor...
vim Sample.sh
___________________________________________________________________________
#At the beginning of the file use Shebang and start writing the script...
#Syntax:
#!/bin/bash
<setOfCommands>...

#Example:
#!/bin/bash

echo "This is my first script"
echo "Thank you!!"

#Now save the file...

Running the sample.sh script

#You can give execute permission to user and then run the file
#Then run as shown...
chmod 777 Sample.sh
./Sample.sh

#Else, you can use bash command (if execute permission is missing)
#Then the file name to execute the file
bash Sample.sh

#Output:
This is my first script 
Thank you!!

To declare the variables in Linux (Temporary)

#Syntax:
<variableName>=<value>

#Example:
myName=Viraj

#The above example shown has a temporary existance, i.e
#if bash command is execute the myName variable won't have it's existance and can't be used anymore
#To declare variables permanently use following method...

To declare the variables in Linux (Permanent)

#Open a file called bashrc or bash.bashrc (according to linux distro's) in vim editor
#Now, define the variables at the last
#Save the file
#Now try to use it in any of the commands, it would have its permanent existance
echo "Good Morning, ${myName}"
Good Morning, Viraj

If - Else Statements

Simple if

 #Syntax:
 if [ expression ]
 then
     statement
 fi

 #Example:
 if [ ${A} -gt ${B} ]
 then
     echo "${A} is greater than ${B}"
 fi

If - Else

 #Syntax:
 if [ expression ]
 then
     statement
 else
     statement
 fi

 #Example:
 if [ ${A} -gt ${B} ]
 then
     echo "${A} is greater than ${B}"
 else
     echo "${B} is greater than ${A}"
 fi

Else-if Ladder

 #Syntax:
 if [ expression1 ]
 then
     statement1

 elif [ expression2 ]
 then
     statement2

 else
     statement5
 fi

 #Example:
 if [ ${age} -lt 18 ]
 then
     echo "You are not allowed to drive!!"
 elif [ ${age} -ge 18 ]
 then
     echo "You are allowed to drive!!"
 elif [ ${age} -ge 100 ]
 then
     echo "Please don't drive!!"
 else
     echo "Invalid Age!!"
 fi

Nested If

 #Syntax:
 if [ expression1 ]
 then 
     if [ expression2 ]
     then
         statement1
     else
         statement2
     fi
 else
     statement3
 fi

 #Example:
 if [ ${firstName} == Viraj ]
 then
     echo "First name verified!!"
     if [ ${lastName} == Nalbalwar ]
         echo "Verified User!!"
     else
         echo "Last name incorrect!!"
     fi
 else
     echo "First name incorrect!!"
 fi

Filters & Shell Script In Linux

Filters

Sort

Tail

Head

WC

Uniq

Comm

Diff

GREP

AWK

Cut

Sed

Shell Script

Types of Shell Script

Shebang

Creating and Running a Script in Linux

To declare the variables in Linux (Temporary)

To declare the variables in Linux (Permanent)

If - Else Statements

Comments

More from this blog

Linux Shell Scripting : Functions &Cases

Shell Scripting - Loops & Command Line Arguments

Booting Process, ACL, Processes, Process State, Alias in Linux

Logs, YUM & RPM, Repositories, Cron in Linux

Command Palette

Filters

Sort

Tail

Head

WC

Uniq

Comm

Diff

GREP

AWK

Cut

Sed

Shell Script

Types of Shell Script

Shebang

Creating and Running a Script in Linux

To declare the variables in Linux (Temporary)

To declare the variables in Linux (Permanent)

If - Else Statements

Comments

More from this blog