Ravindra BagaleCourses & study guides

6. Linux Advanced Commands

6.3 Text Processing: awk and sed

By Last updated:

awk columns sathi, sed find-and-replace sathi. Donhi log analysis aani config automation madhe roj lagtat.

awk — column-based processing

awk splits each line into fields $1, $2, ... (space-separated by default; -F sets the separator).

awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head
#  ^ top 10 client IPs hitting your web server

awk -F: '{print $1, $7}' /etc/passwd          # user name and shell
awk -F: '$3 >= 1000 {print $1}' /etc/passwd   # normal (non-system) users
df -h | awk 'NR>1 {print $5, $6}'             # usage % and mount point
awk '{sum += $10} END {print sum/1024/1024 " MB"}' /var/log/nginx/access.log  # bytes served

sed — stream editor (find & replace)

sed 's/http/https/' file.txt              # replace first match per line (prints result)
sed 's/http/https/g' file.txt             # replace all matches
sed -i 's/Listen 80/Listen 8080/' /etc/httpd/conf/httpd.conf   # edit file in place
sed -i.bak 's/old/new/g' config.ini       # in place, keep backup config.ini.bak
sed -n '10,20p' file.txt                  # print only lines 10-20
sed '/^#/d' file.txt                      # delete comment lines

Test sed before using -i

Run the sed command without -i first and check the output. Once you are happy, add -i (or -i.bak to keep a backup).

Why this matters for security

The first awk one-liner above (top client IPs) is how you spot a brute-force or scanning source in a web log within seconds – the attacker's IP jumps to the top of the list.

Ravindra Bagale's Tip

Many students run sed -i directly on a live config – and one wrong pattern ruins the whole file. First look at the output without -i, then use -i.bak so a backup is created automatically.

Practice task

From /etc/passwd, print the user names whose shell is /bin/bash. Using sed, print lines 5–10 of any config file and delete comment lines in a copy of it.