Text processing, pipes and log analysis

Streams and pipelines

A process normally has stdin (0), stdout (1) and stderr (2). A pipe connects stdout to the next command's stdin. 2> redirects errors; 2>&1 sends errors to the current stdout destination. Quoting prevents unwanted shell expansion. Single quotes preserve literal text; double quotes permit variable substitution.

Prepare a small log

mkdir -p ~/academy
cat > ~/academy/access-demo.log <<'LOG'
10.0.1.10 GET / 200 12
10.0.1.11 GET /missing 404 3
10.0.1.10 POST /upload 500 95
10.0.1.12 GET / 200 15
LOG
cat ~/academy/access-demo.log

Analyze step by step

head -n 2 ~/academy/access-demo.log
tail -n 2 ~/academy/access-demo.log
grep ' 200 ' ~/academy/access-demo.log
awk '$4 >= 400 {print $1, $3, $4}' ~/academy/access-demo.log
awk '{print $1}' ~/academy/access-demo.log | sort | uniq -c | sort -nr
awk '{total += $5; n++} END {if(n) print total/n}' ~/academy/access-demo.log
sed 's/POST/WRITE/' ~/academy/access-demo.log
wc -l ~/academy/access-demo.log

Expect two successful requests, one 404 and one 500. The average duration is 31.25 in the units defined by this sample. uniq collapses adjacent duplicates, which is why sorting happens first. sed without -i prints a transformed copy; it does not modify the source.

Advanced patterns

find ~/academy -type f -name '*.log' -print0 | xargs -0 wc -l
journalctl -u nginx --since '1 hour ago' --no-pager
# Follow a real log when the service is installed:
sudo tail -F /var/log/nginx/error.log

Null delimiters protect filenames containing spaces/newlines. tail -F follows a named log across rotation. Real access-log formats differ: do not reuse field positions from this simplified sample without checking the log format.

Failure analysis

A grep exit code of 1 means no match; 2 means an error. Empty output is not proof there were no incidents if the time range, source log or permissions were wrong. Preserve original logs and redact secrets before sharing extracts.

Assignment

Count requests by status, identify the slowest request, and print unique clients. State the difference between a request count and a unique-user count.

Official references

GNU grep GNU awk

Interview and revision check

Why sort before uniq -c?

uniq groups adjacent equal lines. Sorting brings matching values together so the counts cover all occurrences.