Awk and sed

For those of you non GNU-junkies, awk and sed are an essential part of any admin's toolbox.  They are interpreted languages (meaning there's no compiling involved and your programs just run on the fly) and they've been around for a long long time.  It's pretty rare that I write a script of more than 20 lines and don't use at least one of them.

Awk and sed are pretty feature-heavy languages, and in fact there's an entire O'Reilly book just on those two alone.  However, I really only use them for one thing each; I use awk to extract a particular field from a string, and I use sed to do quick and dirty substitutions.   So, here's the Nate Aiman-Smith quick and dirty howto:

For awk, 99% of the time I use this line:

[command printing out stuff] | awk -F'[field separator]' '{print $[field number]}'

That's it!  So, for example, if I had $numbers (or %numbers% for you windows guys) set to  "one two three four" and I wanted to get the third word, I would just do:

echo %numbers% | awk '{print $3}'

which would print out "three".  Note that the field separator defaults to whitespace.  If my string was "one::two::three::four", I could do:

echo %numbers% | awk -F'::' '{print $3}'

For the same result.

For sed, the construction is:

sed 's/[search]/[replace]/' [file]

or 

[command printing out stuff] | sed 's/[search]/[replace]/'

That does the substitution once (the first time the search string is found).  You can add '/g' at the end to make it global.  So, for example, to change all instances of "Nathan" to "Naton" in my file "recommendation.txt", I could do:

cat (or "type" for you windows folks) recommendation.txt | sed 's/Nathan/Naton/g' > good_recommendation.txt

Which would create a new file, good_recommendation.txt with the substitutions.

Now of course, Perl can do all these things almost as easily, and it can do it a _lot_ faster, but when you're writing the quick and dirty script or one-liner, it's a lot easier just to use one of these.   If you're a Unix administrator, you need to know at least these basics of sed and awk, because you're going to see them a lot.  It saddens me how many Linux admins don't seem to know these basic tools, but on the other hand it makes me look like some kind of god when I'm able to do a search-and-replace on 1000 files in about 2 minutes with a one-liner.

There are actually precompiled windows binaries available for them if you want to start working on your own script-fu.  Or, even better, use Cygwin or a Linux distro (or open a terminal in Mac OS X) to really dig into it.