Tag Archives: bash

Removing empty lines using `sed`

Consider this snippet from a file:

50442


5838


1843


167356


131671

It’s a list of integers (the full file has 122 of them), but the delimiter is three newline characters. I needed to get rid of the empty lines in this file.

I thought of it as replacing three \n characters with one, so it seemed like a good place to use the sed command.

However, according to this FAQ for sed, it’s impossible to use \n in a sed command to strip out newlines.

5.10. Why can’t I match or delete a newline using the \n escape sequence? Why can’t I match 2 or more lines using \n?

The \n will never match the newline at the end-of-line because the newline is always stripped off before the line is placed into the pattern space. To get 2 or more lines into the pattern space, use the ‘N’ command or something similar (such as ‘H;…;g;’).

Sed works like this: sed reads one line at a time, chops off the terminating newline, puts what is left into the pattern space where the sed script can address or change it, and when the pattern space is printed, appends a newline to stdout (or to a file). If the pattern space is entirely or partially deleted with ‘d’ or ‘D’, the newline is not added in such cases. Thus, scripts like

       sed 's/\n//' file       # to delete newlines from each line
       sed 's/\n/foo\n/' file  # to add a word to the end of each line

will never work, because the trailing newline is removed before the line is put into the pattern space. To perform the above tasks, use one of these scripts instead:

       tr -d '\n' < file              # use tr to delete newlines
       sed ':a;N;$!ba;s/\n//g' file   # GNU sed to delete newlines
       sed 's/$/ foo/' file           # add "foo" to end of each line

Basically, the utility reads the input line-by-line, so it never sees the newlines anyway. It looks like we can’t replace multiple newlines with a single newline after all.

However, a regex trick I found was to detect empty lines and delete them:

sed /^$/d FILE

This regex looks for any line that ends immediately after it begins, then deletes that line. It worked perfectly for my needs.

.bash_profile madness on OSX

I ran into some issues setting up my terminal correctly on OSX. Namely, whenever I’d open a new tab on Terminal, my ~/.bashrc script didn’t get executed. So, I had no syntax highlighting, and I had none of my aliases.

It turns out that ~/.bashrc isn’t designed to be run for Terminal. This StackExchange answer has an explanation of why:

Why doesn’t .bashrc run automatically?

Terminal opens a login shell. So I followed their advice and sourced ~/.bashrc in ~/.bash_profile