1. Introduction
When configuring software for compilation or simply running any script, we might sometimes encounter errors with a relatively cryptic text:
- /bin/sh^M: bad interpreter
- /bin/bash^M: bad interpreter
- syntax error: unexpected end of file
- -bash: $’sleep\r’: command not found
In this tutorial, we’ll tackle line endings and errors that we might see when they are incorrect. First, we discuss the general concept of how lines end. Next, we explore different problems that line endings can cause. After that, we’ll talk about specific types of scripts that can exhibit line-end issues. Finally, we delve into ways to make sure a file uses given characters as newlines.
For brevity and clarity, when the characters would otherwise be hidden, we often use
- line ending
- line break
- newline
We tested the code in this tutorial on Debian 12 (Bookworm) with GNU Bash 5.2.15. It should work in most POSIX-compliant environments.
2. Line Endings
How a file line ends isn’t universal. Moreover, we can ignore the concept of lines altogether, instead treating files as binary, i.e., a sequence of bytes.
When it comes to text files with separate lines, Microsoft Windows, Apple macOS, and Linux differ in the default line endings that each operating system (OS) expects:
- Microsoft Windows –
- Apple macOS – initially
, currently - Linux –
Further, since some software products perform automatic conversions between these options, we can end up with different issues.
3. Problems With Line Endings
While many applications support all ways to end a line, some sensitive texts might not work without the correct ending.
3.1. Source Code File Newlines
Also called tokens, character sequences in any source code should adhere to the syntax of their language.
So, depending on the source file and how it’s used, we might encounter different problems stemming from its line endings.
On the one hand, some file types like the Portable Document Format (PDF) can work with any version of a newline, but we have to take the number of bytes into account for, e.g., the xref table:
$ cat lf.pdf
[...]
xref
0 5
0000000000 65535 f
0000000018 00000 n
0000000077 00000 n
0000000178 00000 n
0000000457 00000 n
[...]
$ cat crlf.pdf
[...]
xref
0 5
0000000000 65535 f
0000000021 00000 n
0000000086 00000 n
0000000195 00000 n
0000000490 00000 n
[...]
On the other hand, specific line breaks are sometimes a must as most interpreters and compilers need to know what to expect for a newline sequence in files.
3.2. Shell Newlines
Critically, attempting to mix and match line endings in a command, script, or a sensitive file can result in issues.
For instance, shells have strict rules when it comes to the type and order of character sequences:
$ test<LF>
$
Notably, the
To demonstrate, let’s try to prepend a
$ test<CR><LF>
-bash: $'test\r': command not found
By using Ctrl-v Ctrl-m, we append
Of course, the command line is only accepted after
3.3. Shell Script Newlines
Of course, what applies to the shell is also valid for shell scripts. In addition, scripts have some unique constructs that can pose further issues.
One of these is the shebang, which hints at the interpreter of the source file:
$ cat script.sh
#!/bin/bash<CR><LF>
test<CR><LF>
$ chmod +x script.sh
$ ./script.sh
-bash: ./script.sh: /bin/bash^M: bad interpreter: No such file or directory
Here, we see an error, which includes the special character notation ^M instead of
The error itself states that /bin/bash
$ cat script.sh
#!/bin/bash<CR><LF>
test<CR><LF>
$ bash script.sh
script.sh: line 2: $'test\r': command not found
Notably, we don’t see a shebang problem here.
Let’s see a common scenario where errors like the above arise.
4. Autotools Configure Scripts
The Autotools configure script mechanism is mainly used to prepare code for compilation on a given platform:
- detect available resources
- set executable paths
- identify library locations
- configure linking
Often, we get configure scripts as part of a source package, which we’ve downloaded. In some cases, like when using Git or FTP, the downloads themselves can convert newlines.
Since it’s a shell script produced by Autoconf, configure is susceptible to all issues with newlines we already discussed:
$ cat configure
#!/bin/sh<CR><LF>
[...]
# Define the name and version of the package.<CR><LF>
PACKAGE_NAME='package'<CR><LF>
[...]
$ ./configure
-bash: ./script.sh: /bin/sh^M: bad interpreter: No such file or directory
Notably, the shebang of configure causes an error due to unexpected line endings. However, the rest of the file might not raise any exceptions as it mainly, if not exclusively, consists of variable assignments.
Critically, this is a double-edged sword, as not erroring out doesn’t mean we won’t get issues down the line.
5. External Interpreter Scripts
Same as for internal scripts, the shell can understand the first line of external interpreter scripts as well. Let’s look at some common examples.
5.1. Perl
To begin with, Perl is a scripted language common to many Linux installations.
So, we can write a basic Perl script:
$ cat script.pl
#!/usr/bin/env perl<LF>
print "Perl script output."<LF>
In this short program, we just set the shebang to point to the perl interpreter and then print some text to the standard output:
$ chmod +x script.pl
$ ./script.pl
Perl script output.
Naturally, the code runs without issues.
However, let’s try to run the same script after replacing the
$ cat script.pl
#!/usr/bin/env perl<CR><LF>
print "Perl script output."<CR><LF>
$ ./script.pl
/usr/bin/env: ‘perl\r’: No such file or directory
/usr/bin/env: use -[v]S to pass options in shebang lines
As expected, t*he shell detects the unexpected carriage return and considers it part of the line, leading to an [env]ironment search of perl\r*. Naturally, this results in an error.
5.2. Python
Similar to Perl, unexpected line endings in Python scripts can lead to errors:
$ cat script.py
#!/usr/bin/env python<CR><LF>
print "Python script output."<CR><LF>
$ chmod +x script.py
$ ./script.py
/usr/bin/env: ‘python\r’: No such file or directory
Of course, the same error happens with python, python3, and other Python binaries.
However, because of the different executables for the Python interpreter, we might exhibit another very similar-looking error:
$ cat badscript.py
#!/usr/bin/env pythonr
print "Python script output."
$ chmod +x badscript.py
/usr/bin/env: ‘pythonr’: No such file or directory
In fact, the only difference between the two errors is the backslash. Although a very important character, the distinction might not be immediately obvious.
While the first error tells us that there is a
6. Fixing Line Endings
Generally, if a language or script expects given line endings, we should use them. Still, there are many options to detect and convert line breaks in files:
- vi with the set ff=unix file format command, even automatically as vi +’:wq ++ff=unix’
- sed with a simple substitution regular expression (regex) like ‘s/\r//’
- tr with -d and the unwanted character, e.g., ‘\r’
- awk with the gsub() function like ‘gsub(/\r/,””)’
- perl with an -e one-liner like ‘s/\r//’
- dos2unix for simple
conversion
In addition, we can avoid the shebang issues if we run a script as an argument of its interpreter instead of directly. This fact is especially important for configure scripts:
$ ./configure
-bash: ./script.sh: /bin/sh^M: bad interpreter: No such file or directory
$ bash configure
$
Still, to avoid any trouble, using an editor like vi to manually edit or automatically convert line endings can be beneficial.
7. Summary
In this article, we talked about newlines, how they can affect file processing, and what we can do about it.
In conclusion, having the correct line endings is often critical for a file to properly display or run, especially when it comes to shell scripts.