Introduction to text manipulation on UNIX-based systems

https://www.ibm.com/developerworks/aix/library/au-unixtext/index.html

A basic tenets of UNIX philosophy is to create programs (or processes) that do one thing, and do that one thing well. It is a philosophy demanding careful thought about interfaces and ways of joining these smaller (hopefully more simple) processes together to create useful results. Normally textual data flows between these interfaces. Over time, more and more advanced text processing tools and languages have been developed. For languages, earlier on there was perl, later came python, and ruby. While these and other languages are very capable text processors, such tools are not always available, especially in a production environment. In this article, a number of basic UNIX text processing commands are demonstrated and may be used individually or in conjunction with each other to solve problems which may also be addressed with newer languages. For many people, an example provides more information than long winded explanations. Please note because of the variety of UNIX and UNIX-like systems available, command flags, program behavior, and output differs between implementations.

Use of cat

The cat command is one of the most basic commands. It is used to create, append, display, and concatenate files.

We can create a file with cat using the ‘>’ to redirect standard input (stdin) to a file. Using the ‘>’ operator truncates the contents of the output file specified. Text entered after that is redirected to the file specified to the right of the ‘>’ operator. The control-d signals an end-of-file, returning control to the shell.

Example of cat to create a file:

Use the ‘>>’ operator to append standard input into an existing file.

Example of cat to append a file:

Examine the contents of the grocery.list file, using cat without flags. Notice how the file contents include input from the redirection and append operator examples.

Example of cat without flags:

The cat command can be used to number the lines of a file.

Example of cat to count lines:

Use of nl

The nl filter reads lines from stdin or from specified files. Output is written to stdout, and may be redirected to a file or piped to another process. Behavior of nl is controlled through various command-line options.

By default, nl counts lines similar to cat -n.

Example default usage of nl:

Use the -b flag to specify lines to be numbered. This flag takes as its argument a “type”. The type tells nl which lines need to be numbered – use ‘a’ to number all lines, ‘t’ tells nl to not number empty lines or lines that are only whitespace, ‘n’ specifies no lines be numbered. In the example a type of ‘p’ for pattern is shown. nl numbers the lines specified by a regular expression pattern, in this case, lines starting with the letters ‘a’ or ‘b’.

Example of nl to number lines conforming to a regex:

By default, nl separates the line number from the text using a tab. Use -s to specify a different delimiter, such as the ‘=’ sign.

Example of nl to specify a delimiter:

Use of wc

The wc (wordcount) command counts the number of lines, words (separated by whitespace), and characters in specified files, or from stdin.

Examples of wc usage:

Using grep

The grep command searches specified files or stdin for patterns matching a given expression(s). Output from grep is controlled by various option flags.

For demonstration, a new file was created to use with the grocery.list.

Example of basic grep usage:

grep has a sizable number of option flags. Following are some examples demonstrating usage of a few options.

To display the filename (if multiple files) with number of lines on which the pattern was found – in this case, count the number of lines the word ‘apple’ occurs in each file.

Example of grep - counting number of matches in files:

When searching multiple files, using the -h option suppresses printing the filename as part of the output.

Example of grep - suppress filename in output:

In many situations, a case-insensitive search is desired. The grep command has the -i option to ignore case-sensitivity when doing searches.

Example of grep – case insensitive:

Sometimes, only the filename needs to be printed, not the pattern-matching line. grep provides the -l option to only print filenames containing lines with a matching pattern.

Example of grep – filenames only:

Line numbers can be provided as part of the output. Use the -n option to include line numbers.

Example of grep – include line numbers:

There are times when lines not matching the pattern are the desired output. In such situations use the -v option.

Example of grep – inverted matching:

Sometimes the pattern desired forms a “word” surrounded by whitespace or other characters like a dash or parentheses. Most versions of grep provide a -w option to ease writing searches for these patterns.

Example of grep – word matching:

Streams, pipes, redirects, tee, here docs

In UNIX a terminal by default contains three streams, one for input, and two output-based streams. The input stream is referred to as stdin, and is generally mapped to the keyboard (other input devices may be used, or could be piped from another process). The standard output stream is referred to as stdout, and generally prints to the terminal, or output may be consumed by another process (as stdin). The other output stream stderr primarily used for status reporting usually prints to the terminal like stdout. As each of these streams has its own file descriptor, each may be piped or redirected separately from the other, even if they are all connected to the terminal. File descriptors for each of these streams are:

• stdin = 0
• stdout = 1
• stderr = 2

These streams can be piped and redirected to files or other processes. This construct is commonly referred to as building a pipeline. For example, a programmer may want to merge the stdout and stderr streams, and then display them on the terminal but also save the results to a file to examine build issues. Using 2>&1 the stderr stream with file descriptor 2 is redirected to &1, a ‘pointer’ the stdout stream. This effectively merges stderr into stdout. Using the ‘|’ symbol indicates a pipe. A pipe links stdout from the left-hand process (make) to stdin of the right-hand process (tee). The tee command duplicates the (merged) stdout stream sending the data to the terminal and to a file, in this example, called build.log.

Example of merging and splitting standard streams:

Another example of redirection, a copy of a text file is made using the cat command and some stream redirection.

Example of redirection to make a backup file:

Earlier the nl command was used to add line numbers to a file displayed on stdout. The pipe can be used to send thestdout stream (from cat grocery.list) to another process, in this case, the nl command.

Example simple pipe to nl:

Another example shown earlier was to do a case-insensitive search of a file for a pattern. This can be done using redirection - in this case from stdin, or using a pipe, similar to the simple pipe example above.

Example with grep - stdin redirection and pipe:

In some situations a block of text will be redirected into a command or file as part of a script. A mechanism to accomplish this is to use a ‘here document’ or ‘here-doc’. To embed a here-doc into a script, the ‘<<’ operator is used to redirect the following text, until a end-of-file delimiter is reached. The delimiter is specified after the << operator.

Example basic here-doc on the command line:

This output can be redirected to a file, in this example the delimiter changed from ‘EOF’ to ‘!’. Then tr command (explained later) is used to upper-case the letters with a here-doc.

Example basic here-doc redirected to a file:

Using head and tail

The head and tail commands are used to examine the top (head) or bottom (tail) parts of files. To display the top two lines and bottom two lines of a file use the -n option flag with these commands respectively. Similarly, the -c option displays the first or last characters in the file.

Example basic use of head and tail commands:

A common use for the tail command is to watch log files or the output of running processes to see if there are issues, or to note when a process finishes. The -f (tail –f) option causes tail to continue to watch the stream, even after the end-of-file marker is reached, and continue displaying output when the stream contains more data.

Using tr

The tr command is used to translate characters from stdin, displaying them on stdout. In its general form, tr takes two sets of characters, and replaces characters from the first set with characters from the second set. A number of pre-defined character classes (sets) are available to be used by tr, and some other commands.

These classes are:

• alnum - alphanumeric characters
• alpha - alphabetic characters
• blank - whitespace characters
• cntrl - control characters
• digit - numeric characters
• graph - graphic characters
• lower - lower-case alphabetic characters
• print - printable characters
• punct - punctuation characters
• space - space characters
• upper - upper-case characters
• xdigit - hexadecimal characters

The tr command can translate lowercase characters in a string to upper case.

Example tr - upper-case a string:

tr can be used to delete named characters from a string.

Example tr - delete characters from a string:

Use tr to translate any named characters in a string to a space. When multiple named characters are encountered in sequence, they are translated into a single space.

Behavior of the -s option flag differs between systems.

Example tr - translate characters to a space:

The -s option flag can be used to suppresses extra white space in a sting.

A common problem when transferring files between UNIX and Windows based systems is line delimiters. On UNIX systems the delimiter is a new line, while Windows systems use two characters, a carriage return followed by a newline. Using trwith some redirection is one way to fix this formatting problem.

Example tr - remove carriage returns:

Use of colrm

Using colrm columns of text can be cut from a stream. In the first example, colrm is used to cut from column 4 to the end of line for each line off the pipe. Next, the same file is sent to colrm to remove columns 4 through 5.

Example colrm to remove columns:

Use of expand and unexpand

The expand command changes tabs to spaces, while unexpand changes spaces to tabs. These commands take input off of stdin or from files named on the command line. Using the -t option, one or more tab stop can be set.

Example of expand and unexpand:

Use of comm, cmp, and diff

To demonstrate these commands, two new files are created.

Create files for demonstration:

The diff command compares two files, reporting on differences between them. diff takes a number of option flags. In the following example, a default diff is shown first, followed by a diff using the -w option which ignores whitespace, then finish with an example of the -i option flag which ignores differences between upper and lower case when doing comparisons.

Examples of diff command:

The comm command compares two files, but behaves much differently than diff. comm produces three columns of output - lines only in file1 (column 1), lines only in file2 (column 2), and lines common to both files (column 3). Option flags can be used to suppress columns of output. This command is probably most useful to suppress column 1 and column 2, showing only lines common to both files, as shown below.

Example of comm command:

The cmp command also compares two files. However, different than comm or diff, the cmp command (by default) reports the byte and line number where the two files first are different.

Example of cmp command:

Using fold

Using the fold command, lines are broken at a specified width. Originally this command was used to help format text for fixed-width output devices incapable of wrapping text. The -w option flag allows specification of a line width to use instead of the default 80 columns.

Example using fold:

Using paste

The paste command is used to align files side-by-side, merging records from each file respectively. Using redirection, new files can be created by joining each record of one file, with records in another file.

Create files for demonstration:

Example 1 of paste - lines from multiple files:

There is a -s option flag used to process the files one at a time (serially) instead of in parallel. Notice below, the columns align with the rows in the example above.

Example 2 of paste - lines from multiple files:

If only one file is specified, or paste is processing stdin, by default the input is listed in one column. Using the -s option flag, the output is listed on one line. Since output is condensed to a single line, use a delimiter to separate the returned fields (tab is the default delimiter). In this example, the find command is used to locate directories where 64-bit libraries would likely be found, and builds a path suitable for appending to a $LD_LIBRARY_PATH variable.

Examples of paste - with delimiter:

Using bc

For an easy way to do math in the shell, consider bc, the “basic calculator” or “bench calculator”. Some shells offer ways to do math natively, others rely on expr to evaluate expressions. Using bc, calculations can be portable across shells and UNIX systems, just be careful around vendor extensions.

Example of bc – simple calculations:

bc can perform more than these simple calculations. It is an interpreter defining its own internal and user defined functions, syntax, and flow control, similar to a programming language. By default bc does not include any digits to the right of the decimal point. Increase precision in the output by using the special scale variable. As the example shows, bcscales to large numbers, and carries out to lengthy precisions. Use obase or ibase to control conversion base for input and output numbers. In the example below:

• obase changes the default output base (ten), converting the result to hexadecimal
• for this square root of 2 approximation, scale specifies number of digits to the right of the decimal
• support for large numbers is illustrated by calculating 2 to the 128th
• the sqrt() internal function is called to calculate square root of 2
• From ksh, compute and print a percentage

Example of bc – more calculations:

The man page for bc is detailed and includes examples.

Using split

A useful task for the split command is to break large datafiles into smaller files for processing. In this example, BigFile.dat is shown to have 165782 lines using the wc command. The -l option flag tells split the maximum number of lines for each output file. split allows for a prefix to be specified for output filenames, below BigFile_ is the specified prefix. Other options allow for suffix control, and on BSD, a -p option flag allows for splits to occur at a regular expression like the csplit (context split) command. Take a look at the man page for more information.

Example of split:

Using cut

The cut command is used to ‘trim out’ column-based sections of a file or data piped to it from stdin. It cuts data by bytes (-b), characters (-c), or fields (-f) as specified by a list. The lists of fields or byte/character positions are specified using comma separated lists and hyphens. Just specify the position or field desired if only one is needed for output. A range of fields can be specified using a hyphen such that 1-3 prints fields (or positions) 1 through 3, -2 prints from the beginning of the line up to field 2 (or byte/character 2), and 3- specifies cut to print field (or position 3) through the end of line. Multiple fields can be specified using a comma. Some other useful flags are -d to specify a field delimiter, and -s, to suppress lines without delimiters.

Examples of cut:

Using uniq

The uniq command is typically used to uniquely list lines from an input source (usually a file or stdin). To operate properly, duplicate lines must be contiguously positioned in the input. Normally input to the uniq command is sorted, so duplicate lines are aligned. Two common flags used with the uniq command are -c to print the count of the number of times each line appeared, and -d can be utilized to display one instance of duplicate lines.

Examples of uniq:

Using sort

To arrange rows in stdin or a file in a particular order, such as alphabetic or numeric, the sort command may be used. By default output from sort is written on stdout. Environment variables such as LC_ALL, LC_COLLATE, or LANG can affect the output of sort and other commands. Notice how the example file shows 2 separate duplicate records – one dupe for IBM, the other dupe is a blank line.

Example of sort – default behavior:

sort has a great flag that can substitute in place of the uniq command in many circumstances. The -u option flag sorts the file, removing duplicate rows so a listing of unique rows of output is produced.

Example of sort – sort unique:

Sometimes, the reverse ordering of input is desired. By default sort arranges the order from smallest to largest (for numeric) and in alphabetical order for character data. Use the -r option flag to reverse the default sorting order.

Example of sort – sort reverse order:

Different situations call for a file to be sorted on certain fields or “keys”. Fortunately sort has the -k option flag allowing a sort key to be specified by position. Fields are separated by whitespace by default.

Example of sort – sort with a key:

When case sensitivity is an issue, sort provides the -f option flag, which ignores case when doing comparisons. When combining multiple flags as shown below, some versions of UNIX need these flags specified in a different order.

Example of sort – sort ignoring case:

So far, all the sorts have been the alphabetic variety. When data needs to be sorted in numeric order, use the -n option flag.

Example of sort – numeric sort:

Some inputs may use characters other than white space to separate fields in a row. Use the -t option flag to specify a non-default delimiter such as a comma.

Example of sort – sort on field using non-default delimiter:

Using join

Anyone familiar with writing database queries recognizes the utility of the join command. Like most UNIX commands, output is displayed on stdout. To “join” files together, specified fields from two file are compared on a line by line basis. If no fields are specified, join matches on fields from the beginning of each line. The default field separator is whitespace (some systems it’s simply a space or adjacent spaces). When a field match occurs, one line of output is written for each pair of lines with matching fields. For legitimate results, both files should be sorted on the fields to be matched. Not all systems implement join the same way.

This example uses -t to specify a field separator, and demonstrates joining two files on the first field (default) delimited with commas. Database operators would recognize it as an inner join, displaying only matching rows.

Example of join – using non-default field delimiter:

To specify fields on which to“join” in each file the -j[1,2] x option flag (or simply -1 x or -2 x) can be used. Option flag -j1 2 or -1 2 specifies the second field of file one, the first file listed on the command. This example shows how to join the files based on field 1 in the first file and field 2 in second file, also an inner join only matching rows.

Example of join – specified fields:

Keeping with spirit of database related examples, flags can be used to produce a left outer join. A left outer join includes all rows from the left first file or table and matching rows from the second file or table. Use -a to include all rows from the specified file.

Example of join – left outer join:

Full outer joins include all rows from both files or tables, it does not matter if the fields match. A full outer join can be done by specifying both files with the -a option flag.

Example of join – full outer join:

Using sed

The stream editor sed is a useful text parsing and manipulation utility handy for doing transformations on files or streams of data. It reads in text one line at a time, applying the specified commands on the line of text. By default output goes tostdout. Commands sed uses perform operations such as deleting text from a buffer, appending or inserting text into a buffer, writing to a file, transforming text based on regular expressions, and more.

A basic example of sed substitution shows the -e option flag being used to specify the expression or the edit script. Multiple expressions or edits can be specified for a single sed execution. Notice the components of the sed edit. The “s” at the front of the edit indicates this is a substitution command. Using a “/” as a delimiter, the pattern “IBM” to replace is indicated first. Next, the replacement pattern appears between two “/” delimiters. Last, the “g” indicates to make the change globally in the current text buffer. The third demonstration in this example illustrates a combination of three edits: replace backslashes with slashes, spaces with underscores, and remove colon characters – notice how the backslash “\” characters are escaped.

Example of sed – basic substitution / multiple edits:

In the following example a file is set up to demonstrate another feature of sed. Besides substitution, filtering is another frequent use of sed. The UNIX grep command is a commonly employed filter, it is not unusual to discover multiple ways to manipulate text on the command line. This example shows how to use the sed delete command to remove lines beginning with “#” or whitespace then “#”. An example of grep utilizing the same pattern is shown for reference.

Example of sed - filtering:

To better understand sed behavior a few more patterns are demonstrated. A file is created so the patterns can act upon some text. The first sed pattern shows how to remove the last 4 characters from strings (filenames) listed in the file. Next, the pattern removes all characters to the right of the dot “.” indicating a file extension. A pattern to remove blank lines is shown. A special character, the ampersand “&” allows the search pattern to be used as part of the output. In this example, IBM is part of the input pattern, and is specified as part of the output using the ampersand. The last pattern demonstrated in this series shows how sed can be used to remove carriage returns from a text file transferred from a Windows-based system. The “^M” is entered into the script or on the command line by pressing first the control-v, then pressing control-m. Please note terminal characteristics may affect entry of the control-v, control-m combination.

Example of sed – more patterns:

sed commands can operate on specified address ranges. The following examples show some ways addressing can be controlled by sed. The “-n” option flag suppresses the default behavior of sed to display every line of input as a part of the output. In the first example, sed operates on lines 4 through 7 from the file. Notice how only the first listed table-row from the file is displayed (lines 4 to 7). Next, sed displays only the first and last lines in the file. Some versions of sed allow for patterns to specify a range of addresses to apply the command. Notice in the output, commas are only removed from the table, not from the comments.

Example of sed – address ranges:

Patterns inside an expression can be grouped and then referenced as part of the output. This can prove useful in a number of contexts such as swapping values or positional variables. The parentheses are used to mark out patterns in the expression, and must be escaped with a backslash \(pattern-here\). The pattern is referenced elsewhere in the expression by using a \n where n is the number of the pattern in order of the marked patterns. Decomposing this expression into components makes it easier to recognize how it works:

Example of sed – grouping patterns:

Using awk

The awk program can be a handy text manipulator – performing jobs such as parsing, filtering, and easy formatting of text. It takes its input on stdin or from files and by default displays output on stdout. A variety of releases are available for awkunder different names like nawk and gawk. Behavior between versions and vendor releases of awk vary. awk is different than the other commands reviewed in this article, because it is a programming language. This language provides internal functions for math, string manipulation, flow control, and text formatting. Programmers can also define their own functions, creating libraries of user-defined functions or stand-alone scripts. Because awk contains so many features to demonstrate only a few examples are shown.

At first, in this example, awk is used as a filter to print only full file systems from a Linux system. By default awk uses whitespace to identify separate columns. The example examines column 5 since it shows the percentage of disk space used. If the disk utilization is 100%, the first example prints the record on stdout. The following statement extends the first to format a message, perhaps to send in an email or write as part of a message to a log file. Then an example is shown on how to create a match using numeric comparison.

Example of awk - filter:

Sometimes data is not delimited by whitespace. Take for instance the /etc/passwd file, delimited by the colon “:” character. This example shows how awk uses the -F flag to print the username and UID of the first 5 entries listed in /etc/passwd. Next, the substr() function of awk is demonstrated by printing the first three characters of column 1 in the /etc/passwd file.

Example of awk - field separator / string function:

Many times system administrators or programmers write their own awk scripts to perform some sort of job. Here is an example of an awk program to average numbers found in the third column of a file. The math is done manually by adding the column 3 data up in the total variable. NR is a special internal variable awk uses to keep track of how many records were processed. The average for column 3 is obtained by dividing the total variable by NR. The program displays intermediate results and data so the logic is easier to follow.

Example of awk – program / math:

Shell-based string operations

A shell can be a powerful programming language. Similar to awk, shells offer a wide variety of options for string operations, math functionality, arrays, flow control, and file operations. Below are some examples showing how to extract parts of a string from one side. The operation does not change the value of the string, but echoes what the result would look like, and is often used as an assignment to a variable. Use the percent “%” sign to truncate right of the pattern, and use the hash mark “#” to truncate left of the pattern.

Example of shell script – string extraction:

As an example, a system administrator may need to change the extension of a set of .jpg files to all lower-case letters. Since UNIX servers are case sensitive, some applications may require the lower case extension, or maybe the administrator is simply trying to standardize file extensions. Changing a large number of files by hand or through a GUI could take hours. A sample shell script showing a method to solve this problem follows. The example is made up of two files. First is setup_files.ksh, used to set up a sample directory tree, and populate the tree with some files. It also creates a list of files needing extensions changed. The second script called fix_extension.ksh reads in the list of files, changing the extensions where appropriate. As part of the mv command, the % string operator is used to truncate to the right of the last dot “.” in the filename (truncates the extension). Both scripts also use the find command to display what was accomplished after the run.

Example of shell script – change file extension:

In the spirit of creating useful and reusable tools, the example for changing file extensions should be more generalized. Some improvements that come to mind are to pass in a stream of filenames to be changed, such as part of a pipeline. Option flags could be added to specify file extensions (like .mp3 or .mov) to change, and how to format the file extension such as lower, upper, or mixed case. The possibilities are limited only by the programmer’s imagination and time.

Summary

UNIX offers a wide variety of tools to do text parsing natively, in many cases without the need to rely on special interpreters which may not be installed. This article is rather broad in its survey of commands when compared to depth of their usage. The commands in this article are partially demonstrated, some systems implement flags or behavior differently than other systems. UNIX certainly offers more commands and ways of accomplishing the same tasks – “There‘s more than one way to do it”.

[转帖]Introduction to text manipulation on UNIX-based systems

原文：https://www.cnblogs.com/jinanxiaolaohu/p/10583763.html