My "find" commands run too long.  How can I speed them up by skipping directories?

Solution:  Use the –prune option with find.  Just make sure you put your –prune before your positive criteria.

 Example 1:

 find / -name usr –prune –o –name "sh*" –print

 The above command starts at the top of the directory structure (/) and then does one of two things.  Either:

     If the name is usr (as in /usr), skip it.  Or,

    if the name is "sh*" (begins with sh), print its name.

 This will circumvent the entire /usr directory chain.

 Example 2:

 find / -name usr –prune –o \

          -name proc –prune –o \

          -name tmp –prune –o \

          -name "1*" –print  2> /dev/null

Notes:  The \ as the last character on a command line is the shell's line continuation character.  Also, remember if you throw away your errors (2> /dev/null), you'll be disregarding any command syntax errors as well.  Just make sure you have the command format correct before you add the 2> /dev/null.

How do I detect files that contain carriage-returns? (usually as the result of a binary ftp from Windows to UNIX).

Solution:  A carriage-return is an octal 015.  We can use the "od –b" command with grep to determine if the file contains carriage-returns.

 cat my.file | od –b | cut –f 2- -d" " | grep 015

 If you see any output, your file contains carriage returns

 Following is script file detcr (detect carriage returns):

 #!/usr/bin/ksh
for FILE in $*
do
    [ ! -r $FILE -o ! -f $FILE ] && { echo Skipping file $FILE; continue; }
    cat $FILE | od -b | cut -f2- -d" " | grep -q 015
    [ $? -eq 0 ] &&  echo File $FILE contains carriage returns
done 

Explanation: 

In the above, the script looks at all of your command line arguments (should be ASCII files to interrogate for carriage returns.)  If the file is not readable or the file is not a regular file, skip it.  Type out the contents of the file, pipe the output into od –b, strip off the portion of the od –b output that is a byte offset (with the cut command) , and then check for a 015. 

If the 015 was found by grep, grep's return code will be 0 (success.)  If so, print that the file contains carriage returns.

Note:  If you pass binary files, it is likely that you will find 015 patterns, even though they will not be treated as carriage-returns.

I have several ways of accomplishing my UNIX task.  Which is the most efficient?

Solution:  Write a little, reusable test harness to iterate over your algorithms multiple (or many) times.  I have a test harness named mytester.  First, the invocation syntax:

time mytester          #  Run my algorithm 100 times (the default.)  Give me statistics.
or
time mytester 500   #  Run my algorithm 500 times.  Give me statistics

Results:

real    0m6.79s
user   0m0.14s
sys    0m0.56s

Following is script file mytester (my testing harness):

#!/usr/bin/ksh
COUNT=$1    #  Place argument 1 in variable COUNT
echo "$1" | grep -qv "[0-9]"  && COUNT=""  #  if COUNT contains any non-numerics, blank it out
[ "$COUNT" ] || COUNT=100                      #  Default COUNT to 100 if blank
LIMIT=0
while [ $LIMIT -lt $COUNT ]
do
#  Place test code here *********************************
#  As a sample, 3 different ways to uppercase a string
#  Keep two of the statements commented out for each invocation of this script

echo "abc" | tr "a-z" "A-Z"
#     echo "abc" | awk '{print toupper($1)};'
#      echo "abc" | perl -e 'print uc( <> )';

#  End test code here ***********************************
((  LIMIT += 1 ))
done
echo Executed $COUNT times

Explanation:

In the above, I've put 3 different variations of an uppercasing algorithm (shaded in gray.)  That's where you'll put whatever test code you interested in evaluating.  Two of the methods are commented out.  I'll run mytester 3 different times with a different statement each time.  The invocation that produces the smallest times is usually the most efficient.

In the above example, all three methods produced extremely similar internal time measurements (sys and user.)  However, the Perl version was significantly slower in real time, most likely due to the size of actually loading the Perl executable (which is much larger than awk or tr) many, many times.

I want to quickly get some information about a particular command, regardless of where on the file system it is stored.

Solution:  Use the which command (in reverse tiks with the command you're interested in.)

ls –l  `which ps`

file  `which ps`

strings `which ps`

 Solution 2 (for a little less typing):  Create an alias for which named w.

alias which=w

Then,

ls –l  `w ps`

file  `w ps`

strings `w ps`

Notes:  This tip, obviously, only works with commands that are physical commands (stored on the file system), not built-ins, keywords, or functions.  Also, the command would have to be in a directory that is noted in your PATH variable (most should be.)

My shell terminates when I type <Ctrl>D.  I don't like that.  How do I make it stop?

Solution:

First, take a look at your shell's current option settings: 

set –o   #   set dash lowercase Oh 

Look for the option named "ignoreeof".  If it is turned OFF, the shell will not ignore EOF (in other words, the shell will respond to a <Ctrl>D as an exit.)

To turn "ignoreeof" ON, so the shell will ignore the <Ctrl>D, your command is: 

set –o ignoreeof

Notes:  You can, of course turn ON any of your listed options in the same way we did above.  To turn a shell option OFF: 

set +o option_name.

Also, some of the options have shortcuts for turning them on and off.  For example: 

set –x

is the same as: 

set –o xtrace

The shell options are NOT  inherited by subsequent shells.  For that type of behavior, you would need to put your option settings in .kshrc or .bashrc.

My terminal is hung up.  Nothing I type is visible.  No output is visible.  What's up with that?

Possibility 1:
You've inadvertently typed <Ctrl>S (I accidentally do this about once a week.)  That control sequence is usually defined to stop the transmission between your IO device and the shell.

Solution 1:
Type <Ctrl>Q.  Restart flow.  (By the way, the <Ctrl>S / <Ctrl>Q combination comes in handy for stopping and starting a long stream of output.)

Possibility 2:
A job is actually running in the foreground that you're not aware of.

Solution 2:
Type <Ctrl>Z to see if you can pause (STOP) the job.  If so, you should get to your launching shell.  From there, you can do the "ps" or "jobs" command to see what's going on.  If all is OK and you want to restart your job, use "fg" to continue it in foreground or "bg" to continue it in background.

Possibility 3:
The administrator has sent a STOP signal to your shell.

Solution 3:
This is a tough one.  You really don't have much control over this problem.  Fortunately, this shouldn't happen too often.  Logon to another session.  Type the command:

 ps –u  your_user_id  –l  (That's a dash-ell.)

 Look under the "S" column for any "T" values.  If it's listed on a line that contains a shell, that implies that that particular shell is Stopped.  Shoot it a Continue signal to see if that wakes it up:

 kill –CONT PID (That shell's process id.)

Note:  The "ps" command in solution 3 might be a good candidate for an alias named "psu".

 alias psu="ps –u $LOGNAME –l"

"Show all of the jobs that I own in long format"

I want to get a list of all my Perl Scripts, C Programs, ASCII files, etc.

The Problem:
I need to isolate all of my files, for some reason, of a particular type.

Step 1:  Run the "file" command on all of your files and find a very tight pattern that is unique to the file type you're interested in.

file *

Step 2:  Then, take the output of that command and grep for your tight pattern:

file * | grep " ASCII C Program"

Step 3:  Then, once you're sure you've not gotten too little or too much, cut out everything but the file name:

file * | grep " ASCII C Program" | cut –f1 –d:

Step 4:  Now, whenever you want to feed those file names to another command, wrap the entire sequence in reverse tiks and use them.

Example 1 (Copy all my Perl Scripts to my /home/mark/perl directory)

cp  -i      `file * | grep "perl script" | cut –f1 –d:`     /home/mark/perl

Example 2 (Archive to a tar file all of my English text files)

tar  -cvf  english.tar      `file * | grep "English text" | cut –f1 –d:` 

Example 3 (An alternative to the reverse tiks)

tar  -cvf  english.tar      $(file * | grep "English text" | cut –f1 –d:) 

Note:  Make sure you understand that reverse tiks have a quite different function than the similar looking single-quote.  Reverse tiks always have a command or command sequence inside of them that, should, return some output.  Reverse tiks pass that output to something else.

Next time: We'll see how to automate the above process in a tricky script file that uses part of the script file name as the pattern to search for.

I want to remove some files with names beginning with a hyphen.

The Problem:
Somehow, I've inadvertently created a file named –x, or –y, or –z.   Every time I attempt to remove it with rm –x, rm thinks –x is an option, not a file name.

For example:  I type the following command sequence:

rm –x

and receive this error:

rm:  rm invalid option -x

Solution (fully qualify the file name)

rm /home/barney/-x

#  or

rm ./-x    #  The file is right here, in my current directory. (dot-slash prefixed)

Explanation:  A fully qualified file name tells the shell exactly where the file lives.  You fully qualify a file by prepending the file's directory in front of the file name.style="color: black;">  The shell assumes that unqualified file names live in your current directory.

Additional note:  You can certainly always qualify your file names for any command whether you need to or not (as a matter of fact, it's considered good practice in shell scripts.)  However, the non-full-qualification of file name reference, allowed by the shell, sure saves a lot of typing.

I want to save a man page.  I don't want to see those nasty backspaces.

The Problem:
I know how to save a man page via redirection.  How do I make it more readable? 

For example:  I type the following command sequence: 

man ps > psman.out 

Then, I pull up psman.out in vi, and there are tons of strange character sequences.

Solution (pipe the output through col –b before writing it)

man ps | col –b > psman.out

Explanation:
The col command with the –b option reads its input from its source (the man command, in this example) and assumes that the output device (STDOUT) has no ability to translate backspaces.  Therefore, it removes the backspaces and replaces them with one occurrence of the last character following the last backspace in the sequence. 

The backspace characters are used in the man command's output in case the document is actually sent to an impact printer (dot-matrix, daisy-wheel, line, etc.)  It's sort of the poor-man's way of bolding (just keep hammering the same character in the same place a couple of times.)

Additional note:  If you're not happy with the man command's paging mechanism (more, less, pg, etc.) you can change it to whatever you like with the command:

export PAGER=more
export PAGER=less
export PAGER=pg
export PAGER=cat

This essentially disables paging 

Just make sure you point the PAGER variable to a valid paging command.

I want to store my command history in separate files by month or day.

The Problem:
I would like to store my command history in smaller files that I can identify by month or day.  How can I do this?

For all of the solutions below, include the solution in your .profile (or .bash_profile if you're running a bash.

Solution 1 (Store history in monthly files):
       export HISTFILE=.sh_history_`date +%Y_%b`
Solution 2 (Store history in daily files):
       export HISTFILE=.sh_history_`date +%Y_%m_%d` 

Explanation:
Your HISTFILE variable controls which physical file your command history is written into.  By default, it is usually written into a hidden file named .sh_history.

Solution 1
Your history will be written, this month, into a file named .sh_history_2010_Jan.
Beginning Feb 1^st: .sh_history_2010_Feb.

Solution 2
Your command history will be written, today, into a file named .sh_history_2010_01_25.  Beginning tomorrow: .sh_history_2010_01_26.

Notes:  Make sure that you use reverse tiks around the date command.   Use the solutions above as a guide for formatting your command history file name as you would like it.  Check the man pages for the date command to see other format specifiers you might be interested in.

After you've made the change in your .profile or .bash_profile, on next login type the following to verify the syntax of your HISTFILE assignment:

echo $HISTFILE

Then, after a few commands, run the history command to verify your command history:

history

How to quickly look for patterns through an entire directory chain.

The Problem:
I can identify a file by something in the file’s contents.  However, I don’t remember the file’s name or location.  Essentially, I need to “grep” across multiple directories.

Solution 1:
       find  /home  -exec  grep  -H  “Mr. Peepers”  {}  \;  2> /dev/null
 Solution 2:
       find  /home  –name  “p*.doc”  -exec  grep  -H  “Mr. Peepers”  {}  \;  2> /dev/null
 Solution 3:
       find  /home  -exec  grep  -H  “Mr. Peepers”  {}  \;  2> /dev/null  |  cut  -f1  -d: 

Explanation:

Solution 1
Use the find command with grep.   In the above solution, /home is the starting point. We will look in all directories in /home and beneath.  We will take the default find criteria of “find all files” (-name  “*”). 

-exec tells find to execute a command for each file, in this case, the grep command. 

-H is the grep option to display the file name, in addition to the record that was found. 

“Mr. Peepers” is the pattern we are looking for in each record of the files. 

Don’t forget, when you use –exec, the {} is the placeholder that receives the file name of the “found” file.  Make sure you put a space before the protected semi-colon (\;). 

Disregard the error messages (2> /dev/null)  This is a good extension for many commands if you are not interested in viewing errors. 

Solution 2
If you know a portion of the file’s name, add the –name “pattern” to limit the grep to only those files that match that pattern.  Since grep will otherwise search through every file, this technique will drastically improve the speed of the search.  Of course, this would only work if you were sure of the filename match. 

Solution 3
If you just want the file name without the actual text of the record printed, you can pipe the output to a cut command to retrieve only the first portion of the value returned from grep.  After cut, put a –f1 –d: (that is, dash eff one dash dee colon). 

Notes:  Some grep versions support a recursive option which may be a more concise command.  Check your man pages for details on your version.  The above solution should work on all UNIX platforms.