==================================================
Writing a script to test another script or program
==================================================
 - Ian! D. Allen - idallen@idallen.ca - www.idallen.com

One use of shell programming is to write scripts that themselves execute
other programs or scripts, and verify that correct output was produced.
One script is called the "tester" script, the other script or program is
"the script or program being tested".

The simplest kind of tester script is one that is designed to test one
other program or script.  The tester program contains code to specifically
validate that one program or script; it will not work to test different
programs that have different outputs.  It is a single-purpose tester
script, and this is the kind of tester program we will write here.

For example, suppose we have a program or script named hello.sh that is
supposed to produce the words "Hello World, XXX!" on standard output,
where XXX is the first argument to the hello.sh script:

    $ ./hello.sh Batman
    Hello World, Batman!

    $ ./hello.sh "Ally McBeal"
    Hello World, Ally McBeal!!

    $ ./hello.sh 
    ./hello.sh: Expecting 1 argument, found 0 ().

We can write the following tester.sh script to run the above hello.sh
script and make sure that hello.sh produces the correct output:

    #!/bin/bash -u
    # This tester script tests the output of the hello.sh script.
    # Syntax:  $0   (no arguments)
    # Purpose:  A "tester" script that makes sure hello.sh produces
    #           the correct output on standard output, and produces
    #           nothing on standard error.

    ... insert usual header lines for PATH, umask, etc. here ...

    # Run the hello.sh script with one argument "Ally McBeal".
    #  - save standard output of hello.sh in file "out"
    #  - save standard error of hello.sh in file "errs"
    #
    ./hello.sh "Ally McBeal" 1>out 2>errs

    # Make sure the correct output was produced in the file "out".
    # Print a message, dump the contents of the two files, and exit if not.
    #
    if fgrep "Hello World, Ally McBeal!" out >/dev/null ; then
     	echo "Yes, I found the correct output in file 'out'."
    else
	echo "No, I did not find the correct output in file 'out'."
	echo "STANDARD OUTPUT:"
	cat out
	echo "STANDARD ERROR:"
	cat errs
	exit 1
    fi

    # The standard output is correct (checked above).  Make a second test:
    # Make sure nothing was put into file "errs" (nothing on standard error).
    # Print a message and exit if the file containing stderr is not empty.
    #
    if [ -s "errs" ] ; then
	echo "Unexpected output found on Standard Error (in 'errs')"
	echo "STANDARD OUTPUT:"
	cat out
	echo "STANDARD ERROR:"
	cat errs
	exit 1
    fi

    # We only get to this part of the script if all tests passed.
    #
    echo "The hello.sh file produced the correct output."
    exit 0

We can now run the tester.sh script to test that the hello.sh script
produces the correct output:

    $ ./tester.sh
    Yes, I found the correct output in file 'out'.
    The hello.sh file produced the correct output.

One inflexibility of the tester.sh script is that it can only test
the hello.sh script in the current directory - the name "./hello.sh"
is hard-coded into the tester.sh script.  It would be more useful to
be able to test hello.sh scripts no matter where they were, and even if
they weren't named exactly "hello.sh".

Let's make the tester.sh script take the name of the script to test as
a command line argument.

Here are the changes we made to make the tester.sh script process the
script given as its first argument, instead of always processing the
same hard-coded "./hello.sh" script:

    #!/bin/bash -u
    # This tester script tests the output of the argument script.
    # Syntax:  $0 scriptname
    # Purpose:  A "tester" script that makes sure argument script produces
    #           the correct output on standard output, and produces
    #           nothing on standard error.

    ... insert usual header lines for PATH, umask, etc. here ...
    ... insert code to test for missing argument, too many, etc. ...
    scriptname="$1"
    ... insert code to validate that the argument is a readable script ...

    # Run the argument script with one argument "Ally McBeal".
    #  - save standard output of the argument script in file "out"
    #  - save standard error of the argument script in file "errs"
    #
    "$scriptname" "Ally McBeal" 1>out 2>errs

    # Make sure the correct output was produced in the file "out".
    # Print a message, dump the contents of the two files, and exit if not.
    #
    if fgrep "Hello World, Ally McBeal!" out >/dev/null ; then
     	echo "Yes, I found the correct output in file 'out'."
    else
	echo "No, I did not find the correct output in file 'out'."
	echo "STANDARD OUTPUT:"
	cat out
	echo "STANDARD ERROR:"
	cat errs
	exit 1
    fi

    # The first output test passed (above).  Make a second test:
    # Make sure nothing was put into file "errs" (standard error).
    # Print a message and exit if the file is not empty.
    #
    if [ -s "errs" ] ; then
	echo "Unexpected output found on Standard Error (in 'errs')"
	echo "STANDARD OUTPUT:"
	cat out
	echo "STANDARD ERROR:"
	cat errs
	exit 1
     fi

     # All tests passed.
     #
     echo "The '$scriptname' file produced the correct output."
     exit 0

The above version of the tester.sh script is more useful, since we can
now use one tester.sh script to test different versions of hello.sh with
different names and in different places:

    $ ./tester.sh ./hello.sh
    Yes, I found the correct output in file 'out'.
    The ./hello.sh file produced the correct output.

    $ ./tester.sh ./badhello.sh
    No, I did not find the correct output in file 'out'.
    STANDARD OUTPUT:
    helo wurld Ally McBeal
    STANDARD ERROR:

Note that making the tester.sh script accept the script to test as
a command line argument has not made the tester.sh script any more
general - it still only tests scripts that produce "Hello World, XXX!"
kinds of output.  Accepting a command line argument merely gives us
flexibility in where those scripts are located, since they don't have
to be located in the current directory.

However, our version of tester.sh only runs and tests the hello.sh script
with a single argument ("Ally McBeal").  The tester.sh script isn't
testing what happens if we run hello.sh with zero arguments or with more
than one argument.  To fully test hello.sh, the tester.sh script must
also run hello.sh with zero arguments and with more than one argument,
and test for the correct output in both those cases as well.

To summarize: to fully test hello.sh, the tester.sh script must execute
hello.sh with zero arguments, with one argument (we already have the
code for this, above), and with more than one argument (e.g. two).

The wrong way to make tester.sh perform three tests on hello.sh is to
duplicate all the tester.sh code three times over.  Much of the code is
common among the three tests; we should put the common code into shell
script functions and write that code only once.  Let's sketch out what
the new version of tester.sh should look like.

When the tester.sh script performs a test on hello.sh, either it expects
the test to generate correct output on standard output (saved in file
'out'); or, it expects the test to trigger an error message from hello.sh,
in which case we expect an error message to appear on standard error
(saved in file 'errs').

Let's create one function, named ExpectGoodOutput, to test for good
output saved in file 'out'.  Let's create a second function, named
ExpectBadOutput, to test for the error message saved in file 'errs'.
Both functions will take as one of their arguments the name of a pattern
(a regular expression pattern) to search for in the output file generated
by running the hello.sh script being tested.

The ExpectGoodOutput function will use grep to find a pattern matching
good output in the standard output file ('out'), and the ExpectBadOutput
function will use grep to find a pattern matching a correct error message
in the standard error file ('errs').  If the pattern is found, the test
passes and the hello.sh program is producing the expected output.

It would be nice to combine both functions that use grep into one
function; but, the ExpectGoodOutput function that tests for good output
must also make sure nothing was written to the standard error file
('errs').  Having two functions makes things simpler, at the cost of a
little bit of code duplication.

Start of functional description of tester.sh program:

    Define function ExpectGoodOutput:
	# use grep to find a pattern in the standard output file
	Function arg 1 "testname": name of test being performed
	Function arg 2 "stdout": name of file containing standard output
	Function arg 3 "stderr": name of file containing standard error
	Function arg 4 "regexp": pattern to look for in standard output file

	Use grep to search for $regexp in file $stdout (not in $stderr!)
	 - if not found, print error message, dump the contents of the
	   $stdout and $stderr files, and then return a non-zero status

	Test to see if the $stderr file is empty
	 - if not empty, print error message, dump the contents of the
	   $stdout and $stderr files, and then return a non-zero status

	Print a success message saying $testname PASSED

	Return a zero status (success)

    End definition of function ExpectGoodOutput

    Define function ExpectBadOutput
	# use grep to find a pattern in the standard error file
	Function arg 1 "testname": name of test being performed
	Function arg 2 "stdout": name of file containing standard output
	Function arg 3 "stderr": name of file containing standard error
	Function arg 4 "regexp": pattern to look for in standard output file

	Use grep to search for $regexp in file $stderr (not in $stdout!)
	 - if not found, print error message, dump the contents of the
	   $stdout and $stderr files, and then return a non-zero status

	Print a success message saying $testname PASSED

	Return a zero status (success)

    End definition of function ExpectBadOutput

    Start of main program for tester.sh

      Get one input argument:
        Make sure tester.sh has one script name to process: $scriptname
	 - prompt for and read it if there is no command line argument
	 - print error and exit if more than one command line argument

      Validate the input argument:
	Make sure the $scriptname is a readable, executable file
	 - print error and exit if it is not

      Run three tests on $scriptname:

        1. Test number one (no arguments):
	   Execute $scriptname with zero arguments and save standard output
	     in file 'out' and save standard error in file 'errs'
	   Call the ExpectBadOutput function and pass it four arguments:
	     Function arg 1: "Test one (no arguments)"
	     Function arg 2: out (the file name from above)
	     Function arg 3: errs (the file name from above)
	     Function arg 4: a pattern to look for in the error output 'errs'

        2. Test number two (one argument):
	   Execute $scriptname with one argument and save standard output
	     in file 'out' and save standard error in file 'errs'
	   Call the ExpectGoodOutput function and pass it four arguments:
	     Function arg 1: "Test two (one argument)"
	     Function arg 2: out (the file name from above)
	     Function arg 3: errs (the file name from above)
	     Function arg 4: a pattern to look for in 'out'

        3. Test number three (two arguments):
	   Execute $scriptname with two arguments and save standard output
	     in file 'out' and save standard error in file 'errs'
	   Call the ExpectBadOutput function and pass it four arguments:
	     Function arg 1: "Test three (two arguments)"
	     Function arg 2: out (the file name from above)
	     Function arg 3: errs (the file name from above)
	     Function arg 4: a pattern to look for in the error output 'errs'

    End of main program for tester.sh
End of functional description of tester.sh program.

Test two, above, and the function ExpectGoodOutput contain most of the
code we have already written for our original version of tester.sh.

The other two tests and the ExpectBadOutput function use similar code
to look for a pattern that matches a correctly-worded error message that
should appear in the standard error file ('errs').

With tester.sh performing these three tests on its hello.sh script
argument, the program is complete.