CST 8152 - Assignment Three

Topic: A Lexical Scanner.

Deliverables for this assignment:

• Due Date: 12:00 noon, Friday March 6.  The one-week late penalty is waived for this assignment.
• Hand in, on paper:
  1. the regular expression, transition diagram, and your transition table for C Language Identifiers
  2. the fully documented source listing of the modified Scanner.c source file containing the Scanner and its support functions
  3. your test results, output, and documentation from testing your Scanner
  4. your implementation of the function my_close()
• Do not hand in anything other than the above items.
• The submission must follow the online course submission standards, including an Assignment Submission Label and a Table of Contents. (The Label and Contents may be on the same exterior cover page.)
• Include a description or listing of your input test file(s) (and output, if appropriate) showing how you tested your program.  Don't kill a hundred trees; submit descriptions and short excerpts of your testing input and output if the files are large.   The intent of your test submissions is to convince me that your program works. Short excerpts of actual program output are usually more convincing than mere words, so give me at least some sample output.
• No demonstration is required for this Lab; however, I am always pleased to see working code during lab hours.

Purpose:

This assignment implements and tests the foundation of your lexical analyser: The Scanner. The input to the lexical analyser is a stream of source text. The output of each call to the Scanner is a single Token, to be read, in later assignments, by The Parser. This assignment only counts and prints the tokens; it doesn't try to parse them or build a grammar.

Use the main() program given in the source directory to test your Scanner.  The main() function of this assignment is a modification of that in the previous assignment. Instead of looping reading and printing lines using my_prompt(), we loop reading and printing lexemes using the new scanner() function.  The Scanner will return a TT_EOF token when end-of-file is reached.

The Scanner uses a table-driven deterministic finite automaton (DFA) to recognize the lexemes. To change the lexemes recognized; simply change the tables.  We will be changing the tables in subsequent assignments.

You are given most of the code for the Scanner.c file and all of the code for the Scanner.h file (and all the other header files) in the source directory. Your task is to finish the coding of Scanner.c so that your Scanner recognizes and returns Identifier (TokenType TT_ID) tokens and the end-of-file (TokenType TT_EOF) token.  (The TokenType enum is defined in Scanner.h.)  Identifiers in this assignment follow the rules for C language identifiers.  The TT_EOF token is returned at end-of-file when the scanner is called and no other tokens are available.

Your Scanner may also, at your discretion, recognize and return other TokenTypes, though this is not required (yet).  The given main() driver program will print any TokenType and lexeme returned by your Scanner.  If you don't handle them another way, have your Scanner silently skip over characters that are not part of an identifier.

Instructions:

• This assignment uses a driver function main(), given below, that is similar to the one used in the previous assignment. The driver repeatedly calls scanner() instead of my_prompt().
• Be sure you take advantage of any code that I write for you!  Don't reinvent it if you don't have to.
• Here is a suggested order to tackle this assignment. You don't have to do things in this order.

Part 0 (optional but highly recommended)

Get the MEM package and install it in a simple program.  Learn how it works.   Then install it in your Assignment 2, and use it all subsequent assignments, including this one.

Part 1

Restructure your program source into separate modules of related functions, as outined in the Course Announcements news group.  See my example header files in the source directory, below.  You are not obligated to use these header files; I provide them as samples of how I structured my own version of the assignments.  You will find various macros that I use to simplify my code defined in my header files, e.g. BF_EMPTY().

Part 2

Write a my_close() function according to the specifications below. (You might want to add it to Assignment 2 first, to make sure it works, before you add it to your new assignment.)  The typical use for my_close() is to close files opened by my_open().  The new main() driver given to you needs this function.

Part 3

Copy the given Scanner.c file from the source directory.  Finish the coding of Scanner.c so that your Scanner recognizes and returns Identifier (TokenType TT_ID) tokens and the end-of-file (TokenType TT_EOF) token.  (The TokenType enum is defined in Scanner.h.)  Identifiers in this assignment follow the rules for C language identifiers.  If you do not handle them any other way, silently skip over characters that are not part of an identifier.  The TT_EOF token is returned at end-of-file when the scanner is called and no other tokens are available.

The key places you need to write code are marked in Scanner.c with "//FIXME--" tags in the source file.  Look for them.  If you insert the correct code in these exact locations, you do not need to make any other changes to the file, or to any other files.

Your Scanner may also, at your discretion, recognize and return other TokenTypes, though this is not required.  The given main() driver program will print any TokenType and lexeme returned by your Scanner.  Feel free to enhance your DFA to recognize C comments, integers, floating-point numbers, quoted strings, etc., using the transition diagrams given in the text or in class.  Be on the alert for "put the character back" accepting states!

Note that you cannot push back the EOF flag using ungetc().  Your scanner_ungetc() must take this into account.

Part 4

Test your program on everything you can think of.  Document your testing strategy and submit a summary of what you did and why.

Specifications for my_close()

This function was omitted by mistake from the earlier assignment.  It is the companion to my_open() and it has the following prototype and specifications:

        void
my_close(
        FILE *fd,       /* IN: stream descriptor to close */
        char *fbuf      /* IN: file name associated with fd */
);

• Validate the function arguments.  (Check for NULL pointers, etc.)
• Close or flush the stream descriptor given, and check that the close or flush succeeded.   Print an error message, using the given file name, if the close or flush failed.
• If the stream descriptor is one of the three standard pre-opened FILE * descriptors, do not close it.  (Closing the standard descriptors is bad style in C programs!)  Flush the descriptor using fflush() instead (and, of course, check its error return).

Source directory

Copy these source files and modify them according to the assignment instructions.  The header files are examples only; you are not obligated to use them; however, you must divide up your program source into more than one module.

Ian D. Allen CST8152 Home Page