% Unix/Linux and Shell Basics -- miscellaneous small topics % Ian! D. Allen -- -- [www.idallen.com] % Winter 2018 - January to April 2018 - Updated 2019-01-15 07:47 EST - [Course Home Page] - [Course Outline] - [All Weeks] - [Plain Text] Characters that Look the Same but are Different =============================================== You must pay strict attention to the exact characters used in command lines. Here are some common confusions, shown in two font styles: - These are different characters:   L I l 1   and   `L I l 1` - These are different characters:   0 O o   and   `0 O o` - These are different characters:   - \_   and   `- _` - These are different characters:   , .   and   `, .` - These are different characters:   ; :   and   `; :` - These are different characters:   \\ /   and   `\ /` - These are different characters:   \` \' \"   and   \` `' "` - These are different character pairs:   () {} \[\] \<\>   and   `() {} [] <>` A common error is to think that the option letter `l` after the dash in `ls -l` is the digit `1`. It's a letter `l`, not a digit `1`. Typing Command Lines ==================== In documentation and assignments you will be asked to enter some command lines, and the lines will be written something like this: $ date $ toilet --gay Hello World The above indicates that you should type the command name `date` at a shell prompt, followed by the `Enter` key, then at the next shell prompt type the command name `toilet` followed by some **arguments** `--gay Hello World` followed by the `Enter` key. The shell prompt is indicated by the `$`. You do not *type* the prompt; the prompt is merely a documentation convention telling you that what follows to the right it is a command line that you should type at the prompt. The actual shell prompt might look more like `bash-4.3$`, or perhaps `%` or `>`, but we only need to show the `$` in the examples. Do not type the shell prompt when entering command lines. Enter only the command line to the right of the prompt, followed by the `Enter` key. The command will not be executed until the `Enter` key is used. The word or string of text on the left, closest to the shell prompt, is usually the **command name**, e.g. `date`, `toilet`. The text strings that follow the command name, if any, are the **arguments** to the command, e.g. the `toilet` command name above has arguments `--gay Hello World` The output produced by running the commands may or may not be shown in the documentation. The output, if shown, will not be preceded by the shell prompt: $ date Thu Jan 15 04:33:58 EST 2015 $ echo Hello World Hello World What is a shell for? ==================== The function of a shell is to **find and run programs**. ("Programs" are also called processes, commands, or utilities.) A shell is itself a program that finds and runs other programs. Shells also do programming kinds of things, but that programming is usually to aid in the finding and running of programs, not to do any kind of substantial mathematical or business calculations. (Shells do not do mathematics very well.) Part of running a program involves supplying command line arguments to that program; the shell helps with that, too. Command names typed into shells almost always resolve to be the names of executable files stored in system directories. (Some command names are not files; they are built-in to the shell, such as **cd**.) For example, the shell may resolve the command name **ls** to run the executable file `/bin/ls`, passing it any arguments you type on the command line. You can control where the shell looks for command executables by setting the `$PATH` variable. How does the shell help find and run commands? ============================================== We will learn many of these mechanisms in upcoming notes and lectures: - Shells look for command name executable files in various places, using a list of directories stored in the `$PATH` environment variable that you will learn about later. - Shells provide ways of editing the command line you type, so you can change words, insert text, etc., using fast editor commands instead of the slower arrow and backspace keys (which may not work on all systems). - Shells provide ways of completing command and file names, to save typing. (The shell always spells names correctly when it completes them!) - Shells provide a "history" mechanism to recall and edit the last commands you enter, to save retyping them. You can save your command history and have a written record of what you did yesterday. - Shells provide aliases and variables to save typing the same things (commands or pathnames) over and over. You can create your own customized command names, e.g. a custom version of **ls** that always shows hidden files. - Shells provide wildcards (GLOB patterns) to generate lists of pathnames as arguments for commands, e.g. `*.txt`. Instead of having to select names one-by-one, shell GLOB patterns can match thousands of pathnames. Basic Command Line Syntax ========================= The shell prompts you to enter command lines using a **prompt**. In documentation and help files, the prompt is usually assumed to be a dollar sign `$` but may actually look like `bash-4.3$` or something similar. Later, you will learn how to change what the prompt looks like by assigning quoted text to the `PS1` variable. Most of the time, what you enter in response to the shell prompt are **command lines** that start with a **command name** followed by some optional **arguments**. You type the entire line, then send it for execution by the shell by pushing the **ENTER** or **RETURN** key. Examples of command lines (following the shell prompt): $ date $ figlet "Hello World" $ toilet --gay "Big Letters" Linux commands and file names are case sensitive, which means that typing `date`, `Date`, `datE` or `DATE` are considered different commands, and `file.txt` and `FILE.TXT` are different file names. Almost all Linux command names are all lower-case. File names also tend to be all lower-case with no spaces, for ease of use on the command line, but with increasing use of GUI interfaces mixed-case file names containing blanks are becoming more common. You need to put file names that contain blanks or other special characters inside quote characters, e.g. $ ls 'My Documents' Most commands use the following syntax format where the command name is first on the line (at the left) and all the *option* arguments must *precede* (be to the left of) all other *parameter* arguments such as file names: $ commandname -options... --options... parameters... [Enter] Example: $ ls -ail --full-time /home/user foo/bar [Enter] Above, `ls` is the command name, `-ail` and `--full-time` are four options, and `/home/user` and `foo/bar` are two pathname arguments (one absolute pathname and one relative pathname). You must use the `[Enter]` key to submit the command to the shell. You can use `[Enter]` anywhere in the command line; it doesn't have to be at the end. The `[Enter]` key is never shown in documentation. As shown above, the command name and each argument have to be separated by one or more spaces. The number of additional spaces doesn't affect the command -- one space is as good as a hundred. Usually the first piece of text on the left of a shell command line is taken to be a command name, e.g. `date`, `ls`, `echo`, `figlet`. A command name may be followed by optional space-separated **arguments**. Arguments may be command **options** followed by **parameters** for the command. Verb **AND** Object ------------------- Most Unix/Linux commands need both a **verb** (what to do -- the *command name*) and an **object** (what to do it on -- a command *argument*). Here are some command line examples of verb and object: $ cat /etc/passwd # argument to cat is a file name $ mail idallen@idallen.ca # argument to mail is an email address $ kill 1234 # argument to kill is a process number The following incorrect attempts at command lines are wrong: $ /etc/passwd # missing VERB; what are you trying to DO with this file? $ rm # missing OBJECT; remove WHAT file? Remember to tell Unix/Linux both what you want to do and to what object you wish to do it. Here are examples of different command names that all do different things to the given file name object: $ cat /etc/passwd $ less /etc/passwd $ wc /etc/passwd $ sort /etc/passwd $ rm /etc/passwd # fails: ordinary users do not have permission to do this Most (but not all) commands take what as arguments? =================================================== Most -- but not all -- Unix/Linux commands take *pathnames* as arguments. Much of what people do on their computers is manipulate files and directories. This is why shells have GLOB pattern matching (wildcard) characters. **Pathnames** are names that might be file names, or directory names, or some other file system object. (Unix/Linux has names that are neither files nor directories, e.g. the `/dev/null` pathname is a *character special* device.) The shell has GLOB (wildcard) features to make matching multiple pathnames easier. e.g. try the command line: `$ echo /bin/*` Using a Shell inside a Terminal =============================== Shells run inside "terminals", usually software terminal windows on modern GUI systems. These terminals recognize some useful unprintable *control* characters that aid in typing text, including command lines. Syntax of Control Characters -- `^A` `^B` `^C` etc. --------------------------------------------------- The documentation syntax `^C` stands for the single unprintable control character `CONTROL-C`, also written as `CTRL-C` or perhaps `[CTRL-C]`. To generate this special character from your keyboard, hold down the `CTRL` key and simultaneously push the letter `C` key (upper or lower case; doesn't matter). This syntax and method applies to all the control characters, e.g. `^A`, `^B`, `^C`, `^D`, etc. Common control characters -- `^C ^D ^U ^W ^Z` --------------------------------------------- Shells run inside terminals or terminal emulator programs such as Linux `xterm`, Windows `PuTTY`, or OSX `Terminal`. Here are some useful control characters that you can use in a terminal connection to a Unix/Linux machine: ^C - CTRL-C - *interrupt* (terminate) a running foreground program ^D - CTRL-D - send *End-of-File* (EOF) from your keyboard (end input) ^U - CTRL-U - erase input characters back to the start of the prompt ^W - CTRL-W - erase just the last word you typed (repeat to erase next word) ^Z - CTRL-Z - temporarily *suspend* a program; restart by typing "fg" The above control characters are properties of the *terminal*, so they work for almost *any* program that reads your keyboard, not just shells. You can view or change any of the above characters using the `stty` command. Interrupting programs with `^C` ------------------------------- The usual way to interrupt a command that is executing is by using the `^C` character. For example, at a shell prompt enter the command `ping localhost`, followed by the `Enter` key, and then use `^C` to interrupt the `ping` program and get the shell prompt back: $ ping localhost PING localhost (127.0.0.1) 56(84) bytes of data. 64 bytes from localhost (127.0.0.1): icmp_req=1 ttl=64 time=0.046 ms 64 bytes from localhost (127.0.0.1): icmp_req=2 ttl=64 time=0.057 ms 64 bytes from localhost (127.0.0.1): icmp_req=3 ttl=64 time=0.055 ms 64 bytes from localhost (127.0.0.1): icmp_req=4 ttl=64 time=0.055 ms 64 bytes from localhost (127.0.0.1): icmp_req=5 ttl=64 time=0.061 ms ^C --- localhost ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 3996ms rtt min/avg/max/mdev = 0.046/0.054/0.061/0.010 ms $ Most Unix/Linux programs will terminate when they receive the `^C` interrupt sent from your keyboard. When the command terminates, the shell issues another prompt for more input. Suspending programs using `^Z` ------------------------------ The `^Z` character temporarily suspends a running program. The program doesn't terminate; it just stops executing and you get the shell prompt back so that you can enter other command lines. You can get a list of suspended programs in your shell using the command `jobs` and you can resume a suspended program by typing `fg`: $ ping localhost PING localhost (127.0.0.1) 56(84) bytes of data. 64 bytes from localhost (127.0.0.1): icmp_req=1 ttl=64 time=0.046 ms 64 bytes from localhost (127.0.0.1): icmp_req=2 ttl=64 time=0.057 ms 64 bytes from localhost (127.0.0.1): icmp_req=3 ttl=64 time=0.055 ms 64 bytes from localhost (127.0.0.1): icmp_req=4 ttl=64 time=0.055 ms 64 bytes from localhost (127.0.0.1): icmp_req=5 ttl=64 time=0.061 ms ^Z [1] + Stopped ping localhost $ echo "The ping program is suspended, waiting for fg" The ping program is suspended, waiting for fg $ jobs [1] + Stopped ping localhost $ fg ping localhost 64 bytes from localhost (127.0.0.1): icmp_req=6 ttl=64 time=0.061 ms 64 bytes from localhost (127.0.0.1): icmp_req=7 ttl=64 time=0.051 ms 64 bytes from localhost (127.0.0.1): icmp_req=8 ttl=64 time=0.062 ms 64 bytes from localhost (127.0.0.1): icmp_req=9 ttl=64 time=0.052 ms ^C --- localhost ping statistics --- 9 packets transmitted, 9 received, 0% packet loss, time 13345ms rtt min/avg/max/mdev = 0.046/0.054/0.061/0.010 ms $ Be very careful about suspending programs using `^Z`. Suspending is not the same as killing or terminating! If you accidentally suspend a program (it says `Stopped`), you can immediately type `fg` into the shell to restart it again. > Suspending a text editor (such as `vim`) using `^Z` is a dangerous thing to > do, since having multiple editors all editing the same file can cause data > loss if you write out an old version of a file on top of a newer one. **Do > not suspend text editors unless you know what you are doing!** Bonus Control Characters -- `^L ^R` ----------------------------------- The `bash` shell also recognizes some very useful control characters: ^L - CTRL-L - clear your screen; move current command line to top ^R - CTRL-R - search your current shell command history for a string These depend on running the `bash` shell; they are not part of your terminal. The `^R` is very useful for searching backwards in your shell command history to find a previous command containing the search string you type. You can type multiple `^R` to keep searching backwards for more command lines with the same search string. Interrupting Processes and signalling End-of-File -- `^C` and `^D` ================================================================== The terminal emulator treats the default control characters `^C` (interrupt) and `^D` (end-of-file) specially: Interrupting a Process -- `^C` ------------------------------ If you type a command that keeps running and doesn't stop by itself (e.g. the `ping` command) and decide you want to interrupt (terminate) it, the terminal emulator lets you do this by typing a character that sends an Interrupt signal to the process. To send an interrupt signal to a process that is running on your terminal, use the Interrupt Character, usually `CTRL-C` (`^C`). (You can program a different character; sometimes `DEL` is used.) The Interrupt usually throws away any pending input/output for the process and causes the process to **terminate abnormally**. Use it with caution. Interrupting a process usually terminates the process. Whatever the process was doing is left incomplete and unfinished. Buffers are not flushed; files will be incomplete. An interrupted process did not finish. Signalling End-of-File (EOF) from the keyboard -- `^D` ------------------------------------------------------ When a program reads a file, it naturally stops reading when the end of the file is reached -- this is called End-of-File or `EOF`. When the program is reading from your keyboard, it doesn't know when to stop reading unless you tell it by typing the EOF character into the terminal. Your EOF character, signalling EOF from your keyboard, is usually `CTRL-D` (`^D`). You must type this at the beginning of a line (or type it twice). (You can program a different character; but, it is almost never done.) Typing the EOF character tells the process reading your keyboard that you are finished typing at your terminal (EOF); but, it does not interrupt or terminate the process. The process will finish reading your keyboard and then continue with whatever it is doing. Often, that means producing some output and then exiting successfully. (Note: Unlike DOS/Windows systems, `^D` does not actually send any character to the process reading your keyboard -- it simply tells the process that EOF has happened. You cannot put `^D` inside a file and have it serve as an EOF. The `^D` only works when typed from a keyboard inside a terminal.) Examples of `^C` and `^D` ------------------------- Here is an example showing how `^C` interrupts a program before it finishes, but `^D` signals an EOF and lets the program finish. In both examples below we use shell Output Redirection to save the output of the sort command into a file named `out`: ### `^C` Interrupting a Process (does not finish) $ sort >out # sort keyboard input into file out enter some lines of text and then use ^C CTRL-C to interrupt the program and you will get an empty file ^C $ cat out # display content of file out $ # nothing: empty file ### `^D` Signalling EOF (end-of-file) lets process finish $ sort >out # sort keyboard input into file out enter some lines of text and then use a ^D CTRL-D End-Of-File EOF to signal end of file and sort will sort your data and finish ^D $ cat out # display content of file out and sort will sort your data and finish enter some lines of text and then use a ^D CTRL-D End-Of-File EOF to signal end of file $ Typing `^C` interrupts a process, which usually terminates it abnormally. Any pending output from the process is usually lost. Typing `^D` signals EOF from your keyboard to the process, so the process knows to stop reading your keyboard and finish what it was doing. Getting Out of Programs -- exit, quit, etc. =========================================== Different Unix/Linux programs have different ways of ending or exiting from them, because they were written by different people. If you're stuck inside a program and can't get out to the shell again, try various things such as these: h or help or ? # find out how to exit the program! q or Q or quit # used to exit "less" and "more" :q! # used in VI to exit without saving x or exit ^D # CTRL-D - send End-Of-File - EOF ^C # CTRL-C - send Interrrupt signal ^\ # CTRL-backslash - send Quit signal logout bye ESC # the ESC or Escape key . # a period - used in some mail programs One of the above usually works. Sometimes you can use `^Z` (CTRL-Z) to suspend the process temporarily, and then type `"kill %%"` to terminate it. (Remember to terminate it, or it will sit there suspended, waiting forever. A process stopped with `^Z` still has all its files open.) Nested Shells inside Shells =========================== Your default login shell on Linux is the Bourne-Again shell (`bash`). You can call up other shells by name if they are installed: `sh`, `ksh`, `bash`, `csh`, `tcsh`, etc. Not all shells may be installed on your system. You can call up a shell inside a shell (a "nested" shell) by typing its name at the shell prompt (e.g. type `sh` or even another `bash`). The new copy of the shell will take over reading your keyboard until you exit from it again and the previous shell returns to read your keyboard: bash-4.3$ echo this is my login shell this is my login shell bash-4.3$ sh $ echo this is a nested sh shell this is a nested sh shell $ exit bash-4.3$ echo I am back to the login shell I am back to the login shell To exit from a shell type: `exit` When you exit from the first shell in a terminal, your login shell, either the terminal window closes (if using a terminal window), or, if you are logged in directly to a Unix/Linux console, you log out from Unix. You can [see a video on sub-shells]. (You may need to download this MP4 file to your computer to see the video.) Another way to exit a shell is to type your EOF character. (Your EOF character is usualy `CONTROL-D`.) Some shells can be told to ignore EOF. Remember: all Unix programs (should) have manual pages! $ man sh $ man bash $ man ksh $ man csh $ man tcsh Each new shell is another Unix process. To get a list of your current processes in the current terminal, use: `ps` Shell interactive command and pathname completion ================================================= If you type part of a pathname on a comand line and then push the TAB key, many shells will attempt to complete the pathname typing for you. $ echo /etc/pas # shell will complete as /etc/passwd If the pathname cannot be completed unambiguously (there is more than one name that starts with that prefix), the shell will show you a list of possible completions if you push the TAB key again: $ echo /etc/pas passwd passwd- Command name completion ----------------------- Similarly, if you type part of a command name and push the TAB key once or twice, the shell will complete and then list all the possible commands in your PATH that start with that prefix: $ mkd mkdir mkdiskimage mkdosfs Use the shell to do your typing for you -- learn how to use the TAB key! (The shell always spells your pathnames correctly!) Shell "history" of previous commands ==================================== Most shells keep a record ("history") of the commands you type. The history is often saved in a file when the shell exits, and is restored when you next log in. You can see the history list using the built-in `history` command. Many shells allow you to use the UP-ARROW and DOWN-ARROW keys to move up and down in the command history. (Some shells use other key sequences, such as `^P` and `^N`.) You can use the arrow keys to move up to the command you want to re-execute, modify it as needed, then use the `[Enter]` key (anywhere on the line) to execute it. Some shells let you select items from the history list by number (e.g. `!123`). See the **History** section of your shell's man page. Each shell process has its own separate command history. Starting a new shell (a sub-shell) will start an independent history any may not pick up the history of another shell unless that other shell saved it and your new shell loads it. What is a "Bourne" shell? What is a "C" shell? ============================================== The shells with names `sh`, `ksh`, `zsh`, and `bash` (the "Bourne" shells) all have a common ancestry. They are all derived from the original Bourne shell `sh` written in the 1970s by Stephen Bourne. The programming features of these shells (if statements, for loops, etc.) all look and work the same way. This is the best shell family to study. The shells named `csh` and `tcsh` are called the `C` shells and have a different history dating back to Bill Joy at Berkeley in the 1980s. Their syntax for programming is not the same as the Bourne shells. We do not cover the C shell syntax in this course; these shells are [notoriously buggy] and my 2002 list of `tcsh` bugs is still distributed with the shell. C Shell Bugs: search the Internet for:   `c shell bugs` My list of C Shell Bugs: use the above links or search the Internet for:   `c shell bugs "idallen"` Shell command line alias trouble ================================ Watch out for "helpful" system administrators that define aliases for your shells when you log in. (This is true on most versions of Unix/Linux.) The aliases may mislead you about how Unix commands actually work. For example, the `rm` command does *not* prompt you for confirmation when removing a file. On some systems, when you log in, `rm` has been made to be an alias for `rm -i`, which *does* prompt. If you become accustomed to the behaviour of the alias, you will be in serious trouble when you move to a system that does not define the alias. To avoid pre-defined aliases, sometimes you can start up a fresh copy of the shell that has no aliases defined: $ alias [...many aliases may print here...] $ bash bash$ alias [...no more aliases here...] The other thing you can do is execute `unalias -a` to remove all your aliases for the current shell. You can put this into your shell start-up file (e.g. `.bashrc`) to do it every time you start a new shell. To define your own aliases, look up "aliases" in a Unix/Linux resource, e.g.: $ alias dir='ls -lF' $ alias cd..='cd ..' $ alias mali=mail Aliases defined in the current shell are not saved when the shell exits. You must put your own alias definitions in a shell start-up file to have them restored between sessions (e.g. put them into your `.bashrc` file). Each shell has its own separate aliases. Starting a new shell does not get the aliases defined in other shells. -- | Ian! D. Allen, BA, MMath - idallen@idallen.ca - Ottawa, Ontario, Canada | Home Page: http://idallen.com/ Contact Improv: http://contactimprov.ca/ | College professor (Free/Libre GNU+Linux) at: http://teaching.idallen.com/ | Defend digital freedom: http://eff.org/ and have fun: http://fools.ca/ [Plain Text] - plain text version of this page in [Pandoc Markdown] format [www.idallen.com]: http://www.idallen.com/ [Course Home Page]: .. [Course Outline]: course_outline.pdf [All Weeks]: indexcgi.cgi [Plain Text]: 120_shell_basics.txt [see a video on sub-shells]: 145_subshells.mp4 [notoriously buggy]: http://teaching.idallen.com/cst8129/02f/notes/tcshbugs.txt [Pandoc Markdown]: http://johnmacfarlane.net/pandoc/