============================================================ File transfer to/from Unix/Linux machines ============================================================ -Ian! D. Allen - idallen@idallen.ca - www.idallen.com Contents of this file: * Introduction to File Transfer * SCP and SFTP - Secure Copy Program, Secure FTP 1. using Unix/Linux or Cygwin: get a file from remote to local 2. using Unix/Linux or Cygwin: put a file from local to remote * Using SFTP from Unix/Linux or Cygwin * Insecure FTP - File Transfer Protocol (do not use) * Text File Line End Differences * Binary Mode vs. Text Mode file transfer * Insecure FTP vs. the Unix Shells * Just use EMail for text files * Microsoft Windows Users - Using PuTTY suite PSCP on Windows - Using PuTTY suite PSFTP on Windows ----------------------------- Introduction to File Transfer ----------------------------- This file explains how to copy files between Unix/Linux machines and between Unix/Linux and other machines (e.g. to/from your home Linux or Windows machine or to/from a Windows machine in an Access Lab). How you transfer files depends on where you log in, and which machine is "local" and which machine is "remote". The existence of firewalls that restrict connections makes things more complex than usual. Connecting from the Course Linux Server to a remote machine usually requires that the remote machine have its own IP adddress - it should not itself be behind a firewall, unless you have arranged a pass-through port on that firewall. All Algonquin Access Labs are behind firewalls; you cannot use anything to connect into Algonquin machines from off-campus. Only the Algonquin Unix machine ACADUNIX is not hidden behind a firewall; it accepts both insecure FTP and secure SCP/SFTP connections. If you have an ACADUNIX account, sometimes you have to use ACADUNIX as a staging area to hold a file for copying. MS Windows users may want to skip to the later sections dealing with the lack of secure file transfer programs under Windows (and what to do). ---------------------------------------------- SCP and SFTP - Secure Copy Program, Secure FTP ---------------------------------------------- Avoid the standard FTP program - it sends passwords in clear text across the Internet. (You are slightly safer using the FTP program locall here at school, but realize that anyone snooping packets on your local network will still see your password.) The SCP and SFTP programs should be used to transfer files between machines when a userid/password is required (e.g. to/from the Course Linux Server). Both these programs use the underlying SSH (Secure SHell) protocol that encrypts both your password and the data being transferred. For Windows users, scroll down to the "Microsoft Windows Users" section. 1. Copy a file from a remote machine to the local machine: ---------------------------------------------------------- Using SCP from a Unix/Linux or Cygwin command line: $ scp -p userid@remote.host.name:remote_file local_file The "userid@" part contains your login userid on "remote.host.name". You can leave off "userid@" if your remote userid is the same as your userid on the local machine. You will be prompted to enter your password for the remote machine. The "remote_file" may be an absolute pathname (on "remote.host.name"), or it may be a pathname relative to your home directory on "remote.host.name". The "-p" option to SCP preserves the modify time of the transferred file. Examples: If the remote machine is not behind a firewall and uses a standard SSH port: $ scp -p abcd0001@acadunix.algonquincollege.com:dodo.txt happy.txt $ scp -p acadunix.algonquincollege.com:/tmp/foo mydir/bar If the remote machine is behind a firewall and requires a special port to be used, the -P option (upper case P) must be used to set the firewall pass-through port: $ scp -p -P 2222 abcd0001@example.com:dodo.txt happy.txt 2. Copy a file from the local machine to a remote machine: ---------------------------------------------------------- To copy from local to remote via a Unix/Linux or Cygwin command line, just reverse the order of the arguments to SCP: $ scp -p local_file userid@remote.host.name:remote_file If the remote machine is not behind a firewall and uses a standard SSH port: $ scp -p happy.txt abcd0001@acadunix.algonquincollege.com:dodo.txt $ scp -p mydir/foo acadunix.algonquincollege.com:/tmp/bar If the remote machine is behind a firewall and requires a special port to be used, the -P option (upper case P) must be used to set the firewall pass-through port: $ scp -p -P 2222 happy.txt abcd0001@example.com:dodo.txt ------------------------------------ Using SFTP from Unix/Linux or Cygwin ------------------------------------ The SFTP program is a cover for SSH and SCP that makes things look like you are using the insecure FTP program; however, the actual connection and transfer is done using the secure SSH protocol. From a Unix/Linux or Cygwin command line, you can start SFTP like this: $ sftp abcd0001@acadunix.algonquincollege.com Connecting to acadunix.algonquincollege.com... abcd0001@acadunix.algonquincollege.com's password: sftp> help [... output similar to using insecure FTP ...] sftp> quit If you are familiar with insecure FTP (see below), SFTP will operate much the same way. As with insecure FTP, you can list the contents of remote directories and transfer files both ways (using "put" and "get") on the same connection. The SFTP command has a different way to specify the port number: $ sftp -oPort=2222 example.com Connecting to example.com... idallen@example.com's password: sftp> help [... output similar to using insecure FTP ...] sftp> quit -------------------------------------------------- Insecure FTP - File Transfer Protocol (do not use) -------------------------------------------------- The old way to move files between machines was the insecure FTP (File Transfer Protocol) program. FTP is an insecure form of file transfer; because, any password you type into insecure FTP is visible across the network. Don't use insecure FTP for transfer between machines requiring userids and passwords over an insecure network (e.g. the Internet). If you log in to the Course Linux Server (using secure SSH), you can then use the insecure "ftp" or "lftp" commands on the Server to connect *out* from the Server to other remote machines (e.g. insecure FTP to your home computer or to ACADUNIX), if those other machines accept insecure FTP connections. If you set up your home computer with an insecure FTP server [be careful!], you may use the insecure "ftp" command on the Course Linux Server to connect to your home machine, if your home machine has a public IP address and isn't behind a firewall or NAT router. Once you have an insecure FTP connection set up, you can copy files in either direction using the "put" and "get" commands, as you wish. Many Internet sites support a form of "anonymous" insecure FTP that lets you connect to a site without requring a password, using the special insecure FTP userid "anonymous" or "ftp". Since it requires no password, this form of insecure FTP is safe to use over the Internet. It is how software is often provided for download to Unix/Linux users. Command-line insecure FTP is a "subsystem" kind of program with its own set of subcommands. Once inside the insecure FTP program, your prompt becomes "ftp>". Inside insecure FTP, the "help" command will list the possible insecure FTP commands available, and "help commandname" will give you a bit more help on the given FTP command name. The Unix manual page for insecure FTP ("man ftp") explains the individual insecure FTP subcommands in much better detail. Do not confuse Unix commands with insecure FTP subcommands. Pay attention to which program is prompting you for input. To quit FTP, type "quit". The command-line insecure FTP program is also available under Windows. (You may need to install insecure FTP from the Windows CDROM.) The list of insecure FTP commands is slightly different; but, the basic commands (ls, cd, get, put) are the same as for Unix. (You can run command-line insecure FTP from a DOS window or using the "Run" dialog box.) Remember: FTP is not secure. Insecure FTP will not transfer entire directories; it may only be used to transfer files one at a time. (There are ways to get insecure FTP to fetch multiple files at once; but, the files must all be in the same directory; you can't fetch multiple directories. See the help for the "mget" insecure FTP subcommand.) Below is an example command-line insecure FTP session to a public FTP server. This insecure FTP command could be run on a Unix/Linux machine or under Windows. $ ftp ftp.gnu.org Connected to ftp.gnu.org. 220 GNU FTP server ready. 530 Please login with USER and PASS. Name (ftp.gnu.org:idallen): anonymous # NOTE: special userid used 230 Login successful. # NOTE: no password needed! Remote system type is UNIX. Using binary mode to transfer files. ftp> help [... many lines of FTP commands show here ...] ftp> help ls ls list contents of remote directory ftp> help get get receive file ftp> help cd cd change remote working directory ftp> ls 150 Here comes the directory listing. [... many files show here ...] ftp> cd gnu ftp> ls 150 Here comes the directory listing. [... many files show here ...] ftp> cd chess ftp> ls 150 Here comes the directory listing. [... many files show here ...] ftp> get README.gnuchess local: README.gnuchess remote: README.gnuchess 150 Opening BINARY mode data connection for README.gnuchess (89 bytes). 226 File send OK. 89 bytes received in 0.0066 seconds (13 Kbytes/s) ftp> quit 221 Goodbye. $ ls -l README.gnuchess -rw-r--r-- 1 idallen idallen 89 Oct 2 00:49 README.gnuchess Text File Line End Differences ------------------------------ Note that the line end character in text files is not the same between Unix and Windows. A text file written on Unix contains only linefeed (LF or NL or "\n") characters at the ends of lines; Windows expects lines in text files to end in both a carriage-return (CR or "\r") *and* a linefeed character. This may result in "staircasing" text if you send a Unix text file to a Windows printer from some programs (e.g. Notepad). Print a small sample first, and try using "Write" or "Wordpad" to print a Unix file instead of "Notepad". Binary Mode vs. Text Mode file transfer -------------------------------------- Insecure FTP can transfer files in either "text" or "binary" mode. Almost always use "binary" mode, which makes an exact copy of the file. "Text" mode can be used to translate line ends when copying plain text files between dissimilar systems, e.g. between Unix and Windows, but the line-end translation will corrupt all other non-text files. Insecure FTP vs. the Unix Shells -------------------------------- The syntax of insecure FTP commands is not the same as the syntax of Unix commands. This insecure FTP command doesn't do what you think it does: ftp> ls -l filename output to local-file: filename? If you answer "yes" to this prompt, you will copy the output of "ls -l" into the file "filename" in your current directory, erasing what was there before. This is probably not what you want. Don't do it. The insecure FTP command names resemble Unix command names; but, they are *not* Unix commands. The syntax is different. You are not typing into a shell, you are typing into the insecure FTP program. Be careful. (Also, don't type insecure FTP commands into Unix and expect that they will work, e.g. the BASH shell doesn't understand "put filename".) Just use EMail for text files ----------------------------- If the file you want to take home from Unix is a text file, you can usually EMail it to yourself somewhere using the Unix "mail" program with standard input redirected to come from the file you want: $ mail abcd0001@algonquincollege.com <.bashrc $ mail me@hotmail.com <.bashrc You can only send text files this way, and you can only send one file per mail message using file input redirection (unless you concatenate many files together first). See "man mail" for further details. To send binary programs via email you must encode them as ASCII first and decode them after receiving them. See "man uuencode". ----------------------- Microsoft Windows Users ----------------------- MS Windows has a command-line version of insecure FTP available from a DOS prompt or in a DOS window. You can also download various graphical insecure FTP clients. Many recommend the insecure programs "FileZilla" or "WS_FTP". Some versions of Windows also let you use an insecure FTP URI to connect to an insecure FTP server and log in and transfer files, e.g. using this form of URI: ftp://ftp.algonquincollege.com/ The Windows Explorer (not Internet Explorer!) will let you open an insecure FTP URI such as the one above and drag-and-drop files between your machine and a remote machine graphically. Underneath, Windows is using the insecure FTP protocol and your data and passwords are visible to anyone who can snoop your network connection. Don't use this. Note that you may be able to access a file via FTP that cannot be displayed in a web browser, since the FTP program is logged in as your account name and the web browser accesses your files as "other". You must ensure that your files have read permissions for "other" after you transfer them to the Course Linux Server. MS Windows does not ship with any secure file transfer programs such as SCP or SFTP, even though versions exist that are open source and free software. You can buy expensive commercial versions of SSH/SCP/SFTP for Windows from various vendors; or, you can download and install the free PuTTY program suite that contains programs that work like SSH, SCP, and SFTP. (Do a Google search for: PuTTY download) If you download and install the full PuTTY program suite under Windows (PuTTY comes with an executable auto-installer that will do this for you), you will find the programs PSCP and PSFTP under the installation directory (usually under C:\Program Files\PuTTY). Start up a DOS command prompt, change to this directory, and run the secure commands you need to copy files to/from other systems (see below for examples of how to do this). When transferring files between Windows and Unix/Linux machines, remember that pathnames on the Windows machines contain backward slashes while pathnames on Unix/Linux machines contain forward slashes. For example, you might find yourself typing something like this: psftp> put "d:\folder\myfile.txt" "public_html/dir/page.txt" local:d:\dir\myfile.txt => remote:/home/abcd0001/public_html/page.txt Note how the Windows pathnames contain backslashes and the Unix/Linux pathnames contain forward slashes. Some versions of PSFTP accept forward slashes for Windows pathnames. You must surround the pathnames with double quotes if the pathnames contain blanks. Using the PuTTY suite PSCP on Windows ------------------------------------- If the remote machine is behind a firewall and requires a special port to be used, the -P option (upper case P) must be used to set the firewall pass-through port. Leave it out otherwise. The backslash at the end of a line below indicates that the line continues. Type what is written all on one line without the backslash. Unless you change your DOS search PATH, you will only be able to execute the PSCP and PSFTP commands from the directory into which you downloaded them. C:\> cd "C:\Program Files\PuTTY" C:\Program Files\PuTTY> pscp -h # -h gives a short help listing ...short help listing displays here... C:\Program Files\PuTTY> \ pscp -P 2222 "abcd0001@example.com:dir/foo.txt" "folder\bar.txt" ...only use the -P option if you need the special port number... ...you may be asked to accept the host key here (say yes)... abcd0001@example.com's password: ...file transfers remote "dir/foo.txt" to local "folder\bar.txt"... You must use your own userid on the remote machine. You must replace "example.com" with the machine name or IP address to which you wish to connect. The -P option sets a non-standard port number, if you need it. Windows pathnames should contain backslashes and the Unix/Linux pathnames contain forward slashes. Some versions of PSCP accept forward slashes for Windows pathnames. You must surround the pathnames with double quotes if the pathnames contain blanks. Using the PuTTY suite PSFTP on Windows -------------------------------------- Unless you change your DOS search PATH, you will only be able to execute the PSCP and PSFTP commands from the directory into which you downloaded them. If the remote machine is behind a firewall and requires a special port to be used, the -P option (upper case P) must be used to set the firewall pass-through port. Leave it out otherwise. C:\> cd "C:\Program Files\PuTTY" C:\Program Files\PuTTY> psftp -h # -h gives a short help listing ...short help listing displays here... C:\Program Files\PuTTY> psftp -P 2222 abcd0001@example.com ...you may be asked to accept the host key here (say yes)... abcd0001@example.com's password: Remote working directory is /home/abcd0001 psftp> help ...short help listing displays here... psftp> ls ...listing of directory displays here... psftp> get ".bashrc" "foo.txt" remote:/home/abcd0001/.bashrc => local:foo.txt psftp> put "d:\dir\myfile.txt" "public_html/page.txt" local:d:\dir\myfile.txt => remote:/home/abcd0001/public_html/page.txt psftp> quit ...file "foo.txt" is now in the current directory... You must use your own userid on the remote machine. You must replace "example.com" with the machine name or IP address to which you wish to connect. The -P option sets a non-standard port number, if you need it. Windows pathnames should contain backslashes and the Unix/Linux pathnames contain forward slashes. Some versions of PSFTP accept forward slashes for Windows pathnames. You must surround the pathnames with double quotes if the pathnames contain blanks. Note that the options to the PuTTY Windows version of SFTP (named PSFTP) are not the same as the options to the Unix/Linux version of SFTP. In particular, the option "-P" has different meanings! -- | Ian! D. Allen - idallen@idallen.ca - Ottawa, Ontario, Canada | Home Page: http://idallen.com/ Contact Improv: http://contactimprov.ca/ | College professor (Free/Libre GNU+Linux) at: http://teaching.idallen.com/ | Defend digital freedom: http://eff.org/ and have fun: http://fools.ca/