============================== File transfer between machines (without using a mouse!) ============================== -Ian! D. Allen - idallen@idallen.ca - www.idallen.com This file explains how to copy files between Unix/Linux machines and between Unix/Linux and other machines (e.g. Windows). Copying text using a mouse and cut/paste is usually not possible when working on servers that don't run a GUI. A mouse cut/paste can't copy binary data. You must learn to copy actual files between networked machines. Contents of this file: * File copy terminology * SCP and SFTP - Secure Copy Program, Secure FTP 1. get a file from remote Unix to local Unix 2. put a file from local Unix to remote Unix * Using SSHFS (remote file system) * Using SFTP * FTP - File Transfer Protocol * Just use EMail for text files * Microsoft Windows Users - Using PSCP on Windows - Using PSFTP on Windows - Other Windows secure copy programs --------------------- File copy terminology --------------------- How you transfer files depends on where you log in, and which machine is "local" to the program doing the copying, and which machine is "remote". The existence of firewalls and Network Address Translation (NAT) restrict connections and make things awkward. Connecting to a remote machine usually requires that the remote machine have its own public IP adddress - it should not itself be behind a firewall and/or NAT, unless you have arranged a pass-through port on that firewall. Algonquin Labs (and the Course Linux Server) are behind NAT; you cannot use anything to connect into these machines from off-campus unless you also run some kind of VPN software that puts your machine "on campus". See the Class Notes file "vpn.txt" for details on getting a VPN working. Most modern Unix/Linux systems come equipped with free packages that implement secure file transfer. Microsoft Windows users should skip forward to the section below dealing with getting programs for that O/S. ---------------------------------------------- SCP and SFTP - Secure Copy Program, Secure FTP ---------------------------------------------- The SCP and SFTP programs should be used to transfer files between machines when a userid/password is required (e.g. to/from the Course Linux Server). Both these programs use the underlying SSH (Secure SHell) protocol that encrypts both your password and the data being transferred. Using SCP from a Unix command line - Unix to Unix: 1. Copy a file from a remote machine to the local machine: ---------------------------------------------------------- $ scp -p userid@remote.host.name:remote_file local_file The "userid@" part contains your login userid on "remote.host.name". You can leave off "userid@" if your remote userid is the same as your userid on the local machine. You will be prompted to enter your password for the remote machine. You may also be prompted to accept the server's encryption key for the first transfer. Both the "remote_file" and the "local_file" may be relative or absolute pathnames. Relative pathnames for "remote_file" are relative to your $HOME directory on "remote.host.name". The "-p" option to SCP preserves the modify time of the transferred file. Examples: If the remote machine is not behind a firewall and uses a standard SSH port: $ scp -p alleni99@acadunix.algonquincollege.com:dodo.txt happy.txt $ scp -p acadunix.algonquincollege.com:/tmp/foo mydir/bar $ scp -p 10.50.254.148:/tmp/foo /tmp/bar If the remote machine is behind a firewall and requires a special port to be used, the -P option (upper case P) must be used to set the firewall pass-through port. 2. Copy a file from the local machine to a remote machine: ---------------------------------------------------------- To copy from local to remote, just reverse the order of the arguments to SCP: $ scp -p local_file userid@remote.host.name:remote_file Both the "remote_file" and the "local_file" may be relative or absolute pathnames. Relative pathnames for "remote_file" are relative to your $HOME directory on "remote.host.name". Examples: If the remote machine is not behind a firewall and uses a standard SSH port: $ scp -p happy.txt alleni99@acadunix.algonquincollege.com:dodo.txt $ scp -p mydir/foo acadunix.algonquincollege.com:/tmp/bar $ scp -p /tmp/foo 10.50.254.148:/tmp/bar If the remote machine is behind a firewall and requires a special port to be used, the -P option (upper case P) must be used to set the firewall pass-through port. -------------------------------- Using SSHFS (remote file system) -------------------------------- If you have root access to install packages, you can install the "sshfs" package that will let you "mount" as a directory any remote system to which you can make an SSH connection. - download and install the "sshfs" package onto your system: sudo apt-get install sshfs - add your userid to the "fuse" group: sudo addgroup abcd0001 fuse - log out, log in again (to make the group change take effect) - run "groups" and confirm your userid is in the "fuse" group - make a directory mount point for the remote system: mkdir rem - run sshfs to the the remote system and mount it on the rem dir: sshfs abcd0001@10.50.254.148: rem - access your files via the rem dir as if it were local: ls -l rem ; cp file1 rem/file1 ; cp rem/file2 file2 - unmount the remote system when done: fusermount -u rem ---------- Using SFTP ---------- The SFTP program is a cover for SSH and SCP that makes things look like you are using the insecure FTP program; however, the actual connection and transfer is done using the secure SSH protocol: $ sftp alleni99@acadunix.algonquincollege.com Connecting to acadunix.algonquincollege.com... alleni99@acadunix.algonquincollege.com's password: sftp> help [... output similar to using FTP ...] sftp> quit If you are familiar with FTP (see below), SFTP will operate much the same way. As with FTP, you can list the contents of remote directories and transfer files both ways (using "put" and "get") on the same connection. Command-line SFTP is a "subsystem" kind of program with its own set of subcommands. Once inside the SFTP program, your prompt becomes "sftp>". Inside SFTP, the "help" command will list the possible SFTP commands available, and "help commandname" will give you a bit more help on the given SFTP command name. Do not confuse Unix commands with SFTP subcommands. Pay attention to which program is prompting you for input. To quit SFTP, type "quit". If you need to set a port number to bypass a firewall, the SFTP command has a different way to specify the port number than SCP or SSH; see the documentation. ---------------------------- FTP - File Transfer Protocol (DO NOT USE) ---------------------------- The old way to move files between machines was the "FTP" (File Transfer Protocol) program. FTP is an insecure form of file transfer; because, any password you type into FTP is visible across the network. Don't use FTP for transfer between machines requring userids and passwords over an insecure network (e.g. don't use it over the Internet). If you set up your home computer with a public IP addres and an FTP server [be careful!], you may use the "ftp" command to connect to your home machine. Once you have an FTP connection set up, you can copy files in either direction, as you wish. Many Internet sites support a form of "anonymous" FTP that lets you connect to a site without requring a password, using the special FTP userid "anonymous" or "ftp". Since it requires no password, this form of FTP is safe to use over the Internet. It is how software is often provided for download to Unix/Linux users. Command-line FTP is a "subsystem" kind of program with its own set of subcommands. Once inside the FTP program, your prompt becomes "ftp>". Inside FTP, the "help" command will list the possible FTP commands available, and "help commandname" will give you a bit more help on the given FTP command name. The Unix manual page for FTP ("man ftp") explains the individual FTP subcommands in much better detail. Do not confuse Unix commands with FTP subcommands. Pay attention to which program is prompting you for input. To quit FTP, type "quit". The command-line (DOS-style) FTP program is also available under Windows. (You may need to install FTP from the Windows CDROM.) The range of FTP commands is slightly different; but, the basic commands (ls, cd, get, put) are the same as for Unix. You can run command-line FTP from a DOS window or using the "Run" dialog box. Remember: FTP is not secure. FTP will not transfer entire directories; it may only be used to transfer files one at a time. There are ways to get FTP to fetch multiple files at once; but, the files must all be in the same directory; you can't fetch multiple directories. See the help for the "mget" FTP subcommand. Here is an example command-line FTP session to a public FTP server: $ ftp ftp.gnu.org Connected to ftp.gnu.org. 220 GNU FTP server ready. 530 Please login with USER and PASS. Name (ftp.gnu.org:idallen): anonymous # NOTE: special userid used 230 Login successful. # NOTE: no password needed! Remote system type is UNIX. Using binary mode to transfer files. ftp> help [... many lines of FTP commands show here ...] ftp> help ls ls list contents of remote directory ftp> help get get receive file ftp> help cd cd change remote working directory ftp> ls 150 Here comes the directory listing. [... many files show here ...] ftp> cd gnu ftp> ls 150 Here comes the directory listing. [... many files show here ...] ftp> cd chess ftp> ls 150 Here comes the directory listing. [... many files show here ...] ftp> get README.gnuchess local: README.gnuchess remote: README.gnuchess 150 Opening BINARY mode data connection for README.gnuchess (89 bytes). 226 File send OK. 89 bytes received in 0.0066 seconds (13 Kbytes/s) ftp> quit 221 Goodbye. $ ls -l README.gnuchess -rw-r--r-- 1 idallen idallen 89 Oct 2 00:49 README.gnuchess FTP can transfer files in either "text" or "binary" mode. Always use "binary" mode for data files. "Text" mode can sometimes be usefully used with text files between dissimilar systems; but, it will corrupt non-text files such as data, images, and compressed files. The FTP subsystem vs. the Unix Shells ------------------------------------- The syntax of FTP commands inside the FTP subsystem is not the same as the syntax of Unix commands. For example, this FTP command doesn't do what you think it does: ftp> ls -l filename output to local-file: filename? If you answer "yes" to this prompt, you will copy the output of "ls -l" into the file "filename" in your current directory, erasing what was there before. This is probably not what you want. Don't do it. The FTP command names resemble Unix command names; but, they are *not* Unix commands. The syntax is different. You are not typing into a shell, you are typing into the FTP program. Be careful. Also, don't type FTP commands into Unix and expect that they will work, e.g. "put filename" is not a valid Unix command. ----------------------------- Just use EMail for text files ----------------------------- If the file you want to take home from Unix is a text file, you can often EMail it to yourself somewhere using the Unix "mail" program with standard input redirected to come from the file you want: $ mail alleni99@algonquincollege.com <.bashrc $ mail me@hotmail.com <.bashrc You can only send text files this way (no binary junk!), and you can only send one file per mail message using file input redirection, unless you concatenate many files together first. See "man mail" for further details on how "mail" works. To send binary programs via email you must encode them as ASCII first. See "man uuencode". ----------------------- Microsoft Windows Users ----------------------- MS Windows has a command-line version of FTP available from a DOS-style prompt or in a DOS-style command line window. MS Windows does not ship with SCP or SFTP, even though versions exist that are open source and free software. You can buy expensive commercial versions of SSH/SCP/SFTP for Windows from various vendors; or, you can download and install the free PuTTY program suite that contains programs that work like SSH, SCP, and SFTP. If you download and install the full PuTTY program suite under Windows (PuTTY comes with an executable auto-installer that will do this for you), you will find the programs PSCP and PSFTP under the installation directory (usually under C:\Program Files\PuTTY). Start up a DOS command prompt, change to this directory, and run the commands you need to copy files to/from other systems (see below for examples of how to do this). Note that the line end character in text files is not the same between Unix and Windows. A text file written on Unix contains only linefeed (LF, "\n") characters at the ends of lines; Windows expects lines in text files to end in both a carriage-return (CR, "\r") *and* a linefeed character. This may result in "staircasing" text if you send a Unix text file to a Windows printer from some programs (e.g. Notepad). Print a small sample first, and try using "Write" or "Wordpad" instead of "Notepad". Conversely, if you prepare a text file on Windows and copy it to a Unix system, all the line ends will have extra CR characters on the end. The file may appear corrupt or damaged, and many programs will not deal with the extra CR characters well. Using PSCP on Windows --------------------- C:\> cd "C:\Program Files\PuTTY" C:\Program Files\PuTTY> pscp -h # -h gives a short help listing ...short help listing displays here... C:\Program Files\PuTTY> pscp -l alleni99 me.dyndns.org:.bashrc foo.txt ...you may be asked to accept the host key here (say yes)... alleni99@me.dyndns.org's password: ...file transfers ".bashrc" to "foo.txt" in the current directory... Remember to use the -l option to set your own userid. Using PSFTP on Windows ---------------------- C:\> cd "C:\Program Files\PuTTY" C:\Program Files\PuTTY> psftp -h # -h gives a short help listing ...short help listing displays here... C:\Program Files\PuTTY> psftp -l alleni99 me.dyndns.org ...you may be asked to accept the host key here (say yes)... alleni99@me.dyndns.org's password: Remote working directory is /home/alleni99 psftp> help ...short help listing displays here... psftp> ls ...listing of directory displays here... psftp> get .bashrc foo.txt remote:/home/alleni99.bashrc => local:foo.txt psftp> quit ...file "foo.txt" is now in the current directory... Remember to use -l to set your own userid when using the PuTTY version of PSFTP. Note that the options to the PuTTY Windows version of SFTP (named PSFTP) are not the same as the options to the Unix/Linux version of SFTP. In particular, the option "-P" has different meanings! Other Windows secure copy programs ---------------------------------- Look for on the Internet for "winscp".