Project B

Using wget from a shell script is a way to move files from one server to the other without downloading and reuploading them. Yay!

You’re going to need hosting which allows you SSH access. DreamHost are a popular host because they offer a massive amount of space and bandwidth very cheaply.

You’re also going to need an SSH client, like Putty for Windows XP. Mac OS X users can use the Terminal which is found at /Applications/Utilities/.

1. Enable SSH/telnet access on your webhost account.
   • On DreamHost accounts, log in to your Control Panel then go to the Users tab. On the right hand side of each user, there will be an Edit link. You can use this to enable SSH access on any of your accounts. It may take up to 1 hour to become active, though.
   • All other web hosts, you will need to either contact your host or try and enable SSH access through your control panel.
2. Login and transfer files.
   • Using the Terminal or Putty, type: ssh username@domain.suffix then press return
   • At the prompt for password, type your FTP password. You won’t see what you’re typing. Press return
   • Now you need to navigate to the directory you want to upload files to. Use the “cd” command which means “change directory”:
     cd /home/yourdirectory/yourdownloaddirectory
   • Type: wget http://example.file I suggest you trial wget on a small file first.
   • If you need to stop the process at any time, you must type control-c. Even if you close Putty/the Terminal, the process will continue to run, despite you not being logged in.
   • If you want to log out of the server, type “exit”.

And that’s pretty much it. I’m not sure how it works on Putty as I have no PC on hand to test, but I suspect the instructions I give for the Terminal above will work just the same.

• For YSI files, start the download in your browser window, quickly stop it, copy the url and use it with wget.
• For password protected files, change the url of the file to http://username:password@domain.suffix/file.suffix–eg: http://bananas:monkeys@jungle.net/tiger.avi
• For File Factory, follow all the prompts until you get to the option to use FTP to transfer the file. Then copy the FTP link and use it after wget.
• For Strongspace: login, then use the file browser to start downloading the file you want. Stop the download, copy the url, then change the url like this: http://username:password@the_rest_of_the_strong_space_url

Dealing with special characters in filenames

If the filename contains symbols, then enclose the url with quotation marks. Like this:

wget “http://username:password@server.net/filename(crazy).avi”

To resume a partial upload:

Make sure that you’re uploading to the directory containing the partial file, with exactly the same name as the one you’re trying to grab. Then add -c before the url. Like this:

wget -c http://domain.net/partialfile.avi

How to wget from a site with referrer control

Just as usernames and passwords work as –user= and –passwd=, so too does –referer= pass on the referring url. Like this:

wget –referer=http://livejournal.com/ http://server.net/file.avi

As always, combining elements is just a matter of listing them all in the wget command. E.G. “resume a transfer of a file from a site that’s password protected with referrer control” would look like this:

wget -c –referer=http://referringsite.com/ –user=username –passwd=password “http://server.net/filename(withweirdcharacters).avi”