slashbinbash

shebang!

Downloading with the RapidShare API

leave a comment »

In my attempt to script my way through some typical day-to-day obstacles I came across the RapidShare API and since I’m a fan of APIs I thought I’d share.

RapidShare is great for many reasons but not for downloading a bunch of files one after another, especially as a free user. I’ve also come to the conclusion that RapidShare is the only file hoster that I personally have the most trouble with because the waiting time before and after the download is so long that I often forget to click the download button or activate the next download thus losing a lot of time.
On the other hand RapidShare is the only file hoster that I know of which provides an API to their service, at least officially. For our purpose we need a way to check if a file is still available and if that’s the case to download it.

How it works

A normal RapidShare URL looks like this:
http://rapidshare.com/files/[ID]/[NAME]

To see if a file is still available you call:
http://api.rapidshare.com/cgi-bin/rsapi.cgi?sub=checkfiles&files=[ID]&filenames=[NAME]

You can try this in your browser, it will return a line consisting of seven comma separated values. The fifth one will tell you if the file is available or not. If it says “1″ then the file is available for anonymous, free downloading.

Next you need to make a download request by calling:
http://api.rapidshare.com/cgi-bin/rsapi.cgi?sub=download&fileid=[ID]&filename=[NAME]

It will return another line consisting of four comma separated values that are prefaced by a “DL:” unless you get an error message, then it’s “ERROR:”. The first tells you the server from which to download the file, the second one is an authentication key and the third is the amount of seconds you need to wait before requesting the actual file.

When the time is over the download should start as soon as you call:
http://[SERVER]/cgi-bin/rsapi.cgi?sub=download&fileid=[ID]&filename=[NAME]&dlauth=[AUTHKEY]

For more information check the official RapidShare API.

Implementation for bash and Python

The scripts presented below will download one RapidShare file only! The reason for this is, and I quote the RapidShare API doc:

In our terms of service, we prohibit the automation of downloads for free users. It IS allowed to create tools, which download ONE file as a free user from RapidShare. It is NOT allowed to implement queuing mechanisms.

I haven’t found any of these claims in their terms of service tho.

Both scripts take one parameter which is the RapidShare URL of the file you want to download. They do some simple error handling and MD5 checking. Of course there is room for improvement.

This is a possible bash implementation:

#!/bin/bash

# downloads one RapidShare file only
# version 2011-02-14

function busy_wait {
	TIME=$1

	while [ $TIME -ge 1 ]; do
   		echo -ne "\rwating for $TIME seconds ..."
   		sleep 1

   		TIME=$(( $TIME - 1 ))
	done

	echo -ne "\r"
}

REPEAT=1
URL="http://api.rapidshare.com/cgi-bin/rsapi.cgi"

ID=$( echo $1 | cut -d "/" -f 5 )
NAME=$( echo $1 | cut -d "/" -f 6 )

while [ $REPEAT -eq 1 ]; do

	# retrieve file information
	echo -n "checking $NAME ..."
	RET=$( wget -q -O - "$URL?sub=checkfiles&files=$ID&filenames=$NAME&incmd5=1" )

	SIZE=$( echo $RET | cut -d "," -f 3 )
	STATUS=$( echo $RET | cut -d "," -f 5 )
	MD5=$( echo $RET | cut -d "," -f 7 )

	# check if the file is available
	if [ $STATUS -eq "0" ]; then
		echo " File not found"
	elif [ $STATUS -eq "1" ]; then
		echo " OK"
	elif [ $STATUS -eq "3" ]; then
		echo " Server down"
	elif [ $STATUS -eq "4" ]; then
		echo " File marked as illegal"
	elif [ $STATUS -eq "5" ]; then
		echo " File locked (> 10 downloads)"
	else
		echo " Unknown status code"
	fi

	if [ $STATUS -ne "1" ]; then
		exit 1
	fi

	# request download information
	echo "requesting download"
	RET=$( wget -q -O - "$URL?sub=download&fileid=$ID&filename=$NAME" )

	# check if the file can be downloaded right now
	if [[ $RET == ERROR* ]]; then
		# if $RET starts with ERROR
		COUNTDOWN=$( echo $RET | cut -d " " -f 6 )

		echo "You need to wait $COUNTDOWN seconds before you can download again."
		busy_wait $COUNTDOWN
	else
		# ready for download
		REPEAT=0
	fi

done

HOSTNAME=$( echo $RET | cut -d "," -f 1 | cut -d ":" -f 2 )
DLAUTH=$( echo $RET | cut -d "," -f 2 )
COUNTDOWN=$( echo $RET | cut -d "," -f 3 )

# regular waiting time prior to download
busy_wait $COUNTDOWN

# download the actual file
echo "starting download of $NAME"
wget -O $NAME "http://$HOSTNAME/cgi-bin/rsapi.cgi?sub=download&fileid=$ID&filename=$NAME&dlauth=$DLAUTH"

# check the MD5 hashes to see if the downloaded file is complete
if [ -e $NAME ]; then
	echo -n "checking MD5 ..."
	FMD5=$( md5sum -b $NAME | cut -d " " -f 1 | tr '[:lower:]' '[:upper:]' )

	if [ $MD5 = $FMD5 ]; then
		echo " OK"
	else
		echo " FAIL"
		echo "expected: $MD5 is: $FMD5"
	fi
else
	echo "ERROR: File does not exist."
fi

And if you’re not a Linux user you may be interested in a Python version. It’s not so pretty tho.

#!/usr/bin/env python

# downloads one RapidShare file only
# version 2011-02-14

import hashlib
import os
import sys
import time
import urllib

# read from URL
def read(url):
	res = urllib.urlopen(url)
	line = res.readline()
	res.close()

	return line

# wait for number of seconds
def wait(arg0):
	sec = int(arg0)

	while sec >= 1:
   		sys.stdout.write("\rwating for %d seconds ..." % sec)
		sys.stdout.flush();
   		time.sleep(1)
   		sec -= 1
	sys.stdout.write("\r")

repeat = True
api = "http://api.rapidshare.com/cgi-bin/rsapi.cgi"

fid = sys.argv[1].split("/")[4]
fname = sys.argv[1].split("/")[5]

while repeat:
	# retrieve file information
	print "checking " + fname + " ..."
	ret = read(api + "?sub=checkfiles&files=" + fid + "&filenames=" + fname + "&incmd5=1")

	size = int(ret.split(",")[2])
	status = int(ret.split(",")[4])
	md5 = ret.split(",")[6]

	# check if the file is available
	if status == 0:
		print "File not found"
	elif status == 1:
		print "OK"
	elif status == 3:
		print "Server down"
	elif status == 4:
		print "File marked as illegal"
	elif status == 5:
		print "File locked (> 10 downloads)"
	else:
		print "Unkown status code"

	if status != 1:
		exit(1)

	# request download information
	print "requesting download"
	ret = read(api + "?sub=download&fileid=" + fid + "&filename=" + fname)

	# check if the file can be downloaded right now
	if ret.startswith("ERROR:"):
		countdown = ret.split(" ")[5]

		print "You need to wait " + countdown + " seconds before you can download again."
		wait(countdown)
	else:
		# ready for download
		repeat = False

hostname = ret.split(",")[0].split(":")[1]
dlauth = ret.split(",")[1]
countdown = ret.split(",")[2]

# regular waiting time prior to download
wait(countdown)

# download the actual file
print "starting download of " + fname
urllib.urlretrieve("http://" + hostname + "/cgi-bin/rsapi.cgi?sub=download&fileid=" + fid + "&filename=" + fname + "&dlauth=" + dlauth, fname)

# check the MD5 hashes to see if the downloaded file is complete
if os.path.exists(fname):
	print "checking MD5 ..."

	res = open(fname, 'r')
	m = hashlib.md5()
	m.update(res.read())
	val = m.hexdigest().upper()

	if val == md5:
		print "OK"
	else:
		print "FAILED"
		print "expected: " + md5 + "is:" + val
else:
	print "ERROR: file does not exist."

Next week I’ll talk about how to read a text file line by line and output its contents. ;)

Written by slashbinbash

February 15, 2011 at 12:32

Posted in computers

Tagged with , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: