bollywood, productivity, Search, tricks

How to use cURL to download multiple MP3s/Photos/Videos at once

In response to my previous post about cURL as a swiss army knife for web pages, a number of people have asked my about real example to use pattern based downloading of cURL, so here is a small post about that.

Disclaimer: Please note that this is for educational purpose only and illegal use is not recommended or endorsed by me.

Now, the real part:

Let’s say you wanted to download multiple files, text, data, mp3 or videos etc. Take an example of MP3 sngs for new hindi movie Delhi Belly. These songs are available on multiple sites including one URL here.

Again note, that downloading songs from this or any other site may be against local & any other applicable laws based upon where you reside.

Here are the steps

1) Open the URL.

2) You would find that the download URL is something like http://link1.songspk.pk/song1.php?songid=8520  for the first song. and the download URL for the last song is something like http://link1.songspk.pk/song1.php?songid=8529

3) Now, if you wanted to use cURL pattern base download use, you can do something like:

curl -L http://link1.songspk.pk/song1.php?songid=85{20,21,22,23,24,25,26,27,28,29} -O

-L option is to follow redirect, in case, the server is throwing some intermediate redirect.

-O option is to save them as local files rather than sending it to STDOUT.

4) This would download all files one by one and it would have having typing each file individually, handling redirects, and then saving them to local files.

5) After that you may need to rename all of them as MP3s, s you can run ren * *.mp3 (on windows) or mv * *.mp3 (on UNIX).

Happy cURL ing …

for any questions or doubts, please post in comments.

EDIT: songs.pk has moved to songspk.pk after the old domain was banned.

Advertisements
Blogosphere, Linux, productivity, Technology and Software, tricks

cURL – Swiss Army Knife for Web Sites

Lately, I have been using cURL a lot to diagnose & get more information about web pages download times, header information, automation of such tasks etc. Therefore I thought some of the very useful but not so obvious usage of cURL.

first of all, if you want to download a copy of a page to a local file, you can use

curl http://www.example.com -o local_copy.htm

If you don’t want to neccessarily download the file but only want to see response headers, you can do

curl -I http://www.example.com

If the web site is returning a redirect, 30x HTTP response code, you can ask curl to follow the redirect for example

curl -IL http://www.example.com

You can specify multiple URLs to curl as part of pattern for instance:

curl -I http://{www,play,sawaal}.ibibo.com

Output:

[1/3]: http://www.ibibo.com –> <stdout>
–_curl_–http://www.ibibo.com
HTTP/1.1 200 OK
Date: Thu, 30 Jun 2011 18:19:05 GMT
Server: Apache
Vary: Accept-Encoding
ServerID: Myibibo-34
Content-Type: text/html

[2/3]: http://play.ibibo.com –> <stdout>
–_curl_–http://play.ibibo.com
HTTP/1.1 200 OK
Date: Thu, 30 Jun 2011 18:19:08 GMT
Server: ibibo-WS
Cache-Control: no-transform
Vary: User-Agent,Accept,Accept-Encoding
ServerID: 25
Content-Type: text/html

[3/3]: http://sawaal.ibibo.com –> <stdout>
–_curl_–http://sawaal.ibibo.com
HTTP/1.1 200 OK
Date: Thu, 30 Jun 2011 18:19:08 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
X-AspNet-Version: 2.0.50727
Set-Cookie: ASP.NET_SessionId=tu5c1y555oywv4fvxj1vj245; path=/; HttpOnly
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Length: 112910

You can also specify advanced regular expressions when using the URLs for instance:

ftp://ftp.numericals.com/file[1-100].txt  ftp://ftp.numericals.com/file[001-100].txt (with leading zeros)  ftp://ftp.letters.com/file[a-z].txt

Nested sequences are not supported, but you can use several ones next to each other:

http://any.org/archive[1996-1999]/vol[1-4]/part{a,b,c}.html

You can specify any amount of URLs on the command line. They will be fetched in a sequential manner in the specified order.

You can specify a step counter for the ranges to get every Nth number or letter:

http://www.numericals.com/file[1-100:10].txt  http://www.letters.com/file[a-z:2].txt

More later in another post.