cURL – Swiss Army Knife for Web Sites

Lately, I have been using cURL a lot to diagnose & get more information about web pages download times, header information, automation of such tasks etc. Therefore I thought some of the very useful but not so obvious usage of cURL.

first of all, if you want to download a copy of a page to a local file, you can use

curl http://www.example.com -o local_copy.htm

If you don’t want to neccessarily download the file but only want to see response headers, you can do

curl -I http://www.example.com

If the web site is returning a redirect, 30x HTTP response code, you can ask curl to follow the redirect for example

curl -IL http://www.example.com

You can specify multiple URLs to curl as part of pattern for instance:

curl -I http://{www,play,sawaal}.ibibo.com

Output:

[1/3]: http://www.ibibo.com –> <stdout>
–_curl_–http://www.ibibo.com
HTTP/1.1 200 OK
Date: Thu, 30 Jun 2011 18:19:05 GMT
Server: Apache
Vary: Accept-Encoding
ServerID: Myibibo-34
Content-Type: text/html

[2/3]: http://play.ibibo.com –> <stdout>
–_curl_–http://play.ibibo.com
HTTP/1.1 200 OK
Date: Thu, 30 Jun 2011 18:19:08 GMT
Server: ibibo-WS
Cache-Control: no-transform
Vary: User-Agent,Accept,Accept-Encoding
ServerID: 25
Content-Type: text/html

[3/3]: http://sawaal.ibibo.com –> <stdout>
–_curl_–http://sawaal.ibibo.com
HTTP/1.1 200 OK
Date: Thu, 30 Jun 2011 18:19:08 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
X-AspNet-Version: 2.0.50727
Set-Cookie: ASP.NET_SessionId=tu5c1y555oywv4fvxj1vj245; path=/; HttpOnly
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Length: 112910

You can also specify advanced regular expressions when using the URLs for instance:

ftp://ftp.numericals.com/file[1-100].txt  ftp://ftp.numericals.com/file[001-100].txt (with leading zeros)  ftp://ftp.letters.com/file[a-z].txt

Nested sequences are not supported, but you can use several ones next to each other:

http://any.org/archive[1996-1999]/vol[1-4]/part{a,b,c}.html

You can specify any amount of URLs on the command line. They will be fetched in a sequential manner in the specified order.

You can specify a step counter for the ranges to get every Nth number or letter:

http://www.numericals.com/file[1-100:10].txt  http://www.letters.com/file[a-z:2].txt

More later in another post.

Advertisements

One thought on “cURL – Swiss Army Knife for Web Sites

  1. Pingback: How to use cURL to download multiple MP3s/Photos/Videos at once | Pradeep Sethi's Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s