curl / Docs / Tool / HTTP Scripting
Related:
curl man page
FAQ
Tutorial
Background
This document assumes that you are familiar with HTML and generalnetworking.
The increasing amount of applications moving to the web has made"HTTP Scripting" more frequently requested and wanted. To be able toautomatically extract information from the web, to fake users, to postor upload data to web servers are all important tasks today.
Curl is a command line tool for doing all sorts of URL manipulationsand transfers, but this particular document focuses on how to use itwhen doing HTTP requests for fun and profit. This documents assumes thatyou know how to invoke curl --help orcurl --manual to get basic information about it.
Curl is not written to do everything for you. It makes the requests,it gets the data, it sends data and it retrieves the information. Youprobably need to glue everything together using some kind of scriptlanguage or repeated manual invokes.
The HTTP Protocol
HTTP is the protocol used to fetch data from web servers. It is asimple protocol that is built upon TCP/IP. The protocol also allowsinformation to get sent to the server from the client using a fewdifferent methods, as is shown here.
HTTP is plain ASCII text lines being sent by the client to a serverto request a particular action, and then the server replies a few textlines before the actual requested content is sent to the client.
The client, curl, sends an HTTP request. The request contains amethod (like GET, POST, HEAD etc), a number of request headers andsometimes a request body. The HTTP server responds with a status line(indicating if things went well), response headers and most often also aresponse body. The "body" part is the plain data you requested, like theactual HTML or the image etc.
See the Protocol
Using curl's option --verbose(-v as a short option) displays what kind of commands curlsends to the server, as well as a few other informational texts.
--verbose is the single most useful option when it comesto debug or even understand the curl<->server interaction.
Sometimes even --verbose is not enough. Then --traceand --trace-asciioffer even more details as they show everything curlsends and receives. Use it like this:
curl --trace-ascii debugdump.txt http://www.example.com/See the Timing
Many times you may wonder what exactly is taking all the time, or youjust want to know the amount of milliseconds between two points in atransfer. For those, and other similar situations, the --trace-timeoption is what you need. It prepends the time to each trace outputline:
curl --trace-ascii d.txt --trace-time http://example.com/See which Transfer
When doing parallel transfers, it is relevant to see which transferis doing what. When response headers are received (and logged) you needto know which transfer these are for. --trace-idsoption is what you need. It prepends the transfer and connectionidentifier to each trace output line:
curl --trace-ascii d.txt --trace-ids http://example.com/See the Response
By default curl sends the response to stdout. You need to redirect itsomewhere to avoid that, most often that is done with -o or-O.
Spec
The Uniform Resource Locator format is how you specify the address ofa particular resource on the Internet. You know these, you have seenURLs like https://curl.se or https://example.com a million times. RFC3986 is the canonical spec. The formal name is not URL, it isURI.
Host
The hostname is usually resolved using DNS or your /etc/hosts file toan IP address and that is what curl communicates with. Alternatively youspecify the IP address directly in the URL instead of a name.
For development and other trying out situations, you can point to adifferent IP address for a hostname than what would otherwise be used,by using curl's --resolveoption:
curl --resolve www.example.org:80:127.0.0.1 http://www.example.org/Port number
Each protocol curl supports operates on a default port number, be itover TCP or in some cases UDP. Normally you do not have to take thatinto consideration, but at times you run test servers on other ports orsimilar. Then you can specify the port number in the URL with a colonand a number immediately following the hostname. Like when doing HTTP toport 1234:
curl http://www.example.org:1234/The port number you specify in the URL is the number that the serveruses to offer its services. Sometimes you may use a proxy, and then youmay need to specify that proxy's port number separately from what curlneeds to connect to the server. Like when using an HTTP proxy on port4321:
curl --proxy http://proxy.example.org:4321 http://remote.example.org/Username and password
Some services are setup to require HTTP authentication and then youneed to provide name and password which is then transferred to theremote site in various ways depending on the exact authenticationprotocol used.
You can opt to either insert the user and password in the URL or youcan provide them separately:
curl http://user:password@example.org/or
curl -u user:password http://example.org/You need to pay attention that this kind of HTTP authentication isnot what is usually done and requested by user-oriented websites thesedays. They tend to use forms and cookies instead.
Path part
The path part is just sent off to the server to request that it sendsback the associated response. The path is what is to the right side ofthe slash that follows the hostname and possibly port number.
GET
The simplest and most common request/operation made using HTTP is toGET a URL. The URL could itself refer to a webpage, an image or a file.The client issues a GET request to the server and receives the documentit asked for. If you issue the command line
curl https://curl.seyou get a webpage returned in your terminal window. The entire HTMLdocument that that URL holds.
All HTTP replies contain a set of response headers that are normallyhidden, use curl's --include(-i) option to display them as well as the rest of thedocument.
HEAD
You can ask the remote server for ONLY the headers by using the --head(-I) option which makes curl issue a HEAD request. In somespecial cases servers deny the HEAD method while others still work,which is a particular kind of annoyance.
The HEAD method is defined and made so that the server returns theheaders exactly the way it would do for a GET, but without a body. Itmeans that you may see a Content-Length: in the responseheaders, but there must not be an actual body in the HEAD response.
Multiple URLs in asingle command line
A single curl command line may involve one or many URLs. The mostcommon case is probably to just use one, but you can specify any amountof URLs. Yes any. No limits. You then get requests repeated over andover for all the given URLs.
Example, send two GET requests:
curl http://url1.example.com http://url2.example.comIf you use --data toPOST to the URL, using multiple URLs means that you send that same POSTto all the given URLs.
Example, send two POSTs:
curl --data name=curl http://url1.example.com http://url2.example.comMultiple HTTPmethods in a single command line
Sometimes you need to operate on several URLs in a single commandline and do different HTTP methods on each. For this, you might enjoythe --nextoption. It is basically a separator that separates a bunch of optionsfrom the next. All the URLs before --next get the samemethod and get all the POST data merged into one.
When curl reaches the --next on the command line, itresets the method and the POST data and allow a new set.
Perhaps this is best shown with a few examples. To send first a HEADand then a GET:
curl -I http://example.com --next http://example.comTo first send a POST and then a GET:
curl -d score=10 http://example.com/post.cgi --next http://example.com/results.htmlForms explained
Forms are the general way a website can present an HTML page withfields for the user to enter data in, and then press some kind of 'OK'or 'Submit' button to get that data sent to the server. The server thentypically uses the posted data to decide how to act. Like using theentered words to search in a database, or to add the info in a bugtracking system, display the entered address on a map or using the infoas a login-prompt verifying that the user is allowed to see what it isabout to see.
Of course there has to be some kind of program on the server end toreceive the data you send. You cannot just invent something out of theair.
GET
A GET-form uses the method GET, as specified in HTML like:
<form method="GET" action="junk.cgi"> <input type=text name="birthyear"> <input type=submit name=press value="OK"></form>In your favorite browser, this form appears with a text box to fillin and a press-button labeled "OK". If you fill in '1905' and press theOK button, your browser then creates a new URL to get for you. The URLgets junk.cgi?birthyear=1905&press=OK appended to thepath part of the previous URL.
If the original form was seen on the pagewww.example.com/when/birth.html, the second page you getbecomeswww.example.com/when/junk.cgi?birthyear=1905&press=OK.
Most search engines work this way.
To make curl do the GET form post for you, just enter the expectedcreated URL:
curl "http://www.example.com/when/junk.cgi?birthyear=1905&press=OK"POST
The GET method makes all input field names get displayed in the URLfield of your browser. That is generally a good thing when you want tobe able to bookmark that page with your given data, but it is an obviousdisadvantage if you entered secret information in one of the fields orif there are a large amount of fields creating a long and unreadableURL.
The HTTP protocol then offers the POST method. This way the clientsends the data separated from the URL and thus you do not see any of itin the URL address field.
The form would look similar to the previous one:
<form method="POST" action="junk.cgi"> <input type=text name="birthyear"> <input type=submit name=press value=" OK "></form>And to use curl to post this form with the same data filled in asbefore, we could do it like:
curl --data "birthyear=1905&press=%20OK%20" http://www.example.com/when/junk.cgiThis kind of POST uses the Content-Typeapplication/x-www-form-urlencoded and is the most widelyused POST kind.
The data you send to the server MUST already be properly encoded,curl does not do that for you. For example, if you want the data tocontain a space, you need to replace that space with %20,etc. Failing to comply with this most likely causes your data to bereceived wrongly and messed up.
Recent curl versions can in fact url-encode POST data for you, likethis:
curl --data-urlencode "name=I am Daniel" http://www.example.comIf you repeat --data several times on the command line,curl concatenates all the given data pieces - and put a& symbol between each data segment.
File Upload POST
Back in late 1995 they defined an additional way to post data overHTTP. It is documented in the RFC 1867, why this method sometimes isreferred to as RFC 1867-posting.
This method is mainly designed to better support file uploads. A formthat allows a user to upload a file could be written like this inHTML:
<form method="POST" enctype='multipart/form-data' action="upload.cgi"> <input name=upload type=file> <input type=submit name=press value="OK"></form>This clearly shows that the Content-Type about to be sent ismultipart/form-data.
To post to a form like this with curl, you enter a command linelike:
curl --form upload=@localfilename --form press=OK [URL]A common way for HTML based applications to pass state informationbetween pages is to add hidden fields to the forms. Hidden fields arealready filled in, they are not displayed to the user and they getpassed along just as all the other fields.
A similar example form with one visible field, one hidden field andone submit button could look like:
<form method="POST" action="foobar.cgi"> <input type=text name="birthyear"> <input type=hidden name="person" value="daniel"> <input type=submit name="press" value="OK"></form>To POST this with curl, you do not have to think about if the fieldsare hidden or not. To curl they are all the same:
curl --data "birthyear=1905&press=OK&person=daniel" [URL]Figure Out What A POST LooksLike
When you are about to fill in a form and send it to a server by usingcurl instead of a browser, you are of course interested in sending aPOST exactly the way your browser does.
An easy way to get to see this, is to save the HTML page with theform on your local disk, modify the 'method' to a GET, and press thesubmit button (you could also change the action URL if you want to).
You then clearly see the data get appended to the URL, separated witha ?-letter as GET forms are supposed to.
PUT
Perhaps the best way to upload data to an HTTP server is to use PUT.Then again, this of course requires that someone put a program or scripton the server end that knows how to receive an HTTP PUT stream.
Put a file to an HTTP server with curl:
curl --upload-file uploadfile http://www.example.com/receive.cgiBasic Authentication
HTTP Authentication is the ability to tell the server your usernameand password so that it can verify that you are allowed to do therequest you are doing. The Basic authentication used in HTTP (which isthe type curl uses by default) is plain text based,which means it sends username and password only slightly obfuscated, butstill fully readable by anyone that sniffs on the network between youand the remote server.
To tell curl to use a user and password for authentication:
curl --user name:password http://www.example.comOther Authentication
The site might require a different authentication method (check theheaders returned by the server), and then --ntlm,--digest,--negotiateor even --anyauthmight be options that suit you.
Proxy Authentication
Sometimes your HTTP access is only available through the use of anHTTP proxy. This seems to be especially common at various companies. AnHTTP proxy may require its own user and password to allow the client toget through to the Internet. To specify those with curl, run somethinglike:
curl --proxy-user proxyuser:proxypassword curl.seIf your proxy requires the authentication to be done using the NTLMmethod, use --proxy-ntlm,if it requires Digest use --proxy-digest.
If you use any one of these user+password options but leave out thepassword part, curl prompts for the password interactively.
Hiding credentials
Do note that when a program is run, its parameters might be possibleto see when listing the running processes of the system. Thus, otherusers may be able to watch your passwords if you pass them as plaincommand line options. There are ways to circumvent this.
It is worth noting that while this is how HTTP Authentication works,many websites do not use this concept when they provide logins etc. Seethe Web Login chapter further below for more details on that.
Referer
An HTTP request may include a 'referer' field (yes it is misspelled),which can be used to tell from which URL the client got to thisparticular resource. Some programs/scripts check the referer field ofrequests to verify that this was not arriving from an external site oran unknown page. While this is a stupid way to check something so easilyforged, many scripts still do it. Using curl, you can put anything youwant in the referer-field and thus more easily be able to fool theserver into serving your request.
Use curl to set the referer field with:
curl --referer http://www.example.come http://www.example.comUser Agent
Similar to the referer field, all HTTP requests may set theUser-Agent field. It names what user agent (client) that is being used.Many applications use this information to decide how to display pages.Silly web programmers try to make different pages for users of differentbrowsers to make them look the best possible for their particularbrowsers. They usually also do different kinds of JavaScript etc.
At times, you may learn that getting a page with curl does not returnthe same page that you see when getting the page with your browser. Thenyou know it is time to set the User Agent field to fool the server intothinking you are one of those browsers.
By default, curl uses curl/VERSION, such as User-Agent:curl/8.11.0.
To make curl look like Internet Explorer 5 on a Windows 2000 box:
curl --user-agent "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" [URL]Or why not look like you are using Netscape 4.73 on an old Linuxbox:
curl --user-agent "Mozilla/4.73 [en] (X11; U; Linux 2.2.15 i686)" [URL]Redirects
Location header
When a resource is requested from a server, the reply from the servermay include a hint about where the browser should go next to find thispage, or a new page keeping newly generated output. The header thattells the browser to redirect is Location:.
Curl does not follow Location: headers by default, butsimply displays such pages in the same manner it displays all HTTPreplies. It does however feature an option that makes it attempt tofollow the Location: pointers.
To tell curl to follow a Location:
curl --location http://www.example.comIf you use curl to POST to a site that immediately redirects you toanother page, you can safely use --location(-L) and --data/--form together.Curl only uses POST in the first request, and then revert to GET in thefollowing operations.
Other redirects
Browsers typically support at least two other ways of redirects thatcurl does not: first the html may contain a meta refresh tag that asksthe browser to load a specific URL after a set number of seconds, or itmay use JavaScript to do it.
Cookie Basics
The way the web browsers do "client side state control" is by usingcookies. Cookies are just names with associated contents. The cookiesare sent to the client by the server. The server tells the client forwhat path and hostname it wants the cookie sent back, and it also sendsan expiration date and a few more properties.
When a client communicates with a server with a name and path aspreviously specified in a received cookie, the client sends back thecookies and their contents to the server, unless of course they areexpired.
Many applications and servers use this method to connect a series ofrequests into a single logical session. To be able to use curl in suchoccasions, we must be able to record and send back cookies the way theweb application expects them. The same way browsers deal with them.
Cookie options
The simplest way to send a few cookies to the server when getting apage with curl is to add them on the command line like:
curl --cookie "name=Daniel" http://www.example.comCookies are sent as common HTTP headers. This is practical as itallows curl to record cookies simply by recording headers. Recordcookies with curl by using the --dump-header(-D) option like:
curl --dump-header headers_and_cookies http://www.example.com(Take note that the --cookie-jaroption described below is a better way to store cookies.)
Curl has a full blown cookie parsing engine built-in that comes inuse if you want to reconnect to a server and use cookies that werestored from a previous connection (or hand-crafted manually to fool theserver into believing you had a previous connection). To use previouslystored cookies, you run curl like:
curl --cookie stored_cookies_in_file http://www.example.comCurl's "cookie engine" gets enabled when you use the --cookieoption. If you only want curl to understand received cookies, use--cookie with a file that does not exist. Example, if youwant to let curl understand cookies from a page and follow a location(and thus possibly send back cookies it received), you can invoke itlike:
curl --cookie nada --location http://www.example.comCurl has the ability to read and write cookie files that use the samefile format that Netscape and Mozilla once used. It is a convenient wayto share cookies between scripts or invokes. The --cookie(-b) switch automatically detects if a given file is such acookie file and parses it, and by using the --cookie-jar(-c) option you make curl write a new cookie file at theend of an operation:
curl --cookie cookies.txt --cookie-jar newcookies.txt http://www.example.comHTTPS is HTTP secure
There are a few ways to do secure HTTP transfers. By far the mostcommon protocol for doing this is what is generally known as HTTPS, HTTPover SSL. SSL encrypts all the data that is sent and received over thenetwork and thus makes it harder for attackers to spy on sensitiveinformation.
SSL (or TLS as the current version of the standard is called) offersa set of advanced features to do secure transfers over HTTP.
Curl supports encrypted fetches when built to use a TLS library andit can be built to use one out of a fairly large set of libraries -curl -V shows which one your curl was built to use (ifany). To get a page from an HTTPS server, simply run curl like:
curl https://secure.example.comCertificates
In the HTTPS world, you use certificates to validate that you are theone you claim to be, as an addition to normal passwords. Curl supportsclient- side certificates. All certificates are locked with apassphrase, which you need to enter before the certificate can be usedby curl. The passphrase can be specified on the command line or if not,entered interactively when curl queries for it. Use a certificate withcurl on an HTTPS server like:
curl --cert mycert.pem https://secure.example.comcurl also tries to verify that the server is who it claims to be, byverifying the server's certificate against a locally stored CA certbundle. Failing the verification causes curl to deny the connection. Youmust then use --insecure(-k) in case you want to tell curl to ignore that theserver cannot be verified.
More about server certificate verification and ca cert bundles can beread in the SSLCERTSdocument.
At times you may end up with your own CA cert store and then you cantell curl to use that to verify the server's certificate:
curl --cacert ca-bundle.pem https://example.com/Modify method and headers
Doing fancy stuff, you may need to add or change elements of a singlecurl request.
For example, you can change the POST method to PROPFINDand send the data as Content-Type: text/xml (instead of thedefault Content-Type) like this:
curl --data "<xml>" --header "Content-Type: text/xml" --request PROPFIND example.comYou can delete a default header by providing one without content.Like you can ruin the request by chopping off the Host:header:
curl --header "Host:" http://www.example.comYou can add headers the same way. Your server may want aDestination: header, and you can add it:
curl --header "Destination: http://nowhere" http://example.comMore on changed methods
It should be noted that curl selects which methods to use on its owndepending on what action to ask for. -d makes a POST,-I makes a HEAD and so on. If you use the --request /-X option you can change the method keyword curl selects,but you do not modify curl's behavior. This means that if you forexample use -d "data" to do a POST, you can modify the method to aPROPFIND with -X and curl still thinks itsends a POST. You can change the normal GET to a POST method by simplyadding -X POST in a command line like:
curl -X POST http://example.org/curl however still acts as if it sent a GET so it does not send anyrequest body etc.
Some login tricks
While not strictly just HTTP related, it still causes a lot of peopleproblems so here's the executive run-down of how the vast majority ofall login forms work and how to login to them using curl.
It can also be noted that to do this properly in an automatedfashion, you most certainly need to script things and do multiple curlinvokes etc.
First, servers mostly use cookies to track the logged-in status ofthe client, so you need to capture the cookies you receive in theresponses. Then, many sites also set a special cookie on the login page(to make sure you got there through their login page) so you should makea habit of first getting the login-form page to capture the cookies setthere.
Some web-based login systems feature various amounts of JavaScript,and sometimes they use such code to set or modify cookie contents.Possibly they do that to prevent programmed logins, like this manualdescribes how to... Anyway, if reading the code is not enough to let yourepeat the behavior manually, capturing the HTTP requests done by yourbrowsers and analyzing the sent cookies is usually a working method towork out how to shortcut the JavaScript need.
In the actual <form> tag for the login, lots ofsites fill-in random/session or otherwise secretly generated hidden tagsand you may need to first capture the HTML code for the login form andextract all the hidden fields to be able to do a proper login POST.Remember that the contents need to be URL encoded when sent in a normalPOST.
Some debug tricks
Many times when you run curl on a site, you notice that the site doesnot seem to respond the same way to your curl requests as it does toyour browser's.
Then you need to start making your curl requests more similar to yourbrowser's requests:
Use the
--trace-asciioption to store fully detailedlogs of the requests for easier analyzing and betterunderstandingMake sure you check for and use cookies when needed (both readingwith
--cookieand writing with--cookie-jar)Set user-agent (with -A) to onelike a recent popular browser does
Set referer (with -E) like itis set by the browser
If you use POST, make sure you send all the fields and in thesame order as the browser does it.
Check what the browsers do
A good helper to make sure you do this right, is the web browsers'developers tools that let you view all headers you send and receive(even when using HTTPS).
A more raw approach is to capture the HTTP traffic on the networkwith tools such as Wireshark or tcpdump and check what headers that weresent and received by the browser. (HTTPS forces you to useSSLKEYLOGFILE to do that.)
