curl cheat sheet

One of the best HTTP clients is the open source tool curl. With ongoing development and continuously new updates, it is worth getting everything out of this powerful tool!

Basic options

Some of the curl options are used a lot in combination with others. So it is good to know these if you are a beginner or used curl before.

Creating a shell script? Use the long format option, as this improves the readability. For quick use of curl on the command-line consider using the short notation of the related option.

HTTP protocol version

Enable data encoding (data compression)

By default, curl does no data encoding or decoding. To allow receiving compressed data, used the --compressed option, or define the Accept-Encoding header

curl --compressed --head https://example.com/

Want to test just a specific type of compression, define the header and tell what you support. Multiple options are possible using a comma as separator.

curl --head --header 'Accept-Encoding: br,gzip' https://example.com/

TLS and Certificates

With curl being able to do many things, it definitely knows how to deal with all kinds of protocols. Some interesting options in this area:

Test SSL verification

The SSL verification steps within curl can show its result using the ssl_verify_result variable.

curl --compressed --head --write-out %{ssl_verify_result} https://example.com

A value of ‘0’ means the verification went as expected.

Important: If you use this in a script, consider also checking the exit state of the curl command itself first, because it may show a ‘0’ while something like DNS resolution or the connection did went well in the first place.

Test SSL certificate status using OCSP

Perform the request and include the certification status check:

curl --cert-status --head --verbose https://example.com/

Curl will show a single line in the output to indicate the outcome of the check.

* SSL certificate status: good (0)

Show expiry date of a SSL certificate

Curl can be used to display when a SSL certificate will expire. By extracting the right line from the verbose output, we can get a single line with the information.

curl --verbose --head 2>&1 https://example.com/ | grep '\* expire date:'

This information is great for showing to a human, but not for automated processing. If we extract the specific fields and replace the month name with a two-digit number, then it may result in a more common format for storage.

curl --verbose --head 2>&1 https://example.com/ | \
    grep '\*  expire date:' | \
    sed 's/\*  expire date: //' | \
    awk '{m=sprintf("%02d", (index("JanFebMarAprMayJunJulAugSepOctNovDec",$1)+2)/3)};END{print $4"-"m"-"$2}'

Important note: This one-liner has no checks for failures such a connection errors. If you want to use this in production, consider rewriting it into a full script with proper error checks. Or use a dedicated tool for SSL certification expiry monitoring. That a tool can be used in a wide range of situations does not mean it is the best option.

Testing specific TLS versions

For security testing and troubleshooting it may be useful to test against different protocol versions. Especially if nginx configuration hardening has been applied to limit the allowable TLS versions.

If you only enable TLSv1.2 and later, we can test the minimum TLS version 1.0 en 1.1 as its max. This request then should fail:

curl --head --tls-max 1.1 --tlsv1.0 --verbose https://example.com/

If we want to set the minimum version to TLS version 1.2, it should succeed if TLSv1.2 and TLSv1.3 are allowed:

curl --head --tlsv1.2 --verbose https://example.com/

Files and downloads

The --output option can be used to write the output of a request to a file instead of the screen (stdout).

curl --output myfilename https://example.com/randomfilename.txt

Use --remote-name to save the file with the same file name as the server.

curl --remote-name https://example.com/numbers.txt

In this example, the local file to store will be named numbers.txt.

Headers

In need to change a request header? Curl makes this possible with the --header option, followed by a quoted string.

curl --header "Secret: true" https://example.com/

Specify the User-Agent string

Sometimes a download might be blocked due to server blocking curl or wget as its User-Agent string. Use the --user-agent option to define another value.

curl --user-agent "I-am-not-Firefox" https://example.com/

Automated testing

Curl can be used manually, but is also a great tool for automating tests and displaying or storing the result.

Show response headers

We can use curl to test our website and see if the right headers are returned. For example if the content-type or content-encoding is set correctly, or something like the headers related to caching.

To quickly test the output, use a HEAD request (--head) in combination with the --no-progress-meter option. This last one prevents curl showing a progress output.

curl --head --no-progress-meter https://example.com/ | \
    grep '^content-type\|^date\|^expires\|^content-encoding'

Only return the HTTP status code

With the right combination of options, we can curl to show only the HTTP status code. Great for building a small link checker or to perform a monitoring task.

curl --compressed --head --output /dev/null --silent --write-out '%{http_code}' https://example.com/

Why this combination?

• --compressed: when possible limit data traffic (optional, but typically a good idea)
• --head: don’t query full page, just the response headers
• --output >/dev/null: don’t show or store the output of the headers
• --silent: do not show any errors
• --write-out ‘%{http_code}’: only show the HTTP status code

Use a configuration file for repeated tasks

Want to test the same set of URLs, consider creating a custom configuration file.

curl --config myconfig.conf

Example configuration:

# Test main page headers, with output to screen
url = "https://example.com/"
--compressed
--connect-timeout 3
--head
--max-time 5
--no-progress-meter
user-agent = "michael-test/1.0"
               
# Test logo, send output to file, include a referrer
url = "https://example/logo.svg"
--compressed
--connect-timeout 3
--head
--max-time 5
--no-progress-meter
output = "output-logo.txt"
referer = "https://example.com/"

Note: in the HTTP spec there is a typo, so that is why it is mispelled above as well.

Measure web server performance

headers automation

Curl has a great set of variables available to display. That is nice, but it would be better if we could combine those into a neat formatted output.

The first step is to create a small shell script to do the query.

#!/bin/sh

set -o nounset

if [ $# -eq 0 ]; then echo "Error: URL required to query"; exit 1; fi

# tr-encoding requests a compressed resource, however it is better to define what we can accept. -H "Accept-Encoding: br,gzip,deflate"
curl --silent --compressed --output /dev/null --write-out "@measure-webserver-performance.format" "$1"

# EOF

Next step is to create a formatted template with the variables that you want to show.

    time_namelookup:  %{time_namelookup}\n
       time_connect:  %{time_connect}\n
    time_appconnect:  %{time_appconnect}\n
   time_pretransfer:  %{time_pretransfer}\n
      time_redirect:  %{time_redirect}\n
 time_starttransfer:  %{time_starttransfer}\n


          http_code:  %{http_code}\n
       http_version:  %{http_version}\n
             scheme:  %{scheme}\n
          remote_ip:  %{remote_ip}\n
       content_type:  %{content_type}\n
      num_redirects:  %{num_redirects}\n
  ssl_verify_result:  %{ssl_verify_result}\n

Total bytes downloaded:                 %{size_download}\n
Total bytes of the downloaded headers:  %{size_header}\n
Total bytes sent in the HTTP request:   %{size_request}\n
Total bytes that were uploaded:         %{size_upload}\n


                    ----------\n
         time_total:  %{time_total}\n

When we run the script together with an URL, we get a nice piece of output.

# ./measure-webserver-performance https://linux-audit.com
    time_namelookup:  0.002094
       time_connect:  0.013645
    time_appconnect:  0.055099
   time_pretransfer:  0.055141
      time_redirect:  0.000000
 time_starttransfer:  0.070597
          http_code:  200
       http_version:  2
             scheme:  HTTPS
          remote_ip:  89.41.171.41
       content_type:  text/html
      num_redirects:  0
  ssl_verify_result:  0
Total bytes downloaded:                 5888
Total bytes of the downloaded headers:  1322
Total bytes sent in the HTTP request:   119
Total bytes that were uploaded:         0
                    ----------
         time_total:  0.070788

Using timeouts

When doing automated testing, it makes sense to limit the amount of time a full request may take. Use the --max-time option for that.

curl --max-time 10 https://example.com/

To set the maximum time that the connection phase may take, use the --connect-time. An example to combine them.

curl --connect-timeout 3 --max-time 10 https://example.com/

Send POST with JSON data

Besides the usual HEAD and GET requests, curl can also post data. In this case it needs to be defined using the --request option. The actual data is provided by using the --data option.

curl --header "Content-Type: application/json" --request POST --data '{"name":"michael","value":"123"}' https://example.com/api/

Another option is to send so-called form-encoded data. This is also a POST request and has the type ‘application/x-www-form-urlencoded’.

curl --data 'name=michael' http://example.com/my-form/

Tip: use --data @myfilename to retrieve the data from a file, or use @’-’ to read from STDIN.

Crafting special requests

Sometimes curl is too smart for its own good. To avoid it normalizing the requested URI, such as /test/../file.txt into /file.txt, there is the --path-as-is option.

curl --path-as-is https://example.com/../../etc/passwd

This is useful for security professionals to test against some file inclusion weaknesses or trying a path traversal attack. For webmasters it is a great option to see if your block filters are working correctly.

Other useful options?

Did I miss something that really should be included in this cheat sheet? Let it know!