1. Introduction

This extension gives access to the functions of the libcurl library, the multiprotocol file transfer library which comes with the famous cURL package.

When the STklos cURL extension library is installed, it will be available as a normal STklos library. It can be imported with the following stance:

(import (stklos curl))

Hereafter is a simple function which uses the curl library:

(define (print-content url)
  (let ((handle (curl-init)))              ; (1)
    (curl-set-option handle #:url url)     ; (2)
    (curl-set-option handle #:verbose #t)  ; (3)
    (curl-perform handle)                  ; (4)
    (curl-cleanup handle)))                ; (5)

Here, we have:

1 create a new CURL handle. This function must be called before any connection.
2 add option to the current handle. The #:url option used here tells curl the URL of the text we want to grab. The libcurl uses a long list of options that can be passed here. This is the only option that must be set, as otherwise there can be no transfer.
3 Add another option (be verbose) for this transfer. Any number of option can be added before starting data transfer.
4 Perform the data transfer as described by the previous options.
5 curl-cleanup must be called to end the CURL session and free the resources used by `libcurl for the transer
For more information on the available options, see curl_easy_setopt manual page. To use an option, such as CURLOPT_XYZ, you can use the keyword #:xyz or #:XYZ (case doesn’t matter here in fact).

Note that not all the options are supported for now. Only the options which use a string or an integer as parameter are handled by the ` curl-set-option` function.

Calling (print-content "https://example.com") permits to display the HTML content of the site https://example.com.

To grab the content of a given site, you can use the primitive get-content-as-string which is described below.

2. The (stklos curl) library

2.1. Low level primitives

This section describes the exported symbols of the (stklos curl) library whiche are (more or less) a direct port of the primitives of the Easy Interface of libcurl. The main primitives which fall in this category are the ones used in previous exemple.

STklos procedure

(curl-version)

Returns the version of libcurl as a string

(curl-version)
   => "libcurl/7.81.0 OpenSSL/1.1.1m zlib/1.2.11 brotli/1.0.9 zstd/1.5.2 libidn2/2.3.2 libpsl/0.21.1 (+libidn2/2.3.0) libssh2/1.10.0 nghttp2/1.46.0"

STklos procedure

(curl-init)

This function must be the first function to call, and it returns a CURL handle that you must use as input to other functions in the libcurl interface. This call MUST have a corresponding call to curl-cleanup when the operation is complete.

STklos procedure

(curl-set-option handle key value)

This function permits to tell libcurl how it must do the transfer. It takes an handle obtained by a previous call to curl-init, a keyword key specifying the option that must be set and the value val that must be given to this option. The type of val depends of the options to be set (see the libcurl documentation for complete list settable options).

For now, the current version of the library can only set libcurl options which accepts a string or an integer (a long for libcurl) . Boolean values can also be used for integer parameters (they are converted to 0 and 1). As a special case; Scheme ports can also be passed with the options on ports explained below.

Passing ports to curl-set-option:

By default, libcurl uses the system standard ports for its IO. They can be changed with special options which cannot easily be used in Scheme. As a consequence, curl-set-option has been extended to accept the following keywords:

  • #:iport can be used to set the Scheme port where the input must be read.

  • #:oport can be used to set the Scheme port where the output must be written.

  • #:eport can be used to set the Scheme port where the error messages must be written.

For instance, the following piece of code permits to grab the content of https://example.com as a string:

(import (stklos curl))

(let ((out (open-output-string))
      (handle (curl-init)))
  (curl-set-option handle #:url "https://example.com")
  (curl-set-option handle #:oport out)    ;; place the output in the 'out' string port
  (curl-perform handle)
  (curl-cleanup handle)
  (get-output-string out))

STklos procedure

(curl-perform handle)

Performs the transfer as described in the options given before with curl-set-option on the given CURL handle. The entire request is done and returns when done, or earlier if it fails.

STklos procedure

(curl-cleanup handle)

Closes the current session started with th handle obtained with a previous curl-init. It closes all the connections kept open by handle.

The libcurl documentation says: Do not call this function if you intend to transfer more files, re-using handles is a key to good performance with libcurl.

2.2. The curl primitive

STklos offers a high level function, called curl, which embodies all the low level details necessary to use the libcurl library.

STklos procedure

(curl url)
(curl url key1val1 …​)

The curl function creates a new session on the string url. By default, this function uses the current input, output and error port (instead of the system stdin, stdout and stderr ports used by the libcurl library. Consequently, the redirection of Scheme ports is effective with this function. Furthermore, by default, this function also set the flags

  • #:followlocation which allows to follow a possible redirection sent by the server with a HTTP header 3xx response.

  • #:referer which defines a value for the Referer: header of a HTTP request. The value is something like "STklos-curl/xxx" where xxx is a sub-string of the value given by curl-version. This option is set because some server refuse to respond to requests when there is no referer.

Exemples:

Grab a file by FTP an store its content in the file "/tmp/out"

(let ((p (open-output-file "/tmp/out")))
  (curl "ftp://ftp.gnu.org/gnu/emacs/emacs-27.2.tar.gz.sig" :oport p)
  (close-output-port p))

The following expression in equivalent of the (slurp "http://example.com") call in Clojure, which returns the content of example.com site as a string.

(with-output-to-string
  (lambda ()
    (curl "http://example.com")))
                          => "<!doctype html>\n ... </html>\n"

Use a HTTP POST request with foo=1 and bar=2, using the http://httpbin.org site

(curl "https://httpbin.org/post" #:postfields "foo=1&bar=2")

The JSON answer will be something like:

{
  "args": {},
  "data": "",
  "files": {},
  "form": {
    "bar": "2",
    "foo": "1"
  },
  "headers": {
    "Accept": "*/*",
    "Content-Length": "11",
    "Content-Type": "application/x-www-form-urlencoded",
    "Host": "httpbin.org",
    "Referer": "STklos-curl/libcurl/7.81.0",
  },
  "json": null,
  "origin": "134.59.216.151",
  "url": "https://httpbin.org/post"
}