1. Introduction
This extension gives access to the functions of the libcurl library, the multiprotocol file transfer library which comes with the famous cURL package.
When the STklos cURL extension library is installed, it will be available as a normal STklos library. It can be imported with the following stance:
(import (stklos curl))
Hereafter is a simple function which uses the curl library:
(define (print-content url)
(let ((handle (curl-init))) ; (1)
(curl-set-option handle #:url url) ; (2)
(curl-set-option handle #:verbose #t) ; (3)
(curl-perform handle) ; (4)
(curl-cleanup handle))) ; (5)
Here, we have:
1 | create a new CURL handle. This function must be called before any connection. |
2 | add option to the current handle. The #:url option used here
tells curl the URL of the text we want to grab. The libcurl uses a
long list of options that can be passed here. This is the only option
that must be set, as otherwise there can be no transfer. |
3 | Add another option (be verbose) for this transfer. Any number of option can be added before starting data transfer. |
4 | Perform the data transfer as described by the previous options. |
5 | curl-cleanup must be called to end the CURL session and free the
resources used by `libcurl for the transer |
For more information on the available options, see
curl_easy_setopt
manual page. To use an option, such as CURLOPT_XYZ , you can use the
keyword #:xyz or #:XYZ (case doesn’t matter here in fact).Note that not all the options are supported for now. Only the options which use a string or an integer as parameter are handled by the ` curl-set-option` function. |
Calling (print-content "https://example.com")
permits to display the HTML
content of the site https://example.com
.
To grab the content of a given site, you can use the primitive
get-content-as-string
which is described below.
2. The (stklos curl) library
2.1. Low level primitives
This section describes the exported symbols of the (stklos curl)
library
whiche are (more or less) a direct port of the primitives of the
Easy Interface of libcurl. The main
primitives which fall in this category are the ones used in
previous exemple.
STklos procedure
Returns the version of libcurl as a string
(curl-version)
=> "libcurl/7.81.0 OpenSSL/1.1.1m zlib/1.2.11 brotli/1.0.9 zstd/1.5.2 libidn2/2.3.2 libpsl/0.21.1 (+libidn2/2.3.0) libssh2/1.10.0 nghttp2/1.46.0"
STklos procedure
This function must be the first function to call, and it returns a CURL handle
that you must use as input to other functions in the libcurl interface. This
call MUST have a corresponding call to curl-cleanup
when the operation is
complete.
STklos procedure
This function permits to tell libcurl how it must do the transfer. It takes an
handle
obtained by a previous call to curl-init
, a keyword key
specifying the option that must be set and the value val
that must be given
to this option. The type of val
depends of the options to be set (see the
libcurl documentation
for complete list settable options).
For now, the current version of the library can only set libcurl options which accepts a string or an integer (a long for libcurl) . Boolean values can also be used for integer parameters (they are converted to 0 and 1). As a special case; Scheme ports can also be passed with the options on ports explained below. |
Passing ports to curl-set-option
:
By default, libcurl
uses the system standard ports for its IO. They can be
changed with special options which cannot easily be used in Scheme. As a
consequence, curl-set-option
has been extended to accept the following keywords:
-
#:iport
can be used to set the Scheme port where the input must be read. -
#:oport
can be used to set the Scheme port where the output must be written. -
#:eport
can be used to set the Scheme port where the error messages must be written.
For instance, the following piece of code permits to grab the content of
https://example.com
as a string:
(import (stklos curl))
(let ((out (open-output-string))
(handle (curl-init)))
(curl-set-option handle #:url "https://example.com")
(curl-set-option handle #:oport out) ;; place the output in the 'out' string port
(curl-perform handle)
(curl-cleanup handle)
(get-output-string out))
STklos procedure
Performs the transfer as described in the options given before with
curl-set-option
on the given CURL handle
. The entire request is
done and returns when done, or earlier if it fails.
STklos procedure
Closes the current session started with th handle
obtained with a
previous curl-init
. It closes all the connections kept open by
handle
.
The libcurl documentation says: Do not call this function if you intend to transfer more files, re-using handles is a key to good performance with libcurl. |
2.2. The curl primitive
STklos offers a high level function, called curl,
which embodies
all the low level details necessary to use the libcurl library.
STklos procedure
The curl
function creates a new session on the string url
. By
default, this function uses the current input, output and error port
(instead of the system stdin
, stdout
and stderr
ports used by
the libcurl library. Consequently, the redirection of Scheme ports
is effective with this function. Furthermore, by default, this
function also set the flags
-
#:followlocation
which allows to follow a possible redirection sent by the server with a HTTP header 3xx response. -
#:referer
which defines a value for theReferer:
header of a HTTP request. The value is something like "STklos-curl/xxx" wherexxx
is a sub-string of the value given bycurl-version
. This option is set because some server refuse to respond to requests when there is no referer.
Exemples:
Grab a file by FTP an store its content in the file "/tmp/out"
(let ((p (open-output-file "/tmp/out")))
(curl "ftp://ftp.gnu.org/gnu/emacs/emacs-27.2.tar.gz.sig" :oport p)
(close-output-port p))
The following expression in equivalent of the (slurp "http://example.com")
call in Clojure, which returns the content of example.com
site as a string.
(with-output-to-string
(lambda ()
(curl "http://example.com")))
=> "<!doctype html>\n ... </html>\n"
Use a HTTP POST request with foo=1
and bar=2
, using the http://httpbin.org
site
(curl "https://httpbin.org/post" #:postfields "foo=1&bar=2")
The JSON answer will be something like:
{
"args": {},
"data": "",
"files": {},
"form": {
"bar": "2",
"foo": "1"
},
"headers": {
"Accept": "*/*",
"Content-Length": "11",
"Content-Type": "application/x-www-form-urlencoded",
"Host": "httpbin.org",
"Referer": "STklos-curl/libcurl/7.81.0",
},
"json": null,
"origin": "134.59.216.151",
"url": "https://httpbin.org/post"
}