Commit 9f0db776 authored by Marek Vavrusa's avatar Marek Vavrusa
Browse files

modules/http: doc, auto-tls, cert renewal, ...

added documentation, many fixes in the H2 fallback
code and H2 stream handling, TLS is enabled by
default using ephemeral key and certificate that
is automatically renewed, but custom certificates
are also supported

this also allows other modules to place code
snippets on the webpage
parent 67afc023
.. _mod-http:
HTTP interface
HTTP/2 services
This module provides both DNS/HTTP(s) and web interface that cooperates with the internal
scheduler. It preloads all static assets, so nothing is read from disk after startup and
provides basic foundation for other services wishing to export services over HTTP endpoints.
The module supports HTTP/2, server push and all other shiny things thanks to lua-http.
This is a module that does the heavy lifting to provide an HTTP/2 enabled
server that supports TLS by default and provides endpoint for other modules
in order to enable them to export restful APIs and websocket streams.
One example is statistics module that can stream live metrics on the website,
or publish metrics on request for Prometheus scraper.
The server allows other modules to either use default endpoint that provides
built-in webpage, restful APIs and websocket streams, or create new endpoints.
Example configuration
By default, the web interface starts at port 8053 if HTTP or 4453 if running on TLS.
By default, the web interface starts HTTPS/2 on port 8053 using an ephemeral
certificate that is valid for 90 days and is automatically renewed. It is of
course self-signed, so you should use your own judgement before exposing it
to the outside world. Why not use something like `Let's Encrypt <>`_
for starters?
.. code-block:: lua
-- Load modules
-- Load HTTP module with defaults
modules = {
http = {
host = 'localhost',
port = 8080,
host = 'localhost',
port = 8053,
Now you can reach the web services and APIs, done!
.. code-block:: bash
$ curl -k https://localhost:8053
$ curl -k https://localhost:8053/stats
It is possible to disable HTTPS altogether by passing ``cert = false`` option.
While it's not recommended, it could be fine for localhost tests as, for example,
Safari doesn't allow WebSockets over HTTPS with a self-signed certificate.
Major drawback is that current browsers won't do HTTP/2 over insecure connection.
.. code-block:: lua
http = {
host = 'localhost',
port = 8053,
cert = false,
If you want to provide your own certificate and key, you're welcome to do so:
.. code-block:: lua
http = {
host = 'localhost',
port = 8053,
cert = 'mycert.crt',
key = 'mykey.key',
The format of both certificate and key is expected to be PEM, e.g. equivallent to
the outputs of following:
.. code-block:: bash
openssl ecparam -genkey -name prime256v1 -out mykey.key
openssl req -new -key mykey.key -out csr.pem
openssl req -x509 -days 90 -key mykey.key -in csr.pem -out mycert.crt
How to expose services over HTTP
The module provides a table ``endpoints`` of already existing endpoints, it is free for reading and
writing. It contains tables describing a triplet - ``{mime, on_serve, on_websocket}``.
In order to register a new service, simply add it to the table:
.. code-block:: lua
http.endpoints['/health'] = {'application/json',
function (h, stream)
-- API call, return a JSON table
return {state = 'up', uptime = 0}
function (h, ws)
-- Stream current status every second
local ok = true
while ok do
local push = tojson('up')
ok = ws:send(tojson({'up'}))
-- Finalize the WebSocket
Then you can query the API endpoint, or tail the WebSocket using curl.
.. code-block:: bash
$ curl -k http://localhost:8053/health
$ curl -k -i -N -H "Connection: Upgrade" -H "Upgrade: websocket" -H "Host: localhost:8053/health" -H "Sec-Websocket-Key: nope" -H "Sec-Websocket-Version: 13" https://localhost:8053/health
HTTP/1.1 101 Switching Protocols
upgrade: websocket
sec-websocket-accept: eg18mwU7CDRGUF1Q+EJwPM335eM=
connection: upgrade
Since the stream handlers are effectively coroutines, you are free to keep state and yield using cqueues.
This is especially useful for WebSockets, as you can stream content in a simple loop instead of
chains of callbacks.
Last thing you can publish from modules are *"snippets"*. Snippets are plain pieces of HTML code that are rendered at the end of the built-in webpage. The snippets can be extended with JS code to talk to already
exported restful APIs and subscribe to WebSockets.
.. code-block:: lua
http.snippets['/health'] = {'Health service', '<p>UP!</p>'}
How to expose more interfaces
Services exposed in the previous part share the same external interface. This means that it's either accessible to the outside world or internally, but not one or another. This is not always desired, i.e. you might want to offer DNS/HTTPS to everyone, but allow application firewall configuration only on localhost. ``http`` module allows you to create additional interfaces with custom endpoints for this purpose.
.. code-block:: lua
http.interface('', 8080, {
['/conf'] = {'application/json', function (h, stream) print('configuration API') end},
['/private'] = {'text/html', static_page},
This way you can have different internal-facing and external-facing services at the same time.
-- This is a module that does the heavy lifting to provide an HTTP/2 enabled
-- server that supports TLS by default and provides endpoint for other modules
-- in order to enable them to export restful APIs and websocket streams.
-- One example is statistics module that can stream live metrics on the website,
-- or publish metrics on request for Prometheus scraper.
local cqueues = require('cqueues')
local ce = require('cqueues.errno')
local server = require('http.server')
local headers = require('http.headers')
local websocket = require('http.websocket')
local kres = require('kres')
local x509, pkey = require('openssl.x509'), require('openssl.pkey')
-- Module declaration
local cq =
......@@ -11,10 +15,79 @@ local M = {
servers = {},
-- Map extensions to MIME type
local mime_types = {
js = 'application/javascript',
css = 'text/css',
tpl = 'text/html',
ico = 'image/x-icon'
-- Preload static contents, nothing on runtime will touch the disk
local function pgload(relpath)
local fp, err ='/http/'..relpath, 'r')
if not fp then error(err) end
local data = fp:read('*all')
-- Guess content type
local ext = relpath:match('[^\\.]+$')
return {'/'..relpath, mime_types[ext] or 'text', data}
-- Preloaded static assets
local pages = {
-- Serve preloaded root page
local function serve_root()
local _, mime_root, mime_data = unpack(pgload('main.tpl'))
mime_data = mime_data
:gsub('{{ title }}', 'kresd @ '..hostname())
:gsub('{{ host }}', hostname())
return function (h, stream)
-- Return index page
local rsnippets = {}
for _,v in pairs(M.snippets) do
table.insert(rsnippets, string.format('<h2>%s</h2>\n%s', v[1], v[2]))
local data = mime_data
:gsub('{{ secure }}', stream:checktls() and 'true' or 'false')
:gsub('{{ snippets }}', table.concat(rsnippets, '\n'))
local hsend =
hsend:append(':status', '200')
hsend:append('content/type', mime_root)
assert(stream:write_headers(hsend, false))
assert(stream:write_chunk(data, true))
-- Push assets
-- local path, mime, data = unpack(pages[1])
-- local hpush =
-- hpush:append(':scheme', h:get(':scheme'))
-- hpush:append(':method', 'GET')
-- hpush:append(':authority', h:get(':authority'))
-- hpush:append(':path', path)
-- local nstream = stream:push_promise(hpush)
-- hpush =
-- hpush:append(':status', '200')
-- hpush:append('content/type', mime)
-- print('pushing', path)
-- assert(nstream:write_headers(hpush, false))
-- assert(nstream:write_chunk(data, true))
-- Do not send anything else
return false
-- Load dependent modules
if not stats then modules.load('stats') end
-- Function to sort frequency list
local function freqsort(a, b) return a.count < b.count end
local function stream_stats(h, ws)
local ok, prev = true, stats.list()
while ok do
......@@ -24,55 +97,27 @@ local function stream_stats(h, ws)
stats_dt[k] = v - (prev[k] or 0)
prev = cur
-- Update frequent query list
local cur, freq = stats.frequent(), {}
table.sort(cur, freqsort)
for i = 1,math.min(20, #cur) do
table.insert(freq, cur[i])
-- Publish stats updates periodically
local push = tojson({stats=stats_dt,freq=freq})
local push = tojson({stats=stats_dt})
ok = ws:send(push)
-- Preload static contents, nothing on runtime will touch the disk
local function pgload(relpath)
local fp, err ='/http/'..relpath, 'r')
if not fp then error(err) end
local data = fp:read('*all')
return data
local pages = {
root = pgload('main.tpl'):gsub('{{.Title}}', 'kresd @ '..hostname()),
rootjs = pgload('tinyweb.js'),
datamaps = pgload(''),
topojson = pgload('topojson.js'),
jquery = pgload('jquery.js'),
epochcss = pgload('epoch.css'),
epoch = pgload('epoch.js'),
favicon = pgload('favicon.ico'),
d3 = pgload('d3.js'),
-- Export HTTP service endpoints
M.endpoints = {
['/'] = {'text/html', pages.root},
['/tinyweb.js'] = {'application/json', pages.rootjs},
['/'] = {'application/json', pages.datamaps},
['/topojson.js'] = {'application/json', pages.topojson},
['/jquery.js'] = {'application/json', pages.jquery},
['/epoch.js'] = {'application/json', pages.epoch},
['/epoch.css'] = {'text/css', pages.epochcss},
['/favicon.ico'] = {'text/html', pages.favicon},
['/d3.js'] = {'text/html', pages.d3},
['/'] = {'text/html', serve_root()},
['/stats'] = {'application/json', stats.list, stream_stats},
['/feed'] = {'application/json', stats.frequent},
for _, pg in ipairs(pages) do
local path, mime, data = unpack(pg)
M.endpoints[path] = {mime, data}
-- Export HTTP service page snippets
M.snippets = {}
-- Serve GET requests, we only support a fixed
-- number of endpoints that are actually preloaded
......@@ -80,31 +125,42 @@ M.endpoints = {
local function serve_get(h, stream)
local hsend =
local path = h:get(':path')
local ctype, data = M.endpoints[path]
-- Unpack ctype
if ctype then
ctype, data = unpack(ctype)
local entry = M.endpoints[path]
-- Unpack MIME and data
local mime, data
if entry then
mime, data = unpack(entry)
-- Get string data out of service endpoint
if type(data) == 'function' then data = data(h) end
if type(data) == 'function' then
data = data(h, stream)
-- Handler doesn't provide any data
if data == false then return end
if type(data) == 'table' then data = tojson(data) end
if not ctype or type(data) ~= 'string' then
if not mime or type(data) ~= 'string' then
hsend:append(':status', '404')
hsend:append('connection', 'close')
assert(stream:write_headers(hsend, true))
assert(stream:write_headers(hsend, false))
-- Serve content type appropriately
hsend:append(':status', '200')
hsend:append('content/type', ctype)
hsend:append('connection', 'close')
hsend:append('content/type', mime)
assert(stream:write_headers(hsend, false))
assert(stream:write_chunk(data, true))
-- Web server service closure
function M.route(endpoints)
local function route(endpoints)
return function (stream)
-- HTTP/2: We're only permitted to send in open/half-closed (remote)
local connection = stream.connection
if connection.version >= 2 then
if stream.state ~= 'open' and stream.state ~= 'half closed (remote)' then
-- Start reading headers
local h = assert(stream:get_headers())
local m = h:get(':method')
local path = h:get(':path')
......@@ -126,29 +182,136 @@ function M.route(endpoints)
-- Method is not supported
local hsend =
hsend:append(':status', '500')
hsend:append('connection', 'close')
assert(stream:write_headers(hsend, true))
assert(stream:write_headers(hsend, false))
-- Close multiplexed HTTP/2 connection only when empty
if connection.version < 2 or connection.new_streams:length() == 0 then
-- @function Create self-signed certificate
local function ephemeralcert(host)
-- Import luaossl directly
local name = require('')
local altname = require('openssl.x509.altname')
-- Create self-signed certificate
host = host or hostname()
local crt =
local now = os.time()
local dn =
dn:add("CN", host)
local alt =
alt:add("DNS", host)
-- Valid for 90 days
crt:setLifetime(now, now + 90*60*60*24)
-- Can't be used as a CA
-- Create and set key (default: EC/P-256 as a most "interoperable")
local key = {type = 'EC', curve = 'prime256v1'}
return crt, key
-- @function Prefer HTTP/2 or HTTP/1.1
local function alpnselect(_, protos)
for _, proto in ipairs(protos) do
if proto == 'h2' or proto == 'http/1.1' then
return proto
return nil
-- @function Create TLS context
local function tlscontext(crt, key)
local http_tls = require('http.tls')
local ctx = http_tls.new_server_context()
if ctx.setAlpnSelect then
return ctx
-- @function Refresh self-signed certificates
local function updatecert(crtfile, keyfile)
local f = assert(, 'w'), string.format('cannot open "%s" for writing', crtfile))
local crt, key = ephemeralcert()
-- Write back to file
f = assert(, 'w'), string.format('cannot open "%s" for writing', keyfile))
local pub, priv = key:toPEM('public', 'private')
return crt, key
-- @function Listen on given HTTP(s) host
function M.listen(m, host, port, cb, cert)
function M.interface(host, port, endpoints, crtfile, keyfile)
local crt, key, ephemeral
if crtfile ~= false then
-- Check if the cert file exists
if not crtfile then
crtfile = 'self.crt'
keyfile = 'self.key'
ephemeral = true
else error('certificate provided, but missing key') end
-- Read or create self-signed x509 certificate
local f =, 'r')
if f then
crt = assert('*all')))
-- Continue reading key file
if crt then
f =, 'r')
key = assert('*all')))
elseif ephemeral then
crt, key = updatecert(crtfile, keyfile)
-- Check loaded certificate
if not crt or not key then
error(string.format('failed to load certificate "%s" - %s', crtfile, err or 'error'))
-- Create TLS context and start listening
local s, err = server.listen {
host = host,
port = port,
tls = crt ~= nil,
ctx = crt and tlscontext(crt, key),
if not s then
error(string.format('failed to listen on %s#%d: %s', host, port, err))
-- Compose server handler
local routes = route(endpoints)
cq:wrap(function ()
table.insert(M.servers, s)
-- Create certificate renewal timer if ephemeral
if crt and ephemeral then
local _, expiry = crt:getLifetime()
expiry = math.max(0, expiry - (os.time() - 3 * 24 * 3600))
event.after(expiry, function (ev)
print('[http] refreshed ephemeral certificate')
crt, key = updatecert(crtfile, keyfile)
s.ctx = tlscontext(crt, key)
-- @function Cleanup module
......@@ -156,25 +319,28 @@ function M.deinit()
if M.ev then event.cancel(M.ev) end
M.servers = {}
-- @function Configure module
local ffi = require('ffi')
function M.config(conf)
assert(type(conf) == 'table', 'config { host = "...", port = 443, cert = "..." }')
conf = conf or {}
assert(type(conf) == 'table', 'config { host = "...", port = 443, cert = "...", key = "..." }')
-- Configure web interface for resolver
if not conf.port then conf.port = conf.cert and 80 or 443 end
if not then = 'localhost' end
M:listen(, conf.port, M.route(M.endpoints))
if not conf.port then conf.port = 8053 end
if not then = 'localhost' end
M.interface(, conf.port, M.endpoints, conf.cert, conf.key)
-- TODO: configure DNS/HTTP(s) interface
-- M:listen(, conf.dns/port, serve_web)
if M.ev then return end
-- Schedule both I/O activity notification and timeouts
local poll_step
poll_step = function (ev, status, events)
local ok, err, _, co = cq:step(0)
if not ok then print('[http] '..err, debug.traceback(co)) end
if not ok then print('[http]', err, debug.traceback(co)) end
-- Reschedule timeout or create new one
local timeout = cq:timeout()
if timeout then
-- Throttle web requests
if timeout == 0 then timeout = 0.001 end
-- Convert from seconds to duration
timeout = timeout * sec
if not M.timeout then
// Colour palette
var colours = [
// Unit conversion
function tounit(d) {
d = parseInt(d);
......@@ -25,6 +35,27 @@ window.onload = function() {
data: statsHistory
var fills = { defaultFill: '#F5F5F5' };
for (var i in colours) {
fills['q' + i] = colours[i];
var map = new Datamap({
element: document.getElementById('map'),
fills: fills,
data: {},
height: 350,
geographyConfig: {
highlightOnHover: false,
borderColor: '#ccc',
borderWidth: 0.5,
popupTemplate: function(geo, data) {
return ['<div class="hoverinfo">',
'<strong>',, '</strong>',
'<br>Queries: <strong>', data ? data.queries : '0', '</strong>',
* Realtime updates over WebSockets
......@@ -52,7 +83,7 @@ window.onload = function() {
/* WebSocket endpoints */
var wsStats = 'ws://' + + '/stats';
var wsStats = (secure ? 'wss://' : 'ws://') + + '/stats';
var Socket = "MozWebSocket" in window ? MozWebSocket : WebSocket;
var ws = new Socket(wsStats);
ws.onmessage = function(evt) {
<!DOCTYPE html>
<meta charset="utf-8">
<title>{{ title }}</title>
body { font-family: 'Gill Sans', 'Gill Sans MT', Verdana, sans-serif; color: #555; }
h1, h2, h3 { line-height: 1.5em; color: #000; text-align: center; border-bottom: 1px solid #ccc; }
......@@ -13,24 +14,27 @@
#stats .layer-100ms , .l-100ms { fill: #258FDA; color: #258FDA; }
#stats .layer-1000ms , .l-1000ms { fill: #51A5E1; color: #51A5E1; }
#stats .layer-slow , .l-slow { fill: #E1AC51; color: #E1AC51; }
#feed { width: 100%; }
#feed .secure { color: #74c476; }
.stats-legend { text-align: center; }
.stats-legend li { display: inline; list-style-type: none; padding-right: 20px; }
.map-legend { font-size: 10px; }
<script type="text/javascript">
var host = "{{ host }}";
var secure = {{ secure }};
<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="d3.js"></script>
<script type="text/javascript" src="epoch.js"></script>
<script type="text/javascript" src="topojson.js"></script>
<script type="text/javascript" src=""></script>