Commit 9f0db776 authored by Marek Vavrusa's avatar Marek Vavrusa
Browse files

modules/http: doc, auto-tls, cert renewal, ...

added documentation, many fixes in the H2 fallback
code and H2 stream handling, TLS is enabled by
default using ephemeral key and certificate that
is automatically renewed, but custom certificates
are also supported

this also allows other modules to place code
snippets on the webpage
parent 67afc023
.. _mod-http:
HTTP interface
--------------
HTTP/2 services
---------------
This module provides both DNS/HTTP(s) and web interface that cooperates with the internal
scheduler. It preloads all static assets, so nothing is read from disk after startup and
provides basic foundation for other services wishing to export services over HTTP endpoints.
The module supports HTTP/2, server push and all other shiny things thanks to lua-http.
This is a module that does the heavy lifting to provide an HTTP/2 enabled
server that supports TLS by default and provides endpoint for other modules
in order to enable them to export restful APIs and websocket streams.
One example is statistics module that can stream live metrics on the website,
or publish metrics on request for Prometheus scraper.
The server allows other modules to either use default endpoint that provides
built-in webpage, restful APIs and websocket streams, or create new endpoints.
Example configuration
^^^^^^^^^^^^^^^^^^^^^
By default, the web interface starts at port 8053 if HTTP or 4453 if running on TLS.
By default, the web interface starts HTTPS/2 on port 8053 using an ephemeral
certificate that is valid for 90 days and is automatically renewed. It is of
course self-signed, so you should use your own judgement before exposing it
to the outside world. Why not use something like `Let's Encrypt <https://letsencrypt.org>`_
for starters?
.. code-block:: lua
-- Load modules
-- Load HTTP module with defaults
modules = {
http = {
host = 'localhost',
port = 8080,
host = 'localhost',
port = 8053,
}
}
Now you can reach the web services and APIs, done!
.. code-block:: bash
$ curl -k https://localhost:8053
$ curl -k https://localhost:8053/stats
It is possible to disable HTTPS altogether by passing ``cert = false`` option.
While it's not recommended, it could be fine for localhost tests as, for example,
Safari doesn't allow WebSockets over HTTPS with a self-signed certificate.
Major drawback is that current browsers won't do HTTP/2 over insecure connection.
.. code-block:: lua
http = {
host = 'localhost',
port = 8053,
cert = false,
}
If you want to provide your own certificate and key, you're welcome to do so:
.. code-block:: lua
http = {
host = 'localhost',
port = 8053,
cert = 'mycert.crt',
key = 'mykey.key',
}
The format of both certificate and key is expected to be PEM, e.g. equivallent to
the outputs of following:
.. code-block:: bash
openssl ecparam -genkey -name prime256v1 -out mykey.key
openssl req -new -key mykey.key -out csr.pem
openssl req -x509 -days 90 -key mykey.key -in csr.pem -out mycert.crt
How to expose services over HTTP
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The module provides a table ``endpoints`` of already existing endpoints, it is free for reading and
writing. It contains tables describing a triplet - ``{mime, on_serve, on_websocket}``.
In order to register a new service, simply add it to the table:
.. code-block:: lua
http.endpoints['/health'] = {'application/json',
function (h, stream)
-- API call, return a JSON table
return {state = 'up', uptime = 0}
end,
function (h, ws)
-- Stream current status every second
local ok = true
while ok do
local push = tojson('up')
ok = ws:send(tojson({'up'}))
require('cqueues').sleep(1)
end
-- Finalize the WebSocket
ws:close()
end}
Then you can query the API endpoint, or tail the WebSocket using curl.
.. code-block:: bash
$ curl -k http://localhost:8053/health
{"state":"up","uptime":0}
$ curl -k -i -N -H "Connection: Upgrade" -H "Upgrade: websocket" -H "Host: localhost:8053/health" -H "Sec-Websocket-Key: nope" -H "Sec-Websocket-Version: 13" https://localhost:8053/health
HTTP/1.1 101 Switching Protocols
upgrade: websocket
sec-websocket-accept: eg18mwU7CDRGUF1Q+EJwPM335eM=
connection: upgrade
?["up"]?["up"]?["up"]
Since the stream handlers are effectively coroutines, you are free to keep state and yield using cqueues.
This is especially useful for WebSockets, as you can stream content in a simple loop instead of
chains of callbacks.
Last thing you can publish from modules are *"snippets"*. Snippets are plain pieces of HTML code that are rendered at the end of the built-in webpage. The snippets can be extended with JS code to talk to already
exported restful APIs and subscribe to WebSockets.
.. code-block:: lua
http.snippets['/health'] = {'Health service', '<p>UP!</p>'}
How to expose more interfaces
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Services exposed in the previous part share the same external interface. This means that it's either accessible to the outside world or internally, but not one or another. This is not always desired, i.e. you might want to offer DNS/HTTPS to everyone, but allow application firewall configuration only on localhost. ``http`` module allows you to create additional interfaces with custom endpoints for this purpose.
.. code-block:: lua
http.interface('127.0.0.1', 8080, {
['/conf'] = {'application/json', function (h, stream) print('configuration API') end},
['/private'] = {'text/html', static_page},
})
This way you can have different internal-facing and external-facing services at the same time.
Dependencies
^^^^^^^^^^^^
......
-- This is a module that does the heavy lifting to provide an HTTP/2 enabled
-- server that supports TLS by default and provides endpoint for other modules
-- in order to enable them to export restful APIs and websocket streams.
-- One example is statistics module that can stream live metrics on the website,
-- or publish metrics on request for Prometheus scraper.
local cqueues = require('cqueues')
local ce = require('cqueues.errno')
local server = require('http.server')
local headers = require('http.headers')
local websocket = require('http.websocket')
local kres = require('kres')
local x509, pkey = require('openssl.x509'), require('openssl.pkey')
-- Module declaration
local cq = cqueues.new()
......@@ -11,10 +15,79 @@ local M = {
servers = {},
}
-- Map extensions to MIME type
local mime_types = {
js = 'application/javascript',
css = 'text/css',
tpl = 'text/html',
ico = 'image/x-icon'
}
-- Preload static contents, nothing on runtime will touch the disk
local function pgload(relpath)
local fp, err = io.open(moduledir..'/http/'..relpath, 'r')
if not fp then error(err) end
local data = fp:read('*all')
fp:close()
-- Guess content type
local ext = relpath:match('[^\\.]+$')
return {'/'..relpath, mime_types[ext] or 'text', data}
end
-- Preloaded static assets
local pages = {
pgload('kresd.js'),
pgload('datamaps.world.min.js'),
pgload('topojson.js'),
pgload('jquery.js'),
pgload('epoch.css'),
pgload('epoch.js'),
pgload('favicon.ico'),
pgload('d3.js'),
}
-- Serve preloaded root page
local function serve_root()
local _, mime_root, mime_data = unpack(pgload('main.tpl'))
mime_data = mime_data
:gsub('{{ title }}', 'kresd @ '..hostname())
:gsub('{{ host }}', hostname())
return function (h, stream)
-- Return index page
local rsnippets = {}
for _,v in pairs(M.snippets) do
table.insert(rsnippets, string.format('<h2>%s</h2>\n%s', v[1], v[2]))
end
local data = mime_data
:gsub('{{ secure }}', stream:checktls() and 'true' or 'false')
:gsub('{{ snippets }}', table.concat(rsnippets, '\n'))
local hsend = headers.new()
hsend:append(':status', '200')
hsend:append('content/type', mime_root)
assert(stream:write_headers(hsend, false))
assert(stream:write_chunk(data, true))
-- Push assets
-- local path, mime, data = unpack(pages[1])
-- local hpush = headers.new()
-- hpush:append(':scheme', h:get(':scheme'))
-- hpush:append(':method', 'GET')
-- hpush:append(':authority', h:get(':authority'))
-- hpush:append(':path', path)
-- local nstream = stream:push_promise(hpush)
-- hpush = headers.new()
-- hpush:append(':status', '200')
-- hpush:append('content/type', mime)
-- print('pushing', path)
-- assert(nstream:write_headers(hpush, false))
-- assert(nstream:write_chunk(data, true))
-- Do not send anything else
return false
end
end
-- Load dependent modules
if not stats then modules.load('stats') end
-- Function to sort frequency list
local function freqsort(a, b) return a.count < b.count end
local function stream_stats(h, ws)
local ok, prev = true, stats.list()
while ok do
......@@ -24,55 +97,27 @@ local function stream_stats(h, ws)
stats_dt[k] = v - (prev[k] or 0)
end
prev = cur
-- Update frequent query list
local cur, freq = stats.frequent(), {}
table.sort(cur, freqsort)
for i = 1,math.min(20, #cur) do
table.insert(freq, cur[i])
end
-- Publish stats updates periodically
local push = tojson({stats=stats_dt,freq=freq})
local push = tojson({stats=stats_dt})
ok = ws:send(push)
cqueues.sleep(0.5)
end
ws:close()
end
-- Preload static contents, nothing on runtime will touch the disk
local function pgload(relpath)
local fp, err = io.open(moduledir..'/http/'..relpath, 'r')
if not fp then error(err) end
local data = fp:read('*all')
fp:close()
return data
end
local pages = {
root = pgload('main.tpl'):gsub('{{.Title}}', 'kresd @ '..hostname()),
rootjs = pgload('tinyweb.js'),
datamaps = pgload('datamaps.world.min.js'),
topojson = pgload('topojson.js'),
jquery = pgload('jquery.js'),
epochcss = pgload('epoch.css'),
epoch = pgload('epoch.js'),
favicon = pgload('favicon.ico'),
d3 = pgload('d3.js'),
}
-- Export HTTP service endpoints
M.endpoints = {
['/'] = {'text/html', pages.root},
['/tinyweb.js'] = {'application/json', pages.rootjs},
['/datamaps.world.min.js'] = {'application/json', pages.datamaps},
['/topojson.js'] = {'application/json', pages.topojson},
['/jquery.js'] = {'application/json', pages.jquery},
['/epoch.js'] = {'application/json', pages.epoch},
['/epoch.css'] = {'text/css', pages.epochcss},
['/favicon.ico'] = {'text/html', pages.favicon},
['/d3.js'] = {'text/html', pages.d3},
['/'] = {'text/html', serve_root()},
['/stats'] = {'application/json', stats.list, stream_stats},
['/feed'] = {'application/json', stats.frequent},
}
for _, pg in ipairs(pages) do
local path, mime, data = unpack(pg)
M.endpoints[path] = {mime, data}
end
-- Export HTTP service page snippets
M.snippets = {}
-- Serve GET requests, we only support a fixed
-- number of endpoints that are actually preloaded
......@@ -80,31 +125,42 @@ M.endpoints = {
local function serve_get(h, stream)
local hsend = headers.new()
local path = h:get(':path')
local ctype, data = M.endpoints[path]
-- Unpack ctype
if ctype then
ctype, data = unpack(ctype)
local entry = M.endpoints[path]
-- Unpack MIME and data
local mime, data
if entry then
mime, data = unpack(entry)
end
-- Get string data out of service endpoint
if type(data) == 'function' then data = data(h) end
if type(data) == 'function' then
data = data(h, stream)
-- Handler doesn't provide any data
if data == false then return end
end
if type(data) == 'table' then data = tojson(data) end
if not ctype or type(data) ~= 'string' then
if not mime or type(data) ~= 'string' then
hsend:append(':status', '404')
hsend:append('connection', 'close')
assert(stream:write_headers(hsend, true))
assert(stream:write_headers(hsend, false))
else
-- Serve content type appropriately
hsend:append(':status', '200')
hsend:append('content/type', ctype)
hsend:append('connection', 'close')
hsend:append('content/type', mime)
assert(stream:write_headers(hsend, false))
assert(stream:write_chunk(data, true))
end
end
-- Web server service closure
function M.route(endpoints)
local function route(endpoints)
return function (stream)
-- HTTP/2: We're only permitted to send in open/half-closed (remote)
local connection = stream.connection
if connection.version >= 2 then
if stream.state ~= 'open' and stream.state ~= 'half closed (remote)' then
return
end
end
-- Start reading headers
local h = assert(stream:get_headers())
local m = h:get(':method')
local path = h:get(':path')
......@@ -126,29 +182,136 @@ function M.route(endpoints)
-- Method is not supported
local hsend = headers.new()
hsend:append(':status', '500')
hsend:append('connection', 'close')
assert(stream:write_headers(hsend, true))
assert(stream:write_headers(hsend, false))
end
stream:shutdown()
stream.connection:shutdown()
-- Close multiplexed HTTP/2 connection only when empty
if connection.version < 2 or connection.new_streams:length() == 0 then
connection:shutdown()
end
end
end
-- @function Create self-signed certificate
local function ephemeralcert(host)
-- Import luaossl directly
local name = require('openssl.x509.name')
local altname = require('openssl.x509.altname')
-- Create self-signed certificate
host = host or hostname()
local crt = x509.new()
local now = os.time()
crt:setSerial(now)
local dn = name.new()
dn:add("CN", host)
crt:setSubject(dn)
local alt = altname.new()
alt:add("DNS", host)
crt:setSubjectAlt(alt)
-- Valid for 90 days
crt:setLifetime(now, now + 90*60*60*24)
-- Can't be used as a CA
crt:setBasicConstraints{CA=false}
crt:setBasicConstraintsCritical(true)
-- Create and set key (default: EC/P-256 as a most "interoperable")
local key = pkey.new {type = 'EC', curve = 'prime256v1'}
crt:setPublicKey(key)
crt:sign(key)
return crt, key
end
-- @function Prefer HTTP/2 or HTTP/1.1
local function alpnselect(_, protos)
for _, proto in ipairs(protos) do
if proto == 'h2' or proto == 'http/1.1' then
return proto
end
end
return nil
end
-- @function Create TLS context
local function tlscontext(crt, key)
local http_tls = require('http.tls')
local ctx = http_tls.new_server_context()
if ctx.setAlpnSelect then
ctx:setAlpnSelect(alpnselect)
end
assert(ctx:setPrivateKey(key))
assert(ctx:setCertificate(crt))
return ctx
end
-- @function Refresh self-signed certificates
local function updatecert(crtfile, keyfile)
local f = assert(io.open(crtfile, 'w'), string.format('cannot open "%s" for writing', crtfile))
local crt, key = ephemeralcert()
-- Write back to file
f:write(tostring(crt))
f:close()
f = assert(io.open(keyfile, 'w'), string.format('cannot open "%s" for writing', keyfile))
local pub, priv = key:toPEM('public', 'private')
assert(f:write(pub..priv))
f:close()
return crt, key
end
-- @function Listen on given HTTP(s) host
function M.listen(m, host, port, cb, cert)
function M.interface(host, port, endpoints, crtfile, keyfile)
local crt, key, ephemeral
if crtfile ~= false then
-- Check if the cert file exists
if not crtfile then
crtfile = 'self.crt'
keyfile = 'self.key'
ephemeral = true
else error('certificate provided, but missing key') end
-- Read or create self-signed x509 certificate
local f = io.open(crtfile, 'r')
if f then
crt = assert(x509.new(f:read('*all')))
f:close()
-- Continue reading key file
if crt then
f = io.open(keyfile, 'r')
key = assert(pkey.new(f:read('*all')))
f:close()
end
elseif ephemeral then
crt, key = updatecert(crtfile, keyfile)
end
-- Check loaded certificate
if not crt or not key then
error(string.format('failed to load certificate "%s" - %s', crtfile, err or 'error'))
end
end
-- Create TLS context and start listening
local s, err = server.listen {
host = host,
port = port,
tls = crt ~= nil,
ctx = crt and tlscontext(crt, key),
}
if not s then
error(string.format('failed to listen on %s#%d: %s', host, port, err))
end
-- Compose server handler
local routes = route(endpoints)
cq:wrap(function ()
assert(s:run(cb))
assert(s:run(routes))
s:close()
end)
table.insert(M.servers, s)
-- Create certificate renewal timer if ephemeral
if crt and ephemeral then
local _, expiry = crt:getLifetime()
expiry = math.max(0, expiry - (os.time() - 3 * 24 * 3600))
event.after(expiry, function (ev)
print('[http] refreshed ephemeral certificate')
crt, key = updatecert(crtfile, keyfile)
s.ctx = tlscontext(crt, key)
end)
end
end
-- @function Cleanup module
......@@ -156,25 +319,28 @@ function M.deinit()
if M.ev then event.cancel(M.ev) end
M.servers = {}
end
--
-- @function Configure module
local ffi = require('ffi')
function M.config(conf)
assert(type(conf) == 'table', 'config { host = "...", port = 443, cert = "..." }')
conf = conf or {}
assert(type(conf) == 'table', 'config { host = "...", port = 443, cert = "...", key = "..." }')
-- Configure web interface for resolver
if not conf.port then conf.port = conf.cert and 80 or 443 end
if not conf.host then conf.host = 'localhost' end
M:listen(conf.host, conf.port, M.route(M.endpoints))
if not conf.port then conf.port = 8053 end
if not conf.host then conf.host = 'localhost' end
M.interface(conf.host, conf.port, M.endpoints, conf.cert, conf.key)
-- TODO: configure DNS/HTTP(s) interface
-- M:listen(conf.dns.host, conf.dns/port, serve_web)
if M.ev then return end
-- Schedule both I/O activity notification and timeouts
local poll_step
poll_step = function (ev, status, events)
local ok, err, _, co = cq:step(0)
if not ok then print('[http] '..err, debug.traceback(co)) end
if not ok then print('[http]', err, debug.traceback(co)) end
-- Reschedule timeout or create new one
local timeout = cq:timeout()
if timeout then
-- Throttle web requests
if timeout == 0 then timeout = 0.001 end
-- Convert from seconds to duration
timeout = timeout * sec
if not M.timeout then
......
// Colour palette
var colours = [
'rgb(198,219,239)',
'rgb(158,202,225)',
'rgb(107,174,214)',
'rgb(66,146,198)',
'rgb(33,113,181)',
'rgb(8,81,156)',
'rgb(8,48,107)',
];
// Unit conversion
function tounit(d) {
d = parseInt(d);
......@@ -25,6 +35,27 @@ window.onload = function() {
},
data: statsHistory
});
var fills = { defaultFill: '#F5F5F5' };
for (var i in colours) {
fills['q' + i] = colours[i];
}
var map = new Datamap({
element: document.getElementById('map'),
fills: fills,
data: {},
height: 350,
geographyConfig: {
highlightOnHover: false,
borderColor: '#ccc',
borderWidth: 0.5,
popupTemplate: function(geo, data) {
return ['<div class="hoverinfo">',
'<strong>', geo.properties.name, '</strong>',
'<br>Queries: <strong>', data ? data.queries : '0', '</strong>',
'</div>'].join('');
}
}
});
/*
* Realtime updates over WebSockets
......@@ -52,7 +83,7 @@ window.onload = function() {
}
/* WebSocket endpoints */
var wsStats = 'ws://' + location.host + '/stats';
var wsStats = (secure ? 'wss://' : 'ws://') + location.host + '/stats';
var Socket = "MozWebSocket" in window ? MozWebSocket : WebSocket;
var ws = new Socket(wsStats);
ws.onmessage = function(evt) {
......
<!DOCTYPE html>
<title>{{.Title}}</title>
<meta charset="utf-8">
<title>{{ title }}</title>
<style>
body { font-family: 'Gill Sans', 'Gill Sans MT', Verdana, sans-serif; color: #555; }
h1, h2, h3 { line-height: 1.5em; color: #000; text-align: center; border-bottom: 1px solid #ccc; }
......@@ -13,24 +14,27 @@
#stats .layer-100ms , .l-100ms { fill: #258FDA; color: #258FDA; }
#stats .layer-1000ms , .l-1000ms { fill: #51A5E1; color: #51A5E1; }
#stats .layer-slow , .l-slow { fill: #E1AC51; color: #E1AC51; }
#feed { width: 100%; }
#feed .secure { color: #74c476; }
.stats-legend { text-align: center; }
.stats-legend li { display: inline; list-style-type: none; padding-right: 20px; }
.map-legend { font-size: 10px; }
</style>
<script type="text/javascript">
var host = "{{ host }}";
var secure = {{ secure }};
</script>
<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="d3.js"></script>
<script type="text/javascript" src="epoch.js"></script>
<script type="text/javascript" src="topojson.js"></script>
<script type="text/javascript" src="datamaps.world.min.js"></script>
<script