Using OpenBSD relayd(8) as an Application Layer Gateway

2023-10-10 2935 words, 14 minutes
computer
openbsd
relayd
reverse-proxy

I was lucky enough to attend to EuroBSDCon 2023 and offered the opportunity to talk about one of my favorite OpenBSD stock daemon: relayd(8).

The talk was recorded and made available on the EuroBSDCon YouTube channel. . One may check the EuroBSDCon 2023 program for more material.

This post attempts a reboot of the slides content in a more browser-friendly format.

What is relayd(8)?

Multi-purpose daemon available on OpenBSD since 4.3*:
- load-balancer.
- application layer gateway.
- transparent proxy.
Capable of monitoring groups of hosts for high-availability.
Operates as:
- Layer 3 redirection via communication with pf(4).
- Layer 7 relaying with application level filtering via itself.

* relayd was known as hoststated in OpenBSD 4.1.

How to manage relayd(8)?

The man pages are a must read before proceeding further.

# man relayd
# man relayd.conf
# man relayctl

The configuration file is expected in the standard etc directory. An example is available if you need more inspiration.

# more /etc/examples/relayd.conf
# vi /etc/relayd.conf

One can check for configuration errors using:

# relayd -dvn

The service is enabled and started using the standard rcctl utility:

# rcctl enable relayd
# rcctl start relayd
# rcctl stop relayd

A dedicated command can be used to get more information about relayd(8) state and apply specific actions:

# relayctl 𝘤𝘰𝘮𝘮𝘢𝘯𝘥 [𝘢𝘳𝘨𝘶𝘮𝘦𝘯𝘵 ...]

Terminology

Macros: user-defined variables that can be used later on.
Tables: host or a group of hosts defining traffic targets.
Protocols: settings and filter rules for relays.
Relays: layer 7 proxying instances.

Simplest HTTP relay

This is the simplest HTTP Reverse proxy configuration that you can get:

http protocol www {
    pass
}

relay www {
    listen on 203.0.113.1 port 80
    protocol www
    forward to 192.0.2.10 port 80
}

The first section defines an HTTP PROTOCOL object. Name has been set to ‘www’ but it can be anything.

The RELAY section defines a listening address and port. It links the relay to the previously configured protocol. It defines the backend server that will receive the HTTP requests.

Better simple HTTP relay

This example expands the previous HTTP Reverse proxy configuration with usage of reusable variables (MACROS) and logging of state changes and remote connections.

# Macros -----------------------------------
ext_addr="203.0.113.1"
webhost1="192.0.2.10"

# Global configuration ---------------------
log state changes
log connection

# Tables -----------------------------------
table <webhosts> { $webhost1 }

# Protocols & Relays -----------------------
http protocol www {
    pass
}

relay www {
    listen on $ext_addr port 80
    protocol www

    forward to <webhosts> port 80
}

Encrypt HTTP relay using Transport Layer Security (TLS)

Previous examples are using plain text HTTP. Switching to HTTPS provides secure communication and data transfer between the client and the website.

You’ll need to acquire a TLS certificate. That steps is beyong the scope of this post. But have a look at acme-client(1) and httpd(8) manpages. Those will guide you through the process of getting an HTTPS certificate.

Once acquired, install the certificate under /etc/ssl. If you used acme-client(1), you should get files such as:

/etc/ssl/private/relayd.example.key
/etc/ssl/relayd.example.crt

Reference the certificate name in the protocol section. Then replace the listen directive of the relay section to specify the usage of tls.

# Macros -----------------------------------
ext_addr="203.0.113.1"
webhost1="192.0.2.10"

# Global configuration ---------------------
log state changes
log connection

# Tables -----------------------------------
table <webhosts> { $webhost1 }

# Protocols & Relays -----------------------
http protocol wwwtls {
    tls keypair relayd.example
}

relay wwwtls {
    listen on $ext_addr port 443 tls
    protocol wwwtls

    forward to <webhosts> port 80
}

Renaming the protocol and relay names is not mandatory. I only did it to make it clear what I’m doing.

Load balancing & Failover

relayd(8) allows to distribute incoming requests to several backend servers. Depending on its configuration, you can balance the load on those servers and/or keep the service up and running since you only encounted n-1 failure(s).

ext_addr="203.0.113.1"
whost1="192.0.2.11"
whost2="192.0.2.12"
whost3="192.0.2.13"
interval 5
table <webhosts> { $whost1, $whost2, $whost3 }

http protocol wwwtls {
   tls keypair relayd.example
}

relay wwwtls {
    listen on $ext_addr port 443 tls
    protocol wwwtls
    # l/b using source-IP, check HTTP return code
    forward to <webhosts> port 80  \
      mode loadbalance             \
      check "/health-check" code 200
}

In this example, a TABLE has been created that references all the backend servers - those that will receive the HTTP requests.

There are many scheduling algorithms (aka MODE) available. Check the man page for more details. The default is using roundrobin and no health checks. Here, we’re using the loadbalance algorythm and return code check.

Fallback server(s) - automatic switch

There are cases when you want to implement automatic reaction on server(s) outage events. You may want to switch the whole service to a secondary server pool. You may display an incident status page rather that an HTTP/500 error page. You should probably display a static “be back soon” page while performing maintenance.

This is what the fallback feature can be used for.

ext_addr="203.0.113.1"
whost1="192.0.2.11"
whost2="192.0.2.12"
whost3="192.0.2.13"
interval 5
table <webhosts> { $whost1, $whost2 }
table <fallback> { $whost3 }

http protocol wwwtls {
   tls keypair relayd.example
}

relay wwwtls {
  listen on $ext_addr port 443 tls
  protocol wwwtls

  # l/b using round-robin, check HTTP return code
  forward to <webhosts> port 80 mode roundrobin \
    check http "/" code 200
  # switch service if all previous checks fail
  forward to <fallback> port 80
}

Two TABLEs have been defined. One for the primary server(s). One for the fallback server(s). Then, everything happens in the relay section. The first forward directive load-balances the HTTP requests to the primary servers pool. The second forward directive acts as the fallback target. It will be triggered as soon as no servers from the primary pool are known to be working.

Fallback server(s) - manual switch

In a use-case where you prefer managed operations on server(s) outage, you may configure a non-automatic switch. This mostly apply to Business Continuity Plan where the secondary servers pool is remote or mutualized or resources limited etc.

ext_addr="203.0.113.1"
whost1="192.0.2.11"
whost2="192.0.2.12"
whost3="192.0.2.13"
whost4="192.0.2.14"
interval 5
table <webhosts>         { $whost1, $whost2 }
table <fallback> disable { $whost3, $whost4 }

http protocol wwwtls {
   tls keypair relayd.example
}

relay wwwtls {
  listen on $ext_addr port 443 tls
  protocol wwwtls

  # l/b using source-IP, check HTTP return code
  forward to <webhosts> port 80 mode loadbalance \
    check http "/" code 200
  # l/b using round-robin, check HTTP return code
  forward to <fallback> port 80 mode roundrobin \
    check http "/" code 200
}

The main difference with the previous configuration is the disable property of the fallback table. This implies that it won’t be used by relayd(8) unless being told to.

Initial expected state

Usage of the relayctl command confirms that the primary servers pool is working as expected.

# relayctl show summary
Id      Type            Name                            Avlblty Status
1       relay           wwwtls                                  active
1       table           webhosts:80                             active (2 hosts)
1       host            192.0.2.11                      100.00% up
2       host            192.0.2.12                      100.00% up
2       table           fallback:80                             disabled

Primary hosts are up and running.
Secondary hosts are disabled.

The service is UP.

Primary servers go down

On a clear service breakdown, relayctl will indicate that the primary hosts are down.

# relayctl show summary
Id      Type            Name                            Avlblty Status
1       relay           wwwtls                                  active
1       table           webhosts:80                             empty
1       host            192.0.2.11                      95.56%  down
2       host            192.0.2.12                      95.56%  down
2       table           fallback:80                             disabled

Primary hosts are down.
Secondary hosts are disabled.

The service is DOWN. Nothing happens as relayd(8) was told to start the fallback table disabled.

Switch to the secondary servers pool

relayctl is used to enable the disabled fallback. This happens without the need of restarting relayd.

# relayctl table enable 2
command succeeded

# relayctl show summary
Id      Type            Name                            Avlblty Status
1       relay           wwwtls                                  active
1       table           webhosts:80                             empty
1       host            192.0.2.11                      76.79%  down
2       host            192.0.2.12                      76.79%  down
2       table           fallback:80                             active (2 hosts)
3       host            192.0.2.13                      100.00% up
4       host            192.0.2.14                      100.00% up

Primary hosts are down.
Secondary hosts are enabled.

The service is UP.

Note that failback shall happen as soon as relayd detects a Primary host up. If this is not something you want to happen, use relayctl table disable 1 to prevent such an automatic failback.

Relaying multiple FQDNs

What if you want to expose multiple hostnames using a single IP?
You can do Apache Virtual Hosts or nginx server blocks. Or you can:

(...)
table <blog>  { $whost1, $whost2 }
table <cloud> { $whost3 }

http protocol wwwtls {
  tls keypair blog.example
  tls keypair nextcloud.example

  block
  pass request header "Host" value "blog.example"  \
    forward to <blog>
  pass request header "Host" value "cloud.example" \
    forward to <cloud>
}

relay wwwtls {
  listen on $ext_addr port 443 tls
  protocol wwwtls
  forward to <blog>  port 80 mode roundrobin \
    check http "/" code 200
  forward to <cloud> port 80
}

This configuration example checks every HTTP requests’ header and route them to the proper backend server depending on the value of the “Host” header. The backend servers are referenced in tables.

Using multiple “keypair” directives to reference the HTTPS certificates enables the TLS Server Name Indication (SNI) feature of relayd(8).

Relaying multiple pathnames

Apache has location directives. nginx has location blocks. To design reaction rules (allow, deny, forward…) depending on URL paths, relayd(8) can use FILTER RULES based on the “path” keyword.

(...)
table <blog>  { $whost1, $whost2 }
table <cloud> { $whost3 }

http protocol wwwtls {
  tls keypair relayd.example

  block quick path "/cgi-bin*"
  block quick path "/wp-admin*"
  pass  quick path "/nextcloud/*" forward to <cloud>
  pass  request                   forward to <blog>
}

relay wwwtls {
  listen on $ext_addr port 443 tls
  protocol wwwtls

  forward to <blog> port 80 mode roundrobin \
    check http "/" code 200
  forward to <cloud> port 80
}

This configuration blocks any attempt to access paths that look like /cgi-bin and /wp-admin. It also routes any URL matching https://relayd.example/nextcloud/ to the “cloud” servers pool. Any other URL will be routed to the “blog” table.

Solving problems with HTTP headers

relayd(8) can add, remove or modify HTTP header on the fly. This allows solving various kinds of issues with exposed Web services.

Unencrypted connection

There are software like Baikal, Mastodon or SearxNG that refuse to serve unencrypted content. If you still want to run them using plain text HTTP on the backend server and feel confident about using relayd(8) as an SSL terminator, you shall add an HTTP header to the requests reaching the backend HTTP server.

(...)
http protocol wwwtls {
  tls keypair blog.example
  tls keypair nextcloud.example

  block
  pass request header "Host" value "blog.example"  forward to <blog>
  pass request header "Host" value "cloud.example" forward to <cloud>

  match request header set "X-Forwarded-Proto" value "https"
}
(...)

In this particular case, the X-Forwarded-Proto is set to “https” and passed to the backend server to confirm that the communication is secured using TLS.

Leaking headers

From time to time, you discover that Web services fill their HTTP replies with too many information. Too many meaning not mandatory to provide a functionnal user experience while still leaking sensible information to the external world.

(...)
http protocol wwwtls {
  tls keypair relayd.example

  pass  quick path "/nextcloud/*" forward to <cloud>
  pass  request                   forward to <blog>

  match response header remove "X-Powered-By"
  match response header set "Server" value "Microsoft-IIS/8.5"
}
(...)

This configuration removes any X-Powered-By information from every HTTP replies. It also sets the Server HTTP header to some specific value that can be used to fool script kiddies ; or to deal with faulty Web clients that expects a specific value.

Improve users security and privacy

Some software don’t really bother about security and privacy. Sometimes, it’s just that they expect you to use an htaccess-compatible Web server to provide a couple of HTTP headers that can help protecting the user.

(...)
http protocol wwwtls {
  tls keypair relayd.example

  pass  quick path "/nextcloud/*" forward to <cloud>
  pass  request                   forward to <blog>

  match response header set "X-XSS-Protection"       value "1; mode=block"
  match response header set "X-Content-Type-Options" value "nosniff"
  match response header set "Permissions-Policy"     value "accelerometer=(),
ambient-light-sensor=(),autoplay=(),camera=(),encrypted-media=(),
focus-without-user-activation=(),geolocation=(),gyroscope=(),
magnetometer=(),microphone=(),midi=(),payment=(),picture-in-picture=(),
speaker=(),sync-xhr=(),usb=(),vr=()"
}
(...)

This example only shows a subset of HTTP headers that can be set to improve users privacy and protection. Check your Web application using tools like Mozilla Observatory to get a better overview of what should / could be achieved on your server.

Log management

The default relayd(8) log configuration doesn’t suit me well. Traces appear in several log files and are not detailed enough to my linkings when it comes to debugging issues.

One for log and log for one

To have relayd(8) log in its own dedicated log file, I’d rather tune syslogd(8):

# touch /var/log/relayd

# vi /etc/syslog.conf
!!relayd
*.* /var/log/relayd
!*
(...)

# vi /etc/newsyslog.conf
(...)
/var/log/relayd root:_relayd 640 7 * $D0 ZB

# rcctl restart syslogd

Logging options

Some matching FILTER RULES can be defined to have more information appear in the log files.

http protocol wwwtls {
  tls keypair relayd.example

  match url log
  match header log "Host"
  match header log "User-Agent"
  match response header log "Content-Type"
  match response header log "Content-Length"

  pass  quick path "/nextcloud/*" forward to <cloud>
  pass  request                   forward to <blog>
}

This particular example provides HTTP header information in the relayd(8) logs.

Sep 17 14:35:12 ebsdc relayd[34137]: relay wwwtls, session 1 (1 active), 0, 
    203.0.113.1 -> 127.0.0.1:80, done, 
    [blog.example/about] [Host: blog.example] [User-Agent: curl/8.2.0] 
    GET -> 127.0.0.1:80 {Content-Type: text/html} {Content-Length: 41};

Any HTTP header can be matched and rendered in the logs. Select yours.

Conditional filtering

Using TAGS and INCLUDES, relayd(8) can perform different computation and actions depending on wether or not conditions evaluate to true or false. Here’s an example of some slightly complex conditional filtering that can be designed in relayd(8).

Filtering branched on FQDN #(1,2)

In a dedicated configuration file, a TAG is set if the “Host” HTTP header of the requests matches one of the defined value. Then for each TAGGED connections, relayd(8) will apply additionnal logging and improve users security and privacy.

# cat /etc/relayd-ssg.conf

# Mark using hostnames
match request header "Host" value "www.example"     tag "ssg"
match request header "Host" value "blog.example"    tag "ssg"

# Apply additionnal logging
match header log "Host"       tagged "ssg"
match header log "User-Agent" tagged "ssg"
match url    log              tagged "ssg"

# Improve Security and Privacy
match response tagged "ssg" header set \
  "Strict-Transport-Security" value "max-age=31536000; includeSubDomains; preload"
match response tagged "ssg" header set \
  "X-XSS-Protection" value "1; mode=block"
match response tagged "ssg" header set \
  "X-Content-Type-Options" value "nosniff"

Filtering branched on FQDN #3

In a dedicated configuration file, a TAG is set if the “Host” HTTP header of the requests matches the defined value. Then for each TAGGED connections, a check is done on the “User-Agent” HTTP header and a block happens for the referenced values. Another check is done, based on the URL and the source IP, to ensure only trusted computer can acces an adminstrative URL. In the end, any acceptable HTTP session will have the “Server” HTTP header removed from the HTTP reply.

# cat /etc/relayd-nextcloud.conf

# Mark using hostname
match request header "Host" value "cloud.example"   tag "nextcloud"

# Block User Agents
block request quick tagged "nextcloud" header "User-Agent" value "Googlebot/*"
block request quick tagged "nextcloud" header "User-Agent" value "YandexBot/*"

# Only allow "admin" path from specific subnet
match request                   url "cloud.example/admin/" tag "forbidden"
match request from 192.0.2.0/24 url "cloud.example/admin/" tag "nextcloud"

# Don't let version leak via HTTP header
match response tagged "nextcloud" header remove "Server"

Filtering branched on FQDN #4

In a dedicated configuration file, a TAG is set if some trusted computer try to access an allowed FQDN. Any TAGGED connections will have its path checked for some regex and tagged if matched. Those connections will have an HTTP header set in their reply.

# cat /etc/relayd-grafana.conf

# Mark using client source IP and path
match request from 192.0.2.0/24    url "metrics.example/" tag "grafana"
match request from 198.51.100.8/32 url "metrics.example/" tag "grafana"

# Overwrite caching
match request tagged "grafana" path "*.css" tag "g-cache"
match request tagged "grafana" path "*.js"  tag "g-cache"
match request tagged "grafana" path "*.png" tag "g-cache"

match response tagged "g-cache" header set "Cache-Control" value "max-age=86400"

Connecting the branches

In the main relayd(8) configuration file, the dedicated config files are included so that tagging happens, or not. The routing to the proper backend servers will happen for every TAGGED connections. The block directive will drop any connection that has not been matched by the filtering definition.

# cat /etc/relayd.conf

table <blog>    { $whost1, $whost2 }
table <cloud>   { $whost3 }
table <grafana> { $whost4 }

http protocol wwwtls {
  tls keypair www.example
  tls keypair cloud.example
  tls keypair metrics.example

  block

  include "/etc/relayd-ssg.conf"
  include "/etc/relayd-nextcloud.conf"
  include "/etc/relayd-grafana.conf"

  pass request tagged "ssg"       forward to <blog>
  pass request tagged "nextcloud" forward to <cloud>
  pass request tagged "grafana"   forward to <grafana>
  pass request tagged "g-cache"   forward to <grafana>
}

One more thing

Remarks and answers given during the live session:

Beware that pf(4) and relayd(8) TABLES are not the same things although they share the same name.
PROTOCOL and RELAY names can be any string you like. I had some weird behaviour when using underscores or hyphens. I don’t remember which exactly. So I just avoid them. They just have to match with one another.
I have no load metrics. I use relayd(8) on personal projects and have not faced any specific issues. But if you expect lots of connections, you may need to increase the prefork value and/or tune the tcp options.
In real Enterprise World, you may combine relayd(8) and carp(4) to achieve a better SLA.

The original slides are available here . Because OpenBSD and because fun, you can get the same slides rendered in Comic font .

Special thanks to florian@ and solene@ for having proofread the original slides.