Spaetzblog

Stuff that doesn't matter

Category: Howto

  • Matrix VOIP and Livekit

    Welcome to another technical post with my take on an area with lots of confusion: matrix VOIP and turn. First: Which VOIP standards does matrix support and second, what with that TURN server in livekit. Please skip if that is not of interest to you.

    Matrix VOIP

    Legacy calls: In the beginning, there was direct WebRTC between participants, potentially using a TURN server that was configured in the matrix synapse configuration. This is now called legacy calls and while some very obsolete clients are still supporting it (yes, Element classic), newer ones typically don’t (Element X and others). One of the problems with this style of calls is, that media needs to be streamed directly to all participants and received from all participants, which makes that n*(n-1) streams. This is OK for 1:1, but does not scale when more than a handfull people participates in a call.

    Jitsi: Some Element clients (mostly Element Web/Desktop) are able to embed web widgets. This was used to embed external jitsi servers, so that one could organize jitsi calls from within a matrix room. I don’t think other clients ever supported this. It worked, but always felt a bit like a bandaid.

    MatrixRTC: The new style calls as defined in MSC 4143 and define signalling and call membership over Matrix communication channels, but leave the media transport to pluggable backends. Currently, the only usable backend is Livekit, this is defined in MSC 4195. Alternative backends are planned, e.g. a direct WebRTC mesh (potentially using TURN servers). I refer to MSC 3401 for that. But for now, it is livekit or nothing. Because there is often confusion around that: MatrixRTC calls using livekit can both be group calls and “direct” 1:1 calls (DM).

    TURN and Livekit

    So…., one of the most frequent annoyances, is that people try hard to set up their Matrix voip, install livekit and all that (which is tricky enough). And then they either get told in a support room, or come up with the plan of their own, to get the livekit TURN working or to install a separate coturn TURN server.

    I’d just like to interject for a moment. NOOO, YOU MOST LIKELY DO NOT NEED TO GET THE INTERNAL LIVEKIT TURN SERVER RUNNING AND YOU MOST LIKELY DO NOT NEED TO SETUP A SEPERATE COTURN SERVER!!!

    In order to argue my case, let me explain briefly what a turn server is for, and then turn (HAHA) to livekit, the SFU (selective forwarding unit).

    If there were not firewalls, and NATs, everyone could reach everyone. VOIP would be easy and no TURN servers would be needed. But the fact is, there are many firewalls, preventing incoming connections to random ports. And NATs make participants unreachable from the outside. So a turn server is a bandaid, relaying media traffic in a way, that actors only have outgoing connections and no incoming connections when they cannot receive any. Traffic with a TURN server kind of looks like:

    Alice ---------> TURN relay <----------- Bob

    Now, livekit is a SFU, it receives, combines and distributes media streams from VOIP participants. It can do so over TCP and UDP connections. Traffic using the livekit SFU kind of looks like (simplified):

    Alice ---------> livekit SFU <----------- Bob

    Uhh, ohh, that looks pretty similar, right? Right! The livekit SFU effectively performs like a TURN server, and can take over that functionality. So, in normal cases, there is NO turn needed. So what is the internal livekit turn server for (or an alternative coturn server that one already had set up)?

    There are very restricted network environments (public hotspots, corporate firewalls, censoring governments) where not only incoming connections are hindered, but also outgoing connections might be prevented. The internal livekit TURN server (at least the TCP part) ONLY EVER announces port 443 via TLS (src code) instead of whatever port you might have configured as turn: TLSPort:, so it is for use cases where you are only allowed to reach other HTTPS web sites (trying to hide your media traffic as regular website https traffic). It helps in no other case! And the UDP part of the turn server is pretty much useless, as the SFU is already able to use a whole bunch of UDP ports to do its turnish thing anyway.

    As a result: I have disabled the livekit internal TURN server and not set up another coturn server and am doing my matrixRTC voip calls happily every after.

    Fediverse Reactions

  • How to resolve stuck MatrixRTC calls

    This applies only to new MatrixRTC (aka element-call) calls, not to jitsi-widget-based calls and not to “old-style” direct 1:1 calls.

    The problem

    You’ve got an ongoing element-call, despite no one really being in it. It is stuck and you cannot end it. In Element-Web’s room list this looks like this:

    Resolution

    Step 1: Open devtools

    In Element-Web, open devtools by typing /devtools in the room’s chat text box and hitting <ENTER> twice:

    You should now see the rooms devtools dialog:

    ⏵Click on Explore Room State.

    Step 2: Check Room states

    Below, you can see the room state. In this case, there is a state for org.matrix.msc3401.call and a state for org.matrix.msc3401.call.member, these relate to element-call.

    As long as there are active members in a call, your client will deduce that there is an ongoing call. So you “just” need to remove the members. Slight complication: for technical reasons, you can never remove a state, you can only update it. Therefore we need to update the call members to have an empty state.
    ⏵Click on the button org.matrix.msc3401.call.member.

    Step 3a: Remove members

    You can now see everyone who ever participated in a call (not only those who are currently active). You see that a few alter egos of me have been talking to each other in this room (should I be concerned?)

    ⏵Click on each member entry, the dialog will be looking something like this:

    The above case is a joined (=active) member in a call, as the "content": {} section is not empty. For some members, the content might already be empty ({}). These are not part of the call anymore and we can ignore these entries.

    If the "content": {} section is non-empty (there is stuff within these brackets), click Edit and empty it:

    ⏵Hit Send and go Back. Rinse and repeat for all active members. If none is left, the call should automatically stop showing up as being active in Element-Web.

    Step 3b (optional?): Remove the “org.matrix.msc3401.call”

    From the Room state explorer, open up the state org.matrix.msc3401.call, mine looked something like this:

    and Edit it, setting the content to empty, that is just empty bracket: {}. It should look something like this:

    ⏵Hit Send. I am not sure that Step 3b is actually needed, but it certainly did not hurt.

    Additional Recommendations

    Delayed events

    Usually calls should be properly terminated when people leave it, but in some cases (closing browser tab, connection disrupted,…) things might get stuck. To solve this, “delayed events” have been introduced in synapse, which provide kind of a dead man’s switch:

    If you don’t update the delayed event regularly, your presence from the call will automatically be removed after a while. Delayed events are not enabled by default in synapse yet, you need to do so manually by setting:

    # enable delayed events, so we quit calls that are interrupted somehow
max_event_delay_duration: 24h

    in your homeserver.yaml. There is no downside to enabling them, so I recommend that everyone does.

    Please send corrections and improvement suggestions to me. I usually hang out in the “Element web” and “Element-X-Android” matrix rooms. But I refuse unsolicited DM requests, so please ping me in one of the rooms.

  • MatrixRTC aka Element-call setup (Geek warning)

    This is another geeky and technical post detailing how to get Matrix VOIP running as a self-hosted service. This is about setting up the SFU or Selective Forwarding Unit, that is the VOIP backend driving your self-hosted matrixRTC instance. Some time after publication of this post, element improved its own documentation for self-hosting which you should also check out here. For any MatrixRTC call these are the components involved:

    1. A matrix homeserver where you have control over the .well-known/matrix/client file. Its installation is not described here.
    2. An instance of the element-call widget (the frontend of element-call). Nowadays, this is bundled with the Element-web, Element X mobile clients and gomuks. (possibly other clients too)
    3. lk-jwt-service: A small go binary providing auth tokens. Element calls this MatrixRTC Authorization Service nowadays.
    4. Livekit: The SFU or Selective Forwarding Unit. It takes a video/audio stream and multiplexes it to those users who need it. This way, users in a video conference do not need to send media to ALL other participants (which does NOT scale).
    5. A few rules poking holes into your firewall to make livekit work.
    6. Nginx proxying a few routes from livekit and lk-lwt-service to https://livekit.sspaeth.de.
    7. The livekit-provided TURN server. You will most likely not need this, and there are ways in which you can use existing TURN servers (such as coturn).

    Details

    My main domain is sspaeth.de, the homeserver lives on chat.sspaeth.de, and everything livekit-related lives on rtc.sspaeth.de. I install this on a Debian server using a mix of local services and docker images. OK. Let’s start. It is also recommended to have a look at the official element-call self-hosting docs.

    1. Homeserver configuration

    You need to enable this in your config:

    experimental_features:
  #room previews
  msc3266_enabled: true  # MSC4222 needed for syncv2 state_after. This allow clients to
  # correctly track the state of the room.
  msc4222_enabled: true

# enable delayed events, so we quit calls that are interrupted somehow
max_event_delay_duration: 24h

    Also, if your homeserver is NOT federated, you will need to enable an openid listener on your synapse.

    2. Define the Element Call widget (see update at end of section)

    UPDATE MAR 2025: The frontend now bundled with the Element-Web, Element X mobile clients and gomuks, so you would not need an externally hosted widget frontend with these clients and can ignore this entire section.

    3. Telling the participants which SFU to use

    Second, we need to tell the clients which SFU to use. This is done by adding

    "org.matrix.msc4143.rtc_foci":[
 {"type": "livekit",
  "livekit_service_url": "https://livekit.sspaeth.de"}]

    to https://sspaeth.de/.well-known/matrix/client. Strictly speaking, this will query the lk-jwt-service at https://livekit.sspaeth.de/sfu/get, which will then return the livekit SFU instance together with an JWT SFU access token. Note: the URL to get a token is about to be changed to https://livekit.sspaeth.de/get_token with element call 0.17 (?).

    4. lk-jwt-service / MatrixRTC Authorization Service

    It generates JWT tokens with a given identity for a given room, so that users can use them to authenticate against LiveKit SFU. I dabbled with creating a docker image for this, but that was too complicated for me. Given it is a single binary, I just compiled it locally and just run it on the Debian server (I actually just download the precompiled binary from the releases page). This is how to compile it:

    a) Checkout the git repo at https://github.com/element-hq/lk-jwt-service, I put it into /opt/lk-jwt-service.
    b) With a go compiler installed, compile it: /usr/lib/go-1.24/bin/go build -o lk-jwt-service (use just “go” instead of the full path if go it is in your PATH). If this suceeds, you’ll end up with the binary lk-jwt-service as a result. If you execute it from a shell, it will output: LIVEKIT_KEY[_FILE], LIVEKIT_SECRET[_FILE] and LIVEKIT_URL environment variables must be set and exit.

    Next, I create a systemd service to start it automatically: Note the values from LIVEKIT_SECRET and LIVEKIT_KEY which need to be taken from the livekit configuration.

    [Unit]
Description=LiveKit JWT Service
After=network.target
[Service]
Restart=always
User=www-data
Group=www-data
WorkingDirectory=/opt/lk-jwt-service
Environment="LIVEKIT_URL=wss://livekit.sspaeth.de"
Environment="LIVEKIT_SECRET=this_is_a_secret_from_the_livekit.yaml"
Environment="LIVEKIT_KEY=this_is_a_key_from_the_livekit.yaml"
Environment="LIVEKIT_JWT_BIND=localhost:8081"
Environment="LIVEKIT_FULL_ACCESS_HOMESERVERS=sspaeth.de"
ExecStart=/opt/lk-jwt-service/lk-jwt-service
[Install]
WantedBy=multi-user.target

    P.S. Yes, it would be more elegant to include those environment variables in a separate file than hardcoding them in the service file.
    P.P.S. Note, that I am running this on port 8081 instead of the default 8080 because 8080 is already in use on this box.

    Last but not least, you need to proxy the output of three routes (/sfu/get, /get_token and /healthz) on https://livekit.sspaeth.de. The nginx rules are in the nginx section.

    If you enable and start the lk-jwt-service via systemctl, you should be able to open https://livekit.sspaeth.de/healthz in a web browser and get an empty webpage (aka HTTP STATUS 200). For testing purposes you can also start the service manually and observe its output by executing (in 1 line!): LIVEKIT_URL=wss://livekit.sspaeth.de LIVEKIT_SECRET=this_is_a_secret_from_the_livekit.yaml LIVEKIT_KEY=this_is_a_key_from_the_livekit.yaml LIVEKIT_BIND=localhost:8081 LIVEKIT_FULL_ACCESS_HOMESERVERS=sspaeth.de /opt/lk-jwt-service/lk-jwt-service

    5. livekit

    I installed this one via precompiled docker and did not do the “Just execute this shell script as root and trust us”™.

    I generated in initial configuration file using the livekit/generate imageand pruned most of the resulting stuff, as I do not run caddy for instance, and I do not want these scripts to be installing packages and stuff on my main host.

    5.1 Generating the initial configuration

    docker pull livekit/generate
    docker run --rm -it -v$PWD:/output livekit/generate

    The above creates a folder with the name of domain you provided, containing the following files: caddy.yaml, docker-compose.yaml, livekit.yaml, redis.conf, init_script.sh.

    I discarded most of the above (but built on the resulting livekit.yaml and docker-compose.yaml). I found particularly useful that it creates an API key and secret in livekit.yaml.

    5.2 Final configuration and setup

    This is my docker-compose.yaml file that I ended up using in order to built my livekit image:

    services:
  livekit-docker:
    image: livekit/livekit-server:latest
    command: --config /etc/livekit.yaml
    restart: unless-stopped
    network_mode: "host"
    volumes:
      - ./livekit.yaml:/etc/livekit.yaml
      - /run/redis/redis-server.sock:/run/redis.sock

    Running docker-compose up --no-start resulted in this output

    Creating livekitsspaethde_livekit-docker_1 … done

    This is my /etc/systemd/system/livekit.service file to get livekit started:

    [Unit]
Description=LiveKit Server Container
After=docker.service
Requires=docker.service
After=network.target
Documentation=https://docs.livekit.io

[Service]
LimitNOFILE=500000
Restart=on-failure
WorkingDirectory=/etc/livekit
# start -a attaches STDOUT/STDERR so we get log output and prevents forking
ExecStart=docker start -a livekitsspaethde_livekit-docker_1
ExecStop=docker stop livekitsspaethde_livekit-docker_1

[Install]
WantedBy=multi-user.target

    This is my final livekit.yaml config file

    port: 7880
bind_addresses:
    - ""
rtc:
    tcp_port: 7881
    port_range_start: 51000
    port_range_end: 52000
    use_external_ip: true
    enable_loopback_candidate: false
turn:
    # in most cases you really don't need that turn server
    enabled: false
    domain: livekit.sspaeth.de
    # without a load balancer this is supposed to be port 443, and I am not using this, as my port 443 is occupied.
    tls_port: 443
    udp_port: 443
    external_tls: true
keys:
    # KEY: secret were autogenerated by livekit/generate
    # in the lk-jwt-service environment variables
    APIXCVSDFldksef: DLKlkddfgkjldhjkndfjkldfgkkldkdflkfdglk

    5.3 Firewall rules

    I allow inbound ports=7881/tcp and 51000:52000/udp and these ports never go through the nginx proxy.

    6. nginx configuration

    I am not sure if all of those proxy headers actually need to be set, but they don’t hurt. These

    proxy_set_header Connection “upgrade”;
    proxy_set_header Upgrade $http_upgrade;

    are actually important though (they enable the switch from https to websocket)!

    server {
    access_log /var/log/nginx/livekit.sspaeth.de.log;
    error_log /var/log/nginx/livekit.sspaeth.de.error;
    listen 123.123.123.123:443 ssl;
    listen [dead:beef:dead:beef:dead:beef:dead:beef]:443 ssl;
    ssl_certificate XXXX;
    ssl_certificate_key XXXX;
    server_name livekit.sspaeth.de;

    # This is lk-jwt-service
    location ~ ^(/sfu/get|get_token|/healthz) {
        proxy_pass http://[::1]:8081;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
    #and this is livekit
    location / {
       proxy_pass http://localhost:7880;
       proxy_set_header Connection "upgrade";
       proxy_set_header Upgrade $http_upgrade;
       #add_header Access-Control-Allow-Origin "*" always;

       proxy_set_header Host $host;
       proxy_set_header X-Forwarded-Server $host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
    }
}

    Bonus section: testing your setup

    The testmatrix tool (disclaimer: written by me) is able to test if your setup seems sane or not. Install it via pip install testmatrix or grab a checkout of the code and run it locally. If you want to test the MatrixRTC functionality deeper, you will need to hand it a user name and an authorization token (personal token, access token, compatibility token) in order to authenticate you and receive some MatrixRTC credentials that you can then actually test with the below livekit connection tester.

    https://livekit.io/connection-test can be used to test your livekit setup. You “just need to catch a room token from the jwt service. I get one by starting a call in a room with the Firefox dev network console open, filtering for “/sfu/get” URLS and catch the following line: POST
    https://livekit.sspaeth.de/sfu/get    . The response to that is a JSON blob containing the jwt token or by using testmatrix.

    You can then go to https://livekit.io/connection-test, enter wss://livekit.sspaeth.de as livekit url and the long access token as room token. This will test, whether livekit is reachable, all ports are open, TURN is enabled and working etc…

  • TL;DR: Matrix & Nextcloud setup (geek warning)

    This is the BARE minimum of the specific configuration of my setup. Read the detailed setup post here.

    Synapse

    Synapse runs locally on port 8008 and is served on matrix.sspaeth.de. The only change needed, is to set up MAS as the OIDC server. I put the below in a separate OIDC.yaml file in the conf.d subdirectory of my synapse config:

    experimental_features:
  msc3861:
    enabled: true
    issuer: https://auth.sspaeth.de/
    # Matches the `client_id` in the MAS config
    client_id: 00000000000000000SYNAPSE00
    # Matches the `client_auth_method` in the MAS config
    client_auth_method: client_secret_basic
    # Matches the `client_secret` in the MAS config
    client_secret: 1234CLIENTSECRETHERE56789
    # Matches the `matrix.secret` in the MAS config
    admin_token: 0x97531ADMINTOKENHERE13579

    MAS

    MAS runs locally on port 8080 and is served on auth.sspaeth.de. Below are snippets that I either added to or where I modified the defaults in config.yaml.

    # Set up the client talking to Synapse.
clients:
  - client_id: 00000000000000000SYNAPSE00
    client_auth_method: client_secret_basic
    client_secret: 1234CLIENTSECRETHERE56789
    redirect_uris:
      - https://openidconnect.net/callback
# Tell MAS about Synapse and how to talk locally to it:
matrix:
  homeserver: sspaeth.de
  secret: 0x97531ADMINTOKENHERE13579
  endpoint: http://localhost:8008/

    This should be OK for local password auth through MAS, but the next section is what I added to configure Nextcloud as an upstream OIDC provider:

    upstream_oauth2:
  providers:
  - id: 01B2BSNY1QVVS9ZG3JTVDHNYYE
    # Note, above value is used in the Nextcloud config in the Redirection URI
    human_name: Nextcloud
    issuer: "https://cloud.sspaeth.de"
    client_id: "THISISMYLONGANDSECRETNEXTCLOUDCLIENTID" # needs to be configured in the Nextcloud OIDC settings
    client_secret: "THISISMYLONGANDSECRETNEXTCLOUDCLIENSECRET" # needs to be configured in the Nextcloud OIDC settings
    token_endpoint_auth_method: "client_secret_post"
    scope: "openid profile email"
    claims_imports:
        localpart:
          action: require
          template: "{{ user.preferred_username }}"
        displayname:
          action: suggest
          template: "{{ user.name }}"
        email:
          action: suggest
          template: "{{ user.email }}"
          set_email_verification: import
# Only allow upstream IdP, no local passwords
passwords:
  enabled: false

    Nextcloud

    The OpenID Connect app/plugin adds the central config option OpenID Connect clients. In this I have set:

    Name 	auth.sspaeth.de
# Redirect URI uses the upstream_oauth2->providers->id value from my MAS config
Redirection URI: https://auth.sspaeth.de/upstream/callback/01B2BSNY1QVVS9ZG3JTVDHNYYE
Client Identifier: THISISMYLONGANDSECRETNEXTCLOUDCLIENTID
Secret: THISISMYLONGANDSECRETNEXTCLOUDCLIENTSECRET
Signing Algorithm: RS256
Type: confidential
Flows: Code & Implicit Authorization Flow
Limited to Groups: matrix # my choice to only allow specific users to log in.

    That was simple wasn’t it? Again, this is just specific config changes to all components to get things running. Check out the detailed post on my setup and nginx proxy configuration.

    Fediverse Reactions

  • Matrix server with Nextcloud login

    This is a pretty geeky description of my matrix homeserver setup using Matrix-authentication-service and Nextcloud as the authentication source. It might be useful to others or future me. If not interested in technical details, please skip this post. A minimal version highlighting just the necessary configuration can be found in this TL;DR post. This is the slightly extended version explaining my setup.

    (more…)