<!-- MIT License. Copyright (c) 2026 Decentralized Science Labs LLC. -->
<!-- summary: Symptom, cause and fix for real problems on a COLLAB server: DNS, certificates, nginx, publishing, sign-in, the prerender job, RPCs, the policy, upgrades. -->
# Troubleshooting

This page lists the problems an operator of a COLLAB server is likely to meet, each as what you see,
why it happens and what to do. The cases come from the installer's and the admin service's own
messages, the release rehearsal and the reviews of the server kit. When nothing here fits, start
with `sudo collab status` and `sudo collab logs` ([Status, logs and the collab command](monitoring.md)).

The installer can always be run again: it repairs what is missing and keeps what is there. Its
refusals happen before it changes anything, and say "Nothing was changed".

## Installing

### A name is left without a certificate

**What you see.** In the installer's "DNS and certificate" step, a line such as
`  dao.descifi.org: no DNS record (or it does not resolve from here)` or
`  dao.descifi.org: points at 192.0.2.10, not at this server`, and in the summary
"Not covered:   dao.descifi.org: point their DNS at this server, then run this again (the same
command)." Status names it in the notice "One name has no certificate."

**Why.** The certificate covers only the names whose DNS points at the server when the installer
runs. The record is missing, points elsewhere, or has not spread yet. Behind NAT, where the public
address is not on the server itself, the installer fetches a one-time file through the name from
the server; a router that does not send that request back in fails the check even when the record
is right, and the line names the server's own public address.

**What to do.** Fix the A (and AAAA) record, check it with `dig +short dao.descifi.org`, then run the
same command again: the certificate is extended to the name. Behind NAT with a right record, turn
on the router's NAT loopback (hairpinning) first. Until then, the name still redirects over HTTP,
but not over HTTPS. See [Before you install](before-you-install.md#dns-point-every-name-first).

### The site answers "COLLAB is being set up" (503)

**What you see.** `http://` answers "COLLAB is being set up on this server, HTTPS first. Please come
back in a few minutes." The installer warned "collab.genesisl1.com does not point at this server
yet, so it cannot have a certificate." or "certbot could not get the certificate (above).", and its
summary says "COLLAB 2.2.0 is installed, but not on HTTPS yet: see the warnings above."

**Why.** The main host has no certificate yet, so nginx serves only Let's Encrypt's challenges and
this notice on port 80.

**What to do.** Point the main host's DNS at the server, or fix what stopped certbot (below), then
run the installer again.

### Port 80 or 443 is used by another program

**What you see.**

```text
Port 80 is used by another program:
  …
Stop it, then run this again. Nothing was changed.
```

On a server without nginx yet, the installer checks before it installs nginx's package, which
would otherwise fail to start, and says "nginx is not installed, and port 80 is used by another
program", with the same advice.

**Why.** Another web server, such as Apache or Caddy, holds the port. nginx itself is fine there.

**What to do.** Stop and disable the other program (`sudo systemctl disable --now apache2`, for
example), or move it to other ports, then run the installer again.

### Another site claims a name

**What you see.**

```text
Another site on this nginx answers for a name COLLAB would take:
  dao.genesisl1.com  (/etc/nginx/sites-enabled/example.conf)
Remove the name there, or leave it out here (--main, --alias, --no-aliases). Nothing was changed.
```

**Why.** Another site's `server_name` names one of COLLAB's names exactly. Two sites cannot both
answer for it.

**What to do.** Remove the name from that file, then `sudo nginx -t && sudo systemctl reload nginx`
and run the installer again; or install without that name (`--alias` with the names you keep, or
`--no-aliases`). A wildcard such as `*.genesisl1.com` is only reported, never refused.

### nginx's configuration does not pass its own test

**What you see.** "nginx's configuration does not pass its own test now:", nginx's error lines, then
"Fix that first. Nothing was changed."

**Why.** Something in nginx's configuration was already broken before the installer ran: another
site's file, or a hand edit.

**What to do.** `sudo nginx -t` names the file and line. Fix it, then run the installer again.

### A file at COLLAB's path is not COLLAB's

**What you see.** "/etc/nginx/sites-available/collab.conf exists and was not written by this
installer: move it away (or remove it), then run this again.", or "/etc/nginx/sites-enabled/zz-collab.conf
exists and is not COLLAB's: move it away, then run this again."

**Why.** The installer writes only files that carry its own marker line, so it never overwrites a
configuration someone else wrote, such as one from `deploy/nginx/collab.conf`.

**What to do.** Move that file away (and remove its link in `sites-enabled`), reload nginx, then run
the installer again.

### The installer refuses a folder under shared/

**What you see.** "error: shared/history is a symlink or not a directory; refusing to touch it", or
"error: shared/jobs exists owned by …, not …; refusing to change it".

**Why.** A folder under `/var/www/collab/shared/` was replaced by a symbolic link or belongs to the
wrong user. The services never do that. The installer refuses, rather than follow the link as root.

**What to do.** Treat it as a possible break-in until you know better: find out how it got there.
Then move the link or folder away and run the installer again, which creates the folder anew.

### certbot cannot get or renew the certificate

**What you see.** The installer prints certbot's last lines and "warning: certbot could not get the
certificate (above)."; or, later, Status shows the certificate's days left running down, amber below
21 and red below 8.

**Why.** Let's Encrypt checks each name by fetching a file over HTTP, on port 80, from the
internet. The usual causes:
- port 80 is closed in a firewall or the provider's security group;
- the name's A or AAAA record points elsewhere (an AAAA record to an address the server does not
  answer on counts too);
- a proxy in front of the server does not pass the request through (with Cloudflare, use DNS only
  until the certificate exists);
- too many failed attempts: Let's Encrypt allows only a few failed validations an hour;
- for a renewal, a name was removed from the site but not from the certificate (see
  [Removing a name](domains.md#removing-a-name)).

**What to do.** `/var/log/letsencrypt/letsencrypt.log` has certbot's whole story, and
`sudo certbot certificates` shows what exists. Fix the cause, wait out a rate limit, then run the
installer again. To test renewals without waiting: `sudo certbot renew --dry-run`.

### The installer refuses an option

**What you see.** One of: "--main takes a host name such as collab.genesisl1.com", "--alias takes a
host name such as dao.genesisl1.com (got …)", "--alias or --no-aliases, not both", "--email takes an
address such as admin@example.org", "--rpc takes the https URL of a GenesisL1 JSON-RPC endpoint",
"ports are numbers from 1 to 65535 (got …)", or "Unknown option: … (try --help)".

**What to do.** Give the option a plain host name (no `https://`, no path is needed, though both are
stripped), an address, an https URL, or a port number, as the message says. `--help` lists every
option; see [INSTALL.md](INSTALL.md#options).

### Other refusals at the start

| Message | What to do |
|---|---|
| "Run as root: sudo …" | run it with `sudo` |
| "This installer supports Debian and Ubuntu (apt); this system is …" | use a Debian or Ubuntu server |
| "systemd is not running on this machine: run with --no-systemd and start the services yourself." | a container without systemd: use a server or a virtual machine with systemd |
| "There is no /etc/nginx/nginx.conf: this installer works with the nginx package of Debian and Ubuntu." | remove the other nginx build and let the installer install the distribution's package |
| "This is not a complete COLLAB release: … is missing." | extract the archive again, after `sha256sum -c` |

## Publishing and the admin page

### Every page still says COLLAB is not deployed

**What you see.** After deploying from the admin page, the site still shows "COLLAB is not deployed
on this site yet", and Status says "Not deployed".

**Why.** Deploying puts the contracts on chain; publishing tells this site to use them. The Deploy
tab's last step, **Publish to this site**, was not done: the session had ended, the page was closed
after step 5, or the server refused the record ("The server refused the record.").

**What to do.**
- In the same browser, with the same wallet account, open the Deploy tab again: it resumes where it
  stopped. Sign in in step 6 and press **Publish to this site**.
- From another browser, use the Publish tab: choose the `29.json` you downloaded, or **Read it from
  the chain** with the DAO factory's address (shown in the Deploy tab's ledger as the DAO factory,
  and on GL1X among the deploying account's contract creations).

See [Publish, unpublish and check](publish.md). Pages read `config.js` on every load, so the site
changes at the next reload.

### The admin service is not running (503)

**What you see.** The admin page says "The admin service is not running." and the masthead
**Admin service stopped**. nginx answers its API with "The admin service is not running (sudo
systemctl status collab-admin)."

**Why.** `collab-admin.service` stopped or cannot start. The journal says why: a settings file it
cannot read ("COLLAB_ADMIN_TOKEN in /etc/collab/collab.env is missing or shorter than 16
characters", after a hand edit), its port taken by another program ("Address already in use"), or
an internal error.

**What to do.**

```bash
sudo systemctl status collab-admin
sudo journalctl -u collab-admin -n 30
sudo systemctl restart collab-admin
```

If another program holds port 8411 on 127.0.0.1, run the installer again with `--admin-port` and a
free port. A damaged `collab.env` is put right by restoring it from a backup
([Upgrades, rollbacks and backups](upgrades.md#restoring)). The public site keeps working meanwhile.

### Sign-in says "Too many failed sign-ins" (429)

**What you see.** "Too many failed sign-ins from your address. Try again in … minute(s)." and the
**Sign in** button counting down.

**Why.** Ten failed sign-ins in ten minutes came from your address (an IPv6 /64 counts as one). The
admin service then refuses that address, even with the right token, until the oldest failure is ten
minutes old. Behind a reverse proxy such as Cloudflare's, every visitor shares the proxy's
addresses, so others' failures count against you.

**What to do.** Wait for the countdown, and copy the token from `sudo collab admin-token`. Behind a
proxy, set up nginx's `real_ip` module for it, or reach the server directly. If the failures were
not yours, someone is guessing: replace the token with `sudo collab admin-token --new` (see
[Operator security checklist](server-security.md)).

### "Too many requests from your address" (429 from nginx)

**What you see.** "Too many requests from your address. Wait a few seconds."

**Why.** nginx limits sign-ins to 6 a minute and the admin API to 5 requests a second per address,
with short bursts. Scripts or many open admin tabs can reach that.

**What to do.** Wait about ten seconds. Close extra admin tabs.

### Your session keeps ending

**What you see.** "Your session has ended. Sign in again to continue."

**Why.** A session lasts 12 hours. A new admin token (`collab admin-token --new`) ends every session,
and so does a damaged sessions file, which the admin service replaces at start with a warning in its
log ("… is unreadable or malformed …; starting a fresh signing salt, which ends every session").

**What to do.** Sign in again. If it happens without a reason you know, read
`sudo journalctl -u collab-admin` for that warning, and check the disk: Status warns when less than
a tenth is free.

### Signing out reports an error (500)

**What you see.** "You are signed out here, but the server could not save the sign-out."

**Why.** The admin service could not write `shared/status/sessions.json`, usually because the disk is
full. The session is cleared in this browser and in the running service, but a restart of the
service would bring it back.

**What to do.** Free some disk space, then sign in and out again; or end every session with
`sudo collab admin-token --new`.

### The admin API answers 403

**What you see.** "This request comes from another site.", "The request names no Origin or Referer of
this site." or "Requests must come through the web server."

**Why.** The admin service accepts requests only through nginx, from the main host's own pages. A
request through an alias, another port, a browser extension that strips headers, or straight to
127.0.0.1:8411 is refused.

**What to do.** Use the admin page at `https://<main host>/admin`. Scripts against the admin API are
not supported; see the [Admin API reference](admin-api.md).

## The prerender job

### The prerender job ran with warnings

**What you see.** Status shows **Ran with warnings** and the lines; `sudo collab status` prints
"ran with warnings" and each `warning:` line. The pages were written.

| Warning | Why | What to do |
|---|---|---|
| "L1 price not fetched (…); data/price.json stays as it is" | the server cannot reach `api.coingecko.com` over HTTPS (a firewall, a proxy, DNS), or CoinGecko refused or failed. In the release rehearsal, a proxy that refused the connection gave "Tunnel connection failed: 403 Forbidden". | Allow outbound HTTPS to `api.coingecko.com`. Behind a proxy, give the prerender service the proxy variables ([prerender.md](prerender.md#outbound-https)). The site keeps showing US dollars from the last file for a day, then amounts in L1 only. |
| "tokens not read (…); data/tokens.json and the token pages stay as they are" | the token data could not be read from the chain this run: the RPC endpoint failed or refused a request | Usually passes at the next run. If it stays, check the endpoint (below), or give the job a node of your own with `--rpc`. |
| "member network not read (…); og/network.json and the member cache stay as they are" | the endpoint refused the log queries the member network needs, or dropped logs | Use an endpoint that serves `eth_getLogs` fully, with `--rpc`. The home page then reads the members itself. |
| "0x…: logo not used (…)" or "0x…: description not used (…)" | a DAO's logo or description could not be read, or did not match its hash | Nothing to fix on the server: that DAO's page is written without it. |

### Token holders show "≥" and the 24-hour change is missing

**What you see.** On `/tokens`, holder counts such as "≥ 12" ("At least this many accounts hold a
balance. Older transfers are still being read, so the count may grow."), and no 24-hour change or
7-day line for some tokens, in the first runs after a deployment or on a busy chain.

**Why.** The job reads the tokens' history for at most 240 seconds per run, and the next run resumes
where it stopped. Until it is done, `/data/tokens.json` says `"complete": false`.

**What to do.** Nothing: later runs complete it. Prices, liquidity and market caps are live from the
chain meanwhile. See [the tokens time budget](prerender.md#the-tokens-time-budget).

### The prerender job failed

**What you see.** Status shows **Failed**, "The last run failed … ago; the pages from … are still
served.", and the error line.

| Error | Why | What to do |
|---|---|---|
| "no usable RPC endpoint (… serves chain …, the deployment is for chain 29); no file was changed" | the installer's `--rpc` names an endpoint of another chain | Run the installer again with the right `--rpc`, or `--rpc ""` for the site's own endpoints. |
| "no usable RPC endpoint (…); no file was changed", with connection errors | none of the endpoints answered from the server | Check the server's outbound HTTPS and the endpoints; test one with `curl` (below). |
| "… (DAOLens.directory: is this a COLLAB 2.2 deployment?); no file was changed" | the published record names contracts of another version, or the node does not have them | Check the deployment with the Check tab, and publish one of this release. |
| "stopped after 25 minutes" | a very slow endpoint | Give the job a faster or nearer node with `--rpc`. |

`sudo collab prerender` runs the job in the foreground and prints everything it says. Nothing is
lost meanwhile: a failed run changes no file, and the pages of the last good run keep being served.

### The prerender never runs

**What you see.** Status stays on **Not run yet** or **Requested**, and **Next run** passes without
a run.

**Why.** `collab-prerender.timer` or `collab-prerender.path` is not active.

**What to do.** `systemctl status collab-prerender.timer collab-prerender.path collab-prerender`
shows their state, and `sudo collab prerender` runs the job by hand. Running the installer again
enables and starts the units.

## RPC endpoints and the security policy

### An RPC endpoint fails its test

**What you see.** In Site settings, **Test** answers with one of:
- "This endpoint serves chain …, not GenesisL1 (29).": it is not a GenesisL1 endpoint;
- "Its newest block (…) is … minutes old, so the endpoint is not in sync.": it lags more than 15
  minutes behind;
- "No answer within 8 seconds." or "The endpoint answered HTTP …": it is down or refuses;
- "The browser could not reach it. It may be offline, block other sites (CORS) or be outside this
  site's security policy."

**Why.** The test runs in your browser, as visitors' browsers would use the endpoint. For the last
message, the endpoint usually does not allow requests from other sites (it sends no
`Access-Control-Allow-Origin`), or it is an `http://` endpoint on your own computer, which the
policy allows only once the site settings list it.

**What to do.** Use an https endpoint of GenesisL1 that allows cross-origin requests. To see what an
endpoint answers:

```bash
curl -si -X POST -H 'Content-Type: application/json' -H 'Origin: https://collab.genesisl1.com' \
  --data '{"jsonrpc":"2.0","id":1,"method":"eth_chainId","params":[]}' https://rpc.genesisl1.org
```

A GenesisL1 endpoint answers `"result":"0x1d"` (29), and one that browsers can use sends an
`Access-Control-Allow-Origin` header. Saving does not test endpoints: a broken one in the list makes
visitors' pages fall back to the next one.

### A visitor's own endpoint is refused

**What you see.** In the site's Settings, under **Network**, a visitor's own endpoint is refused with
"Not saved. The browser could not reach it. …".

**Why.** On a COLLAB server, the security policy allows every https endpoint, so the endpoint itself
is at fault (down, or no cross-origin requests), or it is an `http://` node on the visitor's own
computer, which the policy does not allow unless the site settings list that very address.

**What to do.** Visitors should use an https endpoint. Do not list a local `http://` address in the
site settings for one visitor: every visitor's browser would try it.

### The security policy update failed

**What you see.** Status, **Security policy**: **Update failed**, "The last update was refused and
every file was put back: …"; or `sudo collab apply` says "error: the policy was not applied, every
file is as it was: …".

**Why.** The policy step tests the whole nginx configuration with `nginx -t`, so a broken file of
another site makes it fail too ("nginx refuses the new policy: …"). It also refuses a release whose
files are not root's alone ("… must belong to root and be writable by root only").

**What to do.** `sudo nginx -t` shows nginx's complaint. Fix it, then `sudo collab apply`. If a
release's permissions were changed, the message names the path. Give the release back to root,
then run the step again:

```bash
sudo chown -R root:root /var/www/collab/releases/2.2.0
sudo chmod -R go-w /var/www/collab/releases/2.2.0
sudo collab apply
```

Running the installer again does not repair this: it copies a release anew only when its files
differ, and stops at the same check.

### Analytics or a new endpoint does not take effect

**What you see.** Right after saving a Google Analytics ID or new RPC endpoints, Status shows the
policy as **Updating** or **Differs**, and Google's script is blocked on some pages.

**Why.** The new policy reaches the pages through the policy step, then the prerendered pages through
the next prerender run, which the step requests. Until both have run, pages keep the old policy.
Visitors who declined analytics, or whose browser sends Global Privacy Control, are not counted at
all.

**What to do.** Wait for **In step** and the next prerender run; reload. If the badge stays on
**Differs**, check `systemctl status collab-apply.path`, then run `sudo collab apply`.

## Upgrades

### The new release's installer stopped

**What you see.** The installer stopped in "Checking nginx" or earlier, or said "nginx refuses
COLLAB's configuration: … COLLAB's nginx file is back as it was, and nginx keeps running as before."

**Why.** The same checks as at the first install, before anything changes. `current` switches to the
new release only after nginx accepted its file, so the site stays on the old release.

**What to do.** Fix what the message names and run the new installer again. See the sections above.

### "the published deployment does not fit this release"

**What you see.** "warning: the published deployment does not fit this release: … Publish a
deployment of this release from /admin."

**Why.** A release accepts contracts of its own major and minor version, and the published record is
of another one.

**What to do.** Deploy this release's contracts from the Deploy tab and publish them. See
[Upgrades, rollbacks and backups](upgrades.md#a-release-of-another-minor-version).

### Pages misbehave after switching releases by hand

**What you see.** After `ln -sfn` on `/var/www/collab/current`, pages load a mix of old and new files,
or stale scripts stay in browsers.

**Why.** nginx keeps a release's scripts for a year only under that release's `?v=` addresses, and
the prerendered pages belong to the release that wrote them. Only the installer changes both with
`current`.

**What to do.** Run the installer of the release you want: `sudo /var/www/collab/releases/<version>/server/install.sh`.
See [Going back to an earlier release](upgrades.md#going-back-to-an-earlier-release).

## Checking the live site

### check_site.py reports failures

**What you see.** `FAIL` lines, and "… checks passed, … failed".

**What to do.**
- A new server cannot pass before its names point at it: it has no certificate until then.
- "note: /admin/api/ answers 503: the admin service is not running" points at the admin service
  (above).
- Header and caching failures behind a CDN or proxy come from the proxy's own rules: check the
  server itself with `--resolve collab.genesisl1.com:443:SERVER-IP`.
- Every other `FAIL` line names the address and what it expected; compare with
  [Deploying COLLAB: reference](DEPLOY.md#4-after-deploying).

### The server's clock is off

**What you see.** Status, **Server**: the server time followed by "(… s ahead of this computer)" or
"(… s behind this computer)", shown when the two clocks differ by more than two minutes.

**Why and what to do.** Sessions, certificates and the prerender's times depend on the server's
clock. Turn on time synchronisation: `timedatectl status` shows it, and
`sudo timedatectl set-ntp true` turns it on.

## Related

- [Status, logs and the collab command](monitoring.md)
- [Installing COLLAB on a server](INSTALL.md)
- [Domains, redirects and certificates](domains.md)
- [Static pages, data files and prices](prerender.md)
- [Upgrades, rollbacks and backups](upgrades.md)