<!-- MIT License. Copyright (c) 2026 Decentralized Science Labs LLC. -->
<!-- summary: Watching a COLLAB server: the collab command (status, logs, prerender, admin-token, apply), the journal, nginx's logs, check_site.py and healthy values. -->
# Status, logs and the collab command

This page is for the operator who keeps a COLLAB server running: what a healthy server looks like,
the `collab` helper the installer puts in `/usr/local/sbin`, the journal and nginx's logs, and
`deploy/check_site.py`, which checks the live site from outside. The admin page's Status tab shows
most of the same facts in the browser; see [The admin page](admin.md#status).

## What healthy looks like

| Where | Healthy |
|---|---|
| `sudo collab status` | the deployment names chain 29 and its factory; the prerender is `ok` with a last run less than 10 minutes ago; the policy is "in step with config.js"; the certificate covers every name and ends more than 30 days ahead; every unit is `active` |
| The Status tab | **Published**, **Up to date**, **In step**, a green certificate badge, and no disk warning |
| `deploy/check_site.py` | "… checks passed, 0 failed" |
| `systemctl list-timers` | `collab-prerender.timer` and certbot's timer both scheduled |

## The collab command

The installer writes the helper from `server/bin/collab`. Every command but `help` runs as root:

| Command | What it does |
|---|---|
| `sudo collab status` | the release, site, deployment, prerender, content policy, certificate and services |
| `sudo collab logs [-f] [LINES]` | the journal of the admin service, the prerender job and the policy step |
| `sudo collab prerender` | runs the prerender job now, in the foreground, as `collab-prerender` |
| `sudo collab prerender --request` | asks the prerender service for a run instead |
| `sudo collab admin-token` | shows the admin token |
| `sudo collab admin-token --new` | makes a new token and ends every session |
| `sudo collab apply` | the content policy step, by hand |
| `sudo collab renewed` | certbot's deploy hook: test and reload nginx, note the new certificate |
| `collab help` | this list |

### collab status

From the release rehearsal, on a published site with a test certificate (the sandbox installed
under other paths, on port 8802):

```text
Release:      2.2.0  (/var/tmp/collab-rehearsal-22.kC5CMN/root/releases/2.2.0)
Site:         https://collab.genesisl1.com:8802/
Aliases:      dao.genesisl1.com, collab.descifi.org, dao.descifi.org (301 to the site)
Deployment:   chain 29, COLLAB DAO v2.2.0, factory 0x539F713206437EE4c1FC978Fa74874aEF880cB74
Prerender:    ok, last run 2026-11-01T22:13:53Z, last success 2026-11-01T22:13:53Z
Policy:       in step with config.js
Certificate:  self-signed: collab.genesisl1.com, dao.genesisl1.com, collab.descifi.org, dao.descifi.org, valid until 2027-01-30T21:56:09Z
```

What the lines can say:
- **Deployment:** "none yet (open /admin to deploy)" before a deployment is published.
- **Prerender:** `never`, `requested`, `running`, `ok`, `ran with warnings` (followed by one
  `warning:` line each) or `failed` with "error: …"; "(not deployed yet: the pages, the sitemap and
  the price only)" while nothing is published.
- **Policy:** "in step with config.js" or "differs from config.js", with "(an update is pending)"
  while the policy step is due to run.
- **Certificate:** `letsencrypt` or `self-signed`, the names it covers and its end, then "without a
  certificate: …" for the names it does not.

With systemd, one line follows per unit with its state (`active`, `inactive` or `failed`):
`collab-admin.service`, `collab-prerender.timer`, `collab-prerender.path`, `collab-apply.path` and
`nginx.service`. The helper reads the services' files as their own user, never as root.

### collab logs

`sudo collab logs` prints the last 100 lines of the journal of `collab-admin`, `collab-prerender`
and `collab-apply` together; `sudo collab logs 300` prints 300, and `sudo collab logs -f` follows
new lines. What they hold:
- **The admin service:** one line per request (the address, the method, the path, the status and the
  time it took), sign-ins ("signed in from …", "failed sign-in from …", "sign-in refused: too many
  failures from …"), "deployment published: factory …", "deployment unpublished", "site settings
  saved", and the trace of any internal error. The admin token never appears in it.
- **The prerender job:** each run's output: the files it created, updated or removed, its warnings
  and errors.
- **The policy step:** "policy applied: … file(s) updated", or why it was refused.

### collab prerender

`sudo collab prerender` runs one prerender in the foreground, as the service would, prints its
output and ends with the state: `prerender: ok`, `prerender: warn`, or `prerender: failed` and the
error line. Use it to see a failure as it happens. `sudo collab prerender --request` leaves the run
to the service instead, as **Run now** does; follow it with `sudo collab logs -f`.

### collab admin-token

`sudo collab admin-token` prints the token. `sudo collab admin-token --new` writes a new one, restarts
the admin service, prints the new token, and notes "(a new admin token: every session signed in with
the old one has ended)".

### collab apply

`sudo collab apply` writes the Content-Security-Policy that `config.js` needs into the release's pages
and COLLAB's nginx file, tests nginx and reloads it. It prints "policy applied: … file(s) updated",
or with nothing to change, "policy unchanged: the pages and nginx already carry it". When nginx
refuses the result, it puts every file back and says why. The admin page starts the same step by
itself after a setting that changes the policy.

## The journal

Each service logs to the journal under its unit's name:

```bash
sudo journalctl -u collab-admin -n 50        # the admin service
sudo journalctl -u collab-prerender -n 50    # the prerender job
sudo journalctl -u collab-apply -n 50        # the content policy step
sudo journalctl -u nginx -n 50               # nginx's own service
sudo journalctl -u certbot -n 50             # certificate renewals (the Debian and Ubuntu package)
systemctl status collab-admin collab-prerender.timer collab-prerender.path collab-apply.path
```

The prerender job also keeps its own log, `shared/jobs/prerender.log` (the last 256 KB), which Status
shows under **The end of the log**.

## nginx's logs

COLLAB's server blocks have no log files of their own, so their requests go to nginx's main logs,
with those of any other site on the server: on Debian and Ubuntu, `/var/log/nginx/access.log` and
`/var/log/nginx/error.log`. Requests the admin API's limits refused show there with status `429`, and
requests for a stopped admin service with `503`.

## The status files

The services keep their state in small JSON files that Status reads. Only root and the services'
users can read them:

| File | Holds |
|---|---|
| `shared/jobs/prerender.json` | the prerender's state, last run, last success, error, warnings, time taken |
| `shared/status/apply.json` | the policy step's last run: applied, unchanged or failed, and the files it changed |
| `shared/status/certificates.json` | the names the certificate covers, its end, and the names without one |

## Checking the live site

`deploy/check_site.py` checks a COLLAB site from outside, as a browser and a crawler see it. It needs
only Python 3, so it runs on the server or on your computer from the extracted archive:

```bash
python3 /var/www/collab/current/deploy/check_site.py https://collab.genesisl1.com
```

It checks:
- every clean URL, the redirects and the 404 page, and that no hidden or development file is served;
- the security headers on pages, redirects and errors, with the CSP header equal to the pages'
  policy plus `frame-ancestors`;
- caching, compression and content types;
- `robots.txt` and `sitemap.xml`, and that the noindex pages say so and stay out of the sitemap;
- the documentation: one of its pages, that page's Markdown and `/llms.txt`;
- the first DAO pages in the sitemap with their JSON-LD and their 1200 × 630 preview images;
- `/admin` with its noindex header, the admin service and the rules of `/data/`;
- `/og/network.json` when it is there.

It prints one `FAIL` line per failed check, and ends with "… checks passed, … failed"; it exits with
status 1 when a check fails. The test sandbox's run ended "171 checks passed, 0 failed".

| Option | What it does |
|---|---|
| `--resolve HOST:PORT:ADDRESS` | connect to ADDRESS for HOST:PORT, as `curl --resolve`: checks one server, whatever the DNS says (repeatable) |
| `--dao SLUG` | check this DAO's page (repeatable; by default the first ones in the sitemap) |
| `--cafile FILE` | trust a test certificate |
| `--skip-missing` | skip entry pages the site does not have yet |
| `--static-host` | for Netlify or Cloudflare Pages, which serve files as they are and cache by their own defaults |

To check the server itself rather than what its names lead to, such as a proxy or CDN in front of
it, pin the name to the server's address. The HTTPS checks need the server's certificate, which a
new server gets only once its names point at it:

```bash
python3 deploy/check_site.py https://collab.genesisl1.com --resolve collab.genesisl1.com:443:SERVER-IP
```

## What to watch

- **After every install, upgrade or settings change:** `sudo collab status`, then `check_site.py`.
- **Every week or so:** the certificate's days left, the disk, and the prerender's warnings.
- **In the admin service's log:** failed sign-ins from addresses you do not know. Replace the token
  with `sudo collab admin-token --new` when in doubt.

## Related

- [The admin page](admin.md)
- [Static pages, data files and prices](prerender.md)
- [Troubleshooting](troubleshooting.md)
- [Operator security checklist](server-security.md)