Troubleshooting

The installer can always be run again: it repairs what is missing and keeps what is there. Its refusals happen before it changes anything, and say "Nothing was changed".

Installing

A name is left without a certificate

What you see. In the installer's "DNS and certificate" step, a line such as dao.descifi.org: no DNS record (or it does not resolve from here) or dao.descifi.org: points at 192.0.2.10, not at this server, and in the summary "Not covered: dao.descifi.org: point their DNS at this server, then run this again (the same command)." Status names it in the notice "One name has no certificate."

Why. The certificate covers only the names whose DNS points at the server when the installer runs. The record is missing, points elsewhere, or has not spread yet. Behind NAT, where the public address is not on the server itself, the installer fetches a one-time file through the name from the server; a router that does not send that request back in fails the check even when the record is right, and the line names the server's own public address.

What to do. Fix the A (and AAAA) record, check it with dig +short dao.descifi.org, then run the same command again: the certificate is extended to the name. Behind NAT with a right record, turn on the router's NAT loopback (hairpinning) first. Until then, the name still redirects over HTTP, but not over HTTPS. See Before you install.

The site answers "COLLAB is being set up" (503)

What you see. http:// answers "COLLAB is being set up on this server, HTTPS first. Please come back in a few minutes." The installer warned "collab.genesisl1.com does not point at this server yet, so it cannot have a certificate." or "certbot could not get the certificate (above).", and its summary says "COLLAB 2.2.0 is installed, but not on HTTPS yet: see the warnings above."

Why. The main host has no certificate yet, so nginx serves only Let's Encrypt's challenges and this notice on port 80.

What to do. Point the main host's DNS at the server, or fix what stopped certbot (below), then run the installer again.

Port 80 or 443 is used by another program

What you see.

Port 80 is used by another program:
  …
Stop it, then run this again. Nothing was changed.

On a server without nginx yet, the installer checks before it installs nginx's package, which would otherwise fail to start, and says "nginx is not installed, and port 80 is used by another program", with the same advice.

Why. Another web server, such as Apache or Caddy, holds the port. nginx itself is fine there.

What to do. Stop and disable the other program (sudo systemctl disable --now apache2, for example), or move it to other ports, then run the installer again.

Another site claims a name

What you see.

Another site on this nginx answers for a name COLLAB would take:
  dao.genesisl1.com  (/etc/nginx/sites-enabled/example.conf)
Remove the name there, or leave it out here (--main, --alias, --no-aliases). Nothing was changed.

Why. Another site's server_name names one of COLLAB's names exactly. Two sites cannot both answer for it.

What to do. Remove the name from that file, then sudo nginx -t && sudo systemctl reload nginx and run the installer again; or install without that name (--alias with the names you keep, or --no-aliases). A wildcard such as *.genesisl1.com is only reported, never refused.

nginx's configuration does not pass its own test

What you see. "nginx's configuration does not pass its own test now:", nginx's error lines, then "Fix that first. Nothing was changed."

Why. Something in nginx's configuration was already broken before the installer ran: another site's file, or a hand edit.

What to do. sudo nginx -t names the file and line. Fix it, then run the installer again.

A file at COLLAB's path is not COLLAB's

What you see. "/etc/nginx/sites-available/collab.conf exists and was not written by this installer: move it away (or remove it), then run this again.", or "/etc/nginx/sites-enabled/zz-collab.conf exists and is not COLLAB's: move it away, then run this again."

Why. The installer writes only files that carry its own marker line, so it never overwrites a configuration someone else wrote, such as one from deploy/nginx/collab.conf.

What to do. Move that file away (and remove its link in sites-enabled), reload nginx, then run the installer again.

The installer refuses a folder under shared/

What you see. "error: shared/history is a symlink or not a directory; refusing to touch it", or "error: shared/jobs exists owned by …, not …; refusing to change it".

Why. A folder under /var/www/collab/shared/ was replaced by a symbolic link or belongs to the wrong user. The services never do that. The installer refuses, rather than follow the link as root.

What to do. Treat it as a possible break-in until you know better: find out how it got there. Then move the link or folder away and run the installer again, which creates the folder anew.

certbot cannot get or renew the certificate

What you see. The installer prints certbot's last lines and "warning: certbot could not get the certificate (above)."; or, later, Status shows the certificate's days left running down, amber below 21 and red below 8.

Why. Let's Encrypt checks each name by fetching a file over HTTP, on port 80, from the internet. The usual causes:

• port 80 is closed in a firewall or the provider's security group;
• the name's A or AAAA record points elsewhere (an AAAA record to an address the server does not answer on counts too);
• a proxy in front of the server does not pass the request through (with Cloudflare, use DNS only until the certificate exists);
• too many failed attempts: Let's Encrypt allows only a few failed validations an hour;
• for a renewal, a name was removed from the site but not from the certificate (see Removing a name).

What to do. /var/log/letsencrypt/letsencrypt.log has certbot's whole story, and sudo certbot certificates shows what exists. Fix the cause, wait out a rate limit, then run the installer again. To test renewals without waiting: sudo certbot renew --dry-run.

The installer refuses an option

What you see. One of: "--main takes a host name such as collab.genesisl1.com", "--alias takes a host name such as dao.genesisl1.com (got …)", "--alias or --no-aliases, not both", "--email takes an address such as admin@example.org", "--rpc takes the https URL of a GenesisL1 JSON-RPC endpoint", "ports are numbers from 1 to 65535 (got …)", or "Unknown option: … (try --help)".

What to do. Give the option a plain host name (no https://, no path is needed, though both are stripped), an address, an https URL, or a port number, as the message says. --help lists every option; see INSTALL.md.

Other refusals at the start

Publishing and the admin page

Every page still says COLLAB is not deployed

What you see. After deploying from the admin page, the site still shows "COLLAB is not deployed on this site yet", and Status says "Not deployed".

Why. Deploying puts the contracts on chain; publishing tells this site to use them. The Deploy tab's last step, Publish to this site, was not done: the session had ended, the page was closed after step 5, or the server refused the record ("The server refused the record.").

What to do.

• In the same browser, with the same wallet account, open the Deploy tab again: it resumes where it stopped. Sign in in step 6 and press Publish to this site.
• From another browser, use the Publish tab: choose the 29.json you downloaded, or Read it from the chain with the DAO factory's address (shown in the Deploy tab's ledger as the DAO factory, and on GL1X among the deploying account's contract creations).

See Publish, unpublish and check. Pages read config.js on every load, so the site changes at the next reload.

The admin service is not running (503)

What you see. The admin page says "The admin service is not running." and the masthead Admin service stopped. nginx answers its API with "The admin service is not running (sudo systemctl status collab-admin)."

Why. collab-admin.service stopped or cannot start. The journal says why: a settings file it cannot read ("COLLAB_ADMIN_TOKEN in /etc/collab/collab.env is missing or shorter than 16 characters", after a hand edit), its port taken by another program ("Address already in use"), or an internal error.

What to do.

sudo systemctl status collab-admin
sudo journalctl -u collab-admin -n 30
sudo systemctl restart collab-admin

If another program holds port 8411 on 127.0.0.1, run the installer again with --admin-port and a free port. A damaged collab.env is put right by restoring it from a backup (Upgrades, rollbacks and backups). The public site keeps working meanwhile.

Sign-in says "Too many failed sign-ins" (429)

What you see. "Too many failed sign-ins from your address. Try again in … minute(s)." and the Sign in button counting down.

Why. Ten failed sign-ins in ten minutes came from your address (an IPv6 /64 counts as one). The admin service then refuses that address, even with the right token, until the oldest failure is ten minutes old. Behind a reverse proxy such as Cloudflare's, every visitor shares the proxy's addresses, so others' failures count against you.

What to do. Wait for the countdown, and copy the token from sudo collab admin-token. Behind a proxy, set up nginx's real_ip module for it, or reach the server directly. If the failures were not yours, someone is guessing: replace the token with sudo collab admin-token --new (see Operator security checklist).

"Too many requests from your address" (429 from nginx)

What you see. "Too many requests from your address. Wait a few seconds."

Why. nginx limits sign-ins to 6 a minute and the admin API to 5 requests a second per address, with short bursts. Scripts or many open admin tabs can reach that.

What to do. Wait about ten seconds. Close extra admin tabs.

Your session keeps ending

What you see. "Your session has ended. Sign in again to continue."

Why. A session lasts 12 hours. A new admin token (collab admin-token --new) ends every session, and so does a damaged sessions file, which the admin service replaces at start with a warning in its log ("… is unreadable or malformed …; starting a fresh signing salt, which ends every session").

What to do. Sign in again. If it happens without a reason you know, read sudo journalctl -u collab-admin for that warning, and check the disk: Status warns when less than a tenth is free.

Signing out reports an error (500)

What you see. "You are signed out here, but the server could not save the sign-out."

Why. The admin service could not write shared/status/sessions.json, usually because the disk is full. The session is cleared in this browser and in the running service, but a restart of the service would bring it back.

What to do. Free some disk space, then sign in and out again; or end every session with sudo collab admin-token --new.

The admin API answers 403

What you see. "This request comes from another site.", "The request names no Origin or Referer of this site." or "Requests must come through the web server."

Why. The admin service accepts requests only through nginx, from the main host's own pages. A request through an alias, another port, a browser extension that strips headers, or straight to 127.0.0.1:8411 is refused.

What to do. Use the admin page at https://<main host>/admin. Scripts against the admin API are not supported; see the Admin API reference.

The prerender job

The prerender job ran with warnings

What you see. Status shows Ran with warnings and the lines; sudo collab status prints "ran with warnings" and each warning: line. The pages were written.

Token holders show "≥" and the 24-hour change is missing

What you see. On /tokens, holder counts such as "≥ 12" ("At least this many accounts hold a balance. Older transfers are still being read, so the count may grow."), and no 24-hour change or 7-day line for some tokens, in the first runs after a deployment or on a busy chain.

Why. The job reads the tokens' history for at most 240 seconds per run, and the next run resumes where it stopped. Until it is done, /data/tokens.json says "complete": false.

What to do. Nothing: later runs complete it. Prices, liquidity and market caps are live from the chain meanwhile. See the tokens time budget.

The prerender job failed

What you see. Status shows Failed, "The last run failed … ago; the pages from … are still served.", and the error line.

sudo collab prerender runs the job in the foreground and prints everything it says. Nothing is lost meanwhile: a failed run changes no file, and the pages of the last good run keep being served.

The prerender never runs

What you see. Status stays on Not run yet or Requested, and Next run passes without a run.

Why. collab-prerender.timer or collab-prerender.path is not active.

What to do. systemctl status collab-prerender.timer collab-prerender.path collab-prerender shows their state, and sudo collab prerender runs the job by hand. Running the installer again enables and starts the units.

RPC endpoints and the security policy

An RPC endpoint fails its test

What you see. In Site settings, Test answers with one of:

• "This endpoint serves chain …, not GenesisL1 (29).": it is not a GenesisL1 endpoint;
• "Its newest block (…) is … minutes old, so the endpoint is not in sync.": it lags more than 15 minutes behind;
• "No answer within 8 seconds." or "The endpoint answered HTTP …": it is down or refuses;
• "The browser could not reach it. It may be offline, block other sites (CORS) or be outside this site's security policy."

Why. The test runs in your browser, as visitors' browsers would use the endpoint. For the last message, the endpoint usually does not allow requests from other sites (it sends no Access-Control-Allow-Origin), or it is an http:// endpoint on your own computer, which the policy allows only once the site settings list it.

What to do. Use an https endpoint of GenesisL1 that allows cross-origin requests. To see what an endpoint answers:

curl -si -X POST -H 'Content-Type: application/json' -H 'Origin: https://collab.genesisl1.com' \
  --data '{"jsonrpc":"2.0","id":1,"method":"eth_chainId","params":[]}' https://rpc.genesisl1.org

A GenesisL1 endpoint answers "result":"0x1d" (29), and one that browsers can use sends an Access-Control-Allow-Origin header. Saving does not test endpoints: a broken one in the list makes visitors' pages fall back to the next one.

A visitor's own endpoint is refused

What you see. In the site's Settings, under Network, a visitor's own endpoint is refused with "Not saved. The browser could not reach it. …".

Why. On a COLLAB server, the security policy allows every https endpoint, so the endpoint itself is at fault (down, or no cross-origin requests), or it is an http:// node on the visitor's own computer, which the policy does not allow unless the site settings list that very address.

What to do. Visitors should use an https endpoint. Do not list a local http:// address in the site settings for one visitor: every visitor's browser would try it.

The security policy update failed

What you see. Status, Security policy: Update failed, "The last update was refused and every file was put back: …"; or sudo collab apply says "error: the policy was not applied, every file is as it was: …".

Why. The policy step tests the whole nginx configuration with nginx -t, so a broken file of another site makes it fail too ("nginx refuses the new policy: …"). It also refuses a release whose files are not root's alone ("… must belong to root and be writable by root only").

What to do. sudo nginx -t shows nginx's complaint. Fix it, then sudo collab apply. If a release's permissions were changed, the message names the path. Give the release back to root, then run the step again:

sudo chown -R root:root /var/www/collab/releases/2.2.0
sudo chmod -R go-w /var/www/collab/releases/2.2.0
sudo collab apply

Running the installer again does not repair this: it copies a release anew only when its files differ, and stops at the same check.

Analytics or a new endpoint does not take effect

What you see. Right after saving a Google Analytics ID or new RPC endpoints, Status shows the policy as Updating or Differs, and Google's script is blocked on some pages.

Why. The new policy reaches the pages through the policy step, then the prerendered pages through the next prerender run, which the step requests. Until both have run, pages keep the old policy. Visitors who declined analytics, or whose browser sends Global Privacy Control, are not counted at all.

What to do. Wait for In step and the next prerender run; reload. If the badge stays on Differs, check systemctl status collab-apply.path, then run sudo collab apply.

Upgrades

The new release's installer stopped

What you see. The installer stopped in "Checking nginx" or earlier, or said "nginx refuses COLLAB's configuration: … COLLAB's nginx file is back as it was, and nginx keeps running as before."

Why. The same checks as at the first install, before anything changes. current switches to the new release only after nginx accepted its file, so the site stays on the old release.

What to do. Fix what the message names and run the new installer again. See the sections above.

"the published deployment does not fit this release"

What you see. "warning: the published deployment does not fit this release: … Publish a deployment of this release from /admin."

Why. A release accepts contracts of its own major and minor version, and the published record is of another one.

What to do. Deploy this release's contracts from the Deploy tab and publish them. See Upgrades, rollbacks and backups.

Pages misbehave after switching releases by hand

What you see. After ln -sfn on /var/www/collab/current, pages load a mix of old and new files, or stale scripts stay in browsers.

Why. nginx keeps a release's scripts for a year only under that release's ?v= addresses, and the prerendered pages belong to the release that wrote them. Only the installer changes both with current.

What to do. Run the installer of the release you want: sudo /var/www/collab/releases/<version>/server/install.sh. See Going back to an earlier release.

Checking the live site

check_site.py reports failures

What you see. FAIL lines, and "… checks passed, … failed".

What to do.

• A new server cannot pass before its names point at it: it has no certificate until then.
• "note: /admin/api/ answers 503: the admin service is not running" points at the admin service (above).
• Header and caching failures behind a CDN or proxy come from the proxy's own rules: check the server itself with --resolve collab.genesisl1.com:443:SERVER-IP.
• Every other FAIL line names the address and what it expected; compare with Deploying COLLAB: reference.

The server's clock is off

What you see. Status, Server: the server time followed by "(… s ahead of this computer)" or "(… s behind this computer)", shown when the two clocks differ by more than two minutes.

Why and what to do. Sessions, certificates and the prerender's times depend on the server's clock. Turn on time synchronisation: timedatectl status shows it, and sudo timedatectl set-ntp true turns it on.

Related

• Status, logs and the collab command
• Installing COLLAB on a server
• Domains, redirects and certificates
• Static pages, data files and prices
• Upgrades, rollbacks and backups