Static pages, data files and prices

Why the site has a prerender job

The site works without it: every page reads the chain in the visitor's browser. The job adds what a browser cannot do for itself:

• Search engines and link previews get a page for every DAO and token, with its name, description and image, without running any script.
• Faster pages. A DAO's logo and description, the token list and the member network come from one file each instead of many reads from the chain.
• Prices without tracking. The L1 price is fetched from CoinGecko on the server, so no visitor's browser contacts CoinGecko for the default display.
• The site settings on first paint. The alpha strip's text, the announcement, the footer's links and the home page's search-engine texts are written into the pages' HTML.

What it writes

The formats of these files are in Data formats.

Pages, in shared/prerender

The data cache, in shared/data, served at /data/

JSON and text files of 1 KB or more get a .gz copy beside them. Logos and descriptions no DAO shows any more are removed.

Before a deployment is published

While config.js says the site is not deployed, the job reads no chain. It writes the entry pages with the site settings, the sitemap of the static and documentation pages, robots.txt and the price, and Status adds the note "not deployed yet: the pages, the sitemap and the price only".

After an unpublish, such a run also removes what the earlier deployment left: its DAO and token pages, its preview images and og/network.json, and its logos, descriptions and /data/tokens.json. The price stays, and so do the job's caches, which are kept outside the folders nginx serves.

How nginx serves its files

nginx serves the release, and tries the job's folder first only for the page addresses, the 404 page, the sitemap, robots.txt and /og/:

• Never from the job's folder: scripts, styles, config.js, the admin page, the favicon and the manifest. They always come from the release, or config.js from shared/.
• /og/ serves only og/network.json and the two image names above, with a DAO's lowercase address; every other name is not found.
• /data/ serves only files of plain names ending in .png, .json or .txt. A file the job has not written yet answers 204 with no-store, and the page reads the chain instead.
• Caching. Pages and /data/*.json are revalidated on every load; /og/ and the sitemap are kept an hour; files under /data/ named by their hash are kept a year.

When it runs

The admin service asks for a run after a publish, an unpublish and every settings save; the installer asks after every run; the content policy step asks after it rewrote the pages. A request that arrives during a run starts another run when it ends, up to three in a row. Runs never overlap: one that finds another still working exits and does nothing.

Status shows Next run ("in 7 min, every 10 minutes", or "Now" after a request).

The command it runs

collab-prerender.service runs, as user collab-prerender:

python3 -I /var/www/collab/current/tools/prerender.py --web /var/www/collab/current/web
  --deployment /var/www/collab/shared/config.js --out /var/www/collab/shared/prerender
  --data /var/www/collab/shared/data --site https://collab.genesisl1.com
  --lock /var/www/collab/shared/jobs/prerender.lock --prune --quiet [--rpc URL]

• It reads the chain id, the contracts and the RPC endpoints from config.js, trying the endpoints in order. When the installer was given --rpc, the job reads the chain through that endpoint only.
• --prune removes the pages and images of DAOs the factory no longer lists, token pages no longer written, and unused logos and descriptions.
• The job's status is in shared/jobs/prerender.json, and its output in shared/jobs/prerender.log (the last 256 KB) and in the journal (journalctl -u collab-prerender).

Its user and what it may write

The job reads untrusted data from the chain, logos included, which Pillow decodes. So it runs as a user of its own, collab-prerender, confined by systemd:

• it may write only shared/prerender/, shared/data/ and shared/jobs/;
• its settings file, /etc/collab/prerender.env, holds no secret, and it cannot read collab.env, which holds the admin token: it is not in the collab group;
• it cannot write config.js, the release or nginx's configuration;
• nginx never serves a script, a style or the admin page from its folders, and follows a symbolic link in them only when the link and its target have the same owner.

So a flaw in the job, or in Pillow, could deface the prerendered pages, but could not change the contracts the site uses or run code on the site's origin. Security has the full threat model.

It runs at low priority (nice 10, idle I/O). A run that takes more than 25 minutes is stopped ("error: stopped after 25 minutes").

The tokens time budget

The token list needs the history of every pool, from the deployment's start block, and the transfers of every token, to count holders. Reading that history takes the job at most 240 seconds per run; the next run resumes where it stopped, from a cache beside the lock (shared/jobs/prerender.tokens.json, readable by the job's user only). On a busy chain the first runs after a deployment may not finish it. Until they do:

• /data/tokens.json says "complete": false;
• holder counts are lower bounds, shown as "≥ …" with "At least this many accounts hold a balance. Older transfers are still being read, so the count may grow.";
• the 24-hour change and the 7-day line may be missing for some tokens.

Prices, liquidity and market caps do not wait for the history: the pages compute them live from the chain.

The price snapshot

At each run the job asks CoinGecko's simple price API for the L1 price in US dollars, with its 24-hour change, and writes /data/price.json. A failed fetch keeps the previous file, with a warning. The pages show amounts in US dollars from that file while it is less than a day old; with an older file, or none, they show amounts in L1 only, unless the visitor chooses a currency in Settings, which the browser then fetches from CoinGecko itself. Prices are for display only: no amount sent on chain is ever computed from them.

Warnings and failures

A run ends in one of three states, which Status and sudo collab status show:

Warnings

The job warns and goes on when something extra cannot be read; the file it concerns keeps its last good version:

• "L1 price not fetched (…); data/price.json stays as it is": CoinGecko could not be reached or gave no usable answer;
• "tokens not read (…); data/tokens.json and the token pages stay as they are": MarketLens or the token data could not be read this time;
• "member network not read (…); og/network.json and the member cache stay as they are";
• "0x…: description not used (…)" or "0x…: logo not used (…)": a DAO's description or logo could not be read, or did not match the hash its profile stores; its page is written without it;
• "og/network.json is … KiB, more than the 512 KiB the landing page reads: it reads the members from the chain instead".

Status lists up to 20 warning lines; see The admin page.

Failures

The job reads every DAO before it writes anything, so a chain problem changes no file:

• "error: no usable RPC endpoint (…); no file was changed": no endpoint answered for chain 29, or the one given serves another chain;
• "error: …; no file was changed": an RPC error while reading the DAOs;
• "error: stopped after 25 minutes".

A configuration problem (a missing file, an invalid config.js) also fails the run. Status shows "The last run failed … ago; the pages from … are still served." with the error line, and the warnings of that run. sudo collab prerender runs the job in the foreground and prints everything it says; see Troubleshooting.

Outbound HTTPS

The job needs outbound HTTPS to:

• the RPC endpoints: the site's own (from the site settings, or the GenesisL1 RPC and the GL1X JSON-RPC by default), or the installer's --rpc;
• api.coingecko.com, for the price.

When the server reaches the internet only through a proxy, the job passes http_proxy, https_proxy and no_proxy (in lower or upper case) on to tools/prerender.py when its service has them, and SSL_CERT_FILE for a proxy with a certificate authority of its own. Give them to the service with a systemd drop-in, which the installer leaves alone:

sudo systemctl edit collab-prerender.service
# then, in the editor:
# [Service]
# Environment=https_proxy=http://proxy.example.org:3128

After an upgrade or a settings change

• An upgrade removes the previous release's prerendered pages when current switches, so the new release's own pages serve until the job has written them again; the installer asks for that run at once. Images and the /data/ cache stay.
• A settings save asks for a run, which writes the settings into the pages.
• A security policy change asks for a run after the policy step, so the prerendered pages carry the new policy. Until it has run, those pages keep the old policy.

Related

• The admin page
• Data formats
• Status, logs and the collab command
• Troubleshooting
• Deploying COLLAB: reference