README.md 79.8 KB
Newer Older
Aral Balkan's avatar
Aral Balkan committed
1
# Site.js
2
## Small web construction set.
Aral Balkan's avatar
Aral Balkan committed
3

4
[![Person lying on the ground, working on a laptop with the Site.js logo on screen](images/person.svg)](https://sitejs.org)
5

6
## Develop, test, sync, and deploy (using a single tool that comes in a single binary).
Aral Balkan's avatar
Aral Balkan committed
7

8
__Site.js is a [small](https://small-tech.org/about#small-technology) personal web tool for Linux, macOS, and Windows 10.__
Aral Balkan's avatar
Aral Balkan committed
9

10
Most tools today are built for startups and enterprises. Site.js is built for people.
Aral Balkan's avatar
Aral Balkan committed
11

12
13
14
15
16
17
## Like this? Fund us!

[Small Technology Foundation](https://small-tech.org) is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share [our vision](https://small-tech.org/about/#small-technology) and want to support our work, please [become a patron or donate to us](https://small-tech.org/fund-us) today and help us continue to exist.

18
## Feature Highlights
Aral Balkan's avatar
Aral Balkan committed
19

20
  - __Just works.__ No configuration; [get started in seconds](https://sitejs.org/#get-started).
Aral Balkan's avatar
Aral Balkan committed
21

22
  - __Free as in freedom.__ And small as in [small tech](https://small-tech.org/about/#small-technology).
Aral Balkan's avatar
Aral Balkan committed
23

24
  - __Seamless single binary [install](#install)__ (thanks to [Nexe](https://github.com/nexe/nexe)).
Aral Balkan's avatar
Aral Balkan committed
25

26
  - __Secure by default.__
Aral Balkan's avatar
Aral Balkan committed
27

28
    __At localhost:__ Automatically provisions locally-trusted TLS for development (courtesy of [mkcert](https://github.com/FiloSottile/mkcert) seamlessly integrated via [Auto Encrypt Localhost](https://github.com/small-tech/auto-encrypt-localhost)).
Aral Balkan's avatar
Aral Balkan committed
29

30
    __And everywhere else:__ Automatically provisions globally-trusted TLS for staging and production (courtesy of [Let’s Encrypt](https://letsencrypt.org/) seamlessly integrated via [Auto Encrypt](https://github.com/small-tech/auto-encrypt) and [systemd](https://freedesktop.org/wiki/Software/systemd/)).
Aral Balkan's avatar
Aral Balkan committed
31

32
    Your server will score [an A+](https://www.ssllabs.com/ssltest/analyze.html?d=sitejs.org) on the [SSL Labs SSL Server Test](https://www.ssllabs.com/ssltest).
Aral Balkan's avatar
Aral Balkan committed
33

34
  - __Supports the creation of static web sites, dynamic web sites, and hybrid sites__ (via integrated [Node.js](https://nodejs.org/) and [Express](https://expressjs.com)).
Aral Balkan's avatar
Aral Balkan committed
35

36
  - __Includes a fast and simple database__ ([JSDB](https://github.com/small-tech/jsdb)).
37

38
  - __Supports [wildcard routes](#wildcard-routes)__ for purely client-side specialisation using path-based parameters.
39

40
  - __Supports [DotJS](#dotjs) for dynamic routes.__ (DotJS is PHP-like simple routing for Node.js introduced by Site.js for quickly prototyping and building dynamic sites).
Aral Balkan's avatar
Aral Balkan committed
41

42
  - __Includes [Hugo static site generator](#static-site-generation).__
43

Aral Balkan's avatar
Aral Balkan committed
44
45
  - __Includes [one command installation of production-ready Owncast server](#owncast-integration).__

46
  - __[Sync](#sync) to deploy__ (uses rsync for quick deployments). Can also [Live Sync](#live-sync) for live blogging, etc. For sites that implement the [Small Web](https://ar.al/2020/08/07/what-is-the-small-web/) conventions, you can also use the simplified [pull and push commands](#pull-and-push).
Aral Balkan's avatar
Aral Balkan committed
47

48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
  - __Has privacy-respecting [ephemeral statics](#ephemeral-statistics)__. Gives you insight into how your site is being used, not into the people using it.

  - __Supports [WebSockets](#websocket-wss-routes)__ (via integrated [express-ws](https://github.com/HenningM/express-ws), which itself wraps [ws](https://github.com/websockets/ws)).

  - __Can be used as a proxy server__ (via integrated [http-proxy-middleware](https://github.com/chimurai/http-proxy-middleware)).

  - __Supports [an evergreen web](#native-support-for-an-evergreen-web).__

    [The archival cascade](#the-archival-cascade) and [Native 404 → 302 support](#native-404--302-support) help you migrate and evolve your existing sites using Site.js without breaking existing links.

  - __Live reload__ on static pages.

  - __Automatic server reload__ when the source code of your dynamic routes change.

  - __Auto updates__ of production servers.
63
64

  <ins>Note:</ins> Production use via startup daemon is only supported on Linux distributions with systemd.
Aral Balkan's avatar
Aral Balkan committed
65

66
## Install
Aral Balkan's avatar
Aral Balkan committed
67

Aral Balkan's avatar
Aral Balkan committed
68
69
Copy and paste the following commands into your terminal:

Aral Balkan's avatar
Aral Balkan committed
70
__(Note: all commands should be run in your regular account, not as root.)__ (As of 15.4.0, Site.js will refuse to run if launched from the root account.)
71

72
### Native binaries
Aral Balkan's avatar
Aral Balkan committed
73

74
__Before you pipe any script into your computer, always view the source code ([Linux and macOS](https://should-i-pipe.it/https://sitejs.org/install), [Windows](https://should-i-pipe.it/https://sitejs.org/install.txt)) and make sure you understand what it does.__
75
76

#### Linux
Aral Balkan's avatar
Aral Balkan committed
77

78
```shell
Aral Balkan's avatar
Aral Balkan committed
79
wget -qO- https://sitejs.org/install | bash
Aral Balkan's avatar
Aral Balkan committed
80
81
```

82
83
(To use curl instead, see the macOS instructions, below.)

84
85
86
87
88
89
#### macOS

```shell
curl -s https://sitejs.org/install | bash
```

Aral Balkan's avatar
Aral Balkan committed
90
#### Windows 10 with PowerShell running under Windows Terminal
91
92

```shell
93
iex(iwr -UseBasicParsing https://sitejs.org/install.txt).Content
94
95
```

Aral Balkan's avatar
Aral Balkan committed
96
97
### Node.js

98
```shell
Aral Balkan's avatar
Aral Balkan committed
99
npm i -g @small-tech/site.js
Aral Balkan's avatar
Aral Balkan committed
100
101
```

102
103
### Alpha and Beta channels

Aral Balkan's avatar
Aral Balkan committed
104
On Linux and macOS, in addition to the release build channel, there is also an alpha build and beta build channel available. Pass either `alpha` or `beta` as an argument to the Bash pipe to install the latest build from the respective channel.
105
106
107
108
109
110
111

For example, to install the latest beta build on Linux:

```shell
wget -qO- https://sitejs.org/install | bash -s -- beta
```

112
113
114
115
116
117
__Note:__ On Macs, `wget` is not installed by default but `curl` is so you can use that instead:

```shell
curl -s https://sitejs.org/install | bash -s -- beta
```

118
119
120
121
Alpha builds are strictly for local testing and should not, under any circumstances, be used in production. We do not test Alpha builds in production.

Servers deployed using release builds check for updates every six hours whereas beta builds check every 10 minutes.

Aral Balkan's avatar
Aral Balkan committed
122
123
Note that the latest alpha or beta build available may be older than the latest release build. You can check the date on the build via the `version` command.

124
125
126
127
## System Requirements

### Linux

128
Any recent Linux distribution should work. However, Site.js is most thoroughly tested at Small Technology Foundation on Ubuntu 20.04/Pop!_OS 20.04 (development and staging) and Ubuntu 18.04 LTS (production).
129

130
There are builds available for x64, ARM, and ARM64.
131

132
For production use, [systemd](https://en.wikipedia.org/wiki/Systemd) is required.
133
134
135

### macOS

136
macOS 10.14.x Mojave and macOS 10.15.x Catalina are supported (the latter as of Site.js 12.5.1).
137
138
139
140
141
142
143

_Production use is not possible under macOS._

### Windows 10

The current version of Windows 10 is supported with PowerShell running under [Windows Terminal](https://github.com/Microsoft/Terminal).

Aral Balkan's avatar
Aral Balkan committed
144
__Windows Subsystem for Linux (WSL) is _not_ supported.__ (You can install and run Site.js under WSL but seamless TLS certificate handling for local servers will not work out of the box as WSL and Windows 10 do not share certificate stores. If you do want to use Site.js under WSL, you have to first install Site.js on Windows 10 and run a local server (`site`) to create the certificate authority and certificates, then install and run Site.js under WSL and then manually copy the contents of `~/.small-tech.org/site.js/tls/local/` from Windows 10 to WSL.)
145
146
147

_Production use is not possible under Windows._

148
149
## Dependencies

150
Site.js tries to seamlessly install the dependencies it needs when run. That said, there are certain basic components it expects on a Linux-like system. These are:
151
152

  - `sudo`
Aral Balkan's avatar
Aral Balkan committed
153
  - `bash` (on Linux, macOS, etc.) or `PowerShell` running under [Windows Terminal](https://github.com/Microsoft/Terminal) (on Windows 10).
154
  - `wget` or `curl` (on Linux and macOS) are required to download the installation script when installing Site.js using the one-line installation command. On Linux, you can install either via your distribution’s package manager (e.g., `sudo apt install wget` on Ubuntu-like systems). macOS comes with `curl` installed.
155

156
If it turns out that any of these prerequisites are a widespread cause of first-run woe, we can look into having them installed automatically in the future. Please [open an issue](https://github.com/small-tech/site.js/issues) if any of these affects you during your deployments or in everyday use.
157

Aral Balkan's avatar
Aral Balkan committed
158
159
160
161
### Automatically-installed dependencies

__For production use, passwordless sudo is required.__ On systems where the sudo configuration directory is set to `/etc/sudoers.d`, Site.js will automatically install this rule. On other systems, you might have to [set it up yourself](https://serverfault.com/questions/160581/how-to-setup-passwordless-sudo-on-linux).

Aral Balkan's avatar
Aral Balkan committed
162
__For localhost servers__, the bundled [mkcert](https://github.com/FiloSottile/mkcert) requires [certutil](https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/tools/NSS_Tools_certutil) and the [Network Security Services](https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS) (NSS) dynamic libraries. Site.js will attempt to automatically install the required libraries using popular package managers. <strike>Please note that this will fail on PinePhones running UBPorts as NSS is missing from the apt package manager for that distribution.</strike> ([The PinePhone issue](https://bugzilla.mozilla.org/show_bug.cgi?id=1652739) has been resolved.)
163

164
## Update (as of version 12.9.5; properly functioning as of version 12.9.6)
Aral Balkan's avatar
Aral Balkan committed
165
166
167
168
169
170
171
172
173

To seamlessly update the native binary if a newer version exists:

```shell
site update
```

This command will automatically restart a running Site.js daemon if one exists. If you are running Site.js as a regular process, it will continue to run and you will run the newer version the next time you launch a regular Site.js process.

174
175
176
177
__Note:__ There is a bug in the semantic version comparison in the original release with the update feature (version 12.9.5) that will prevent upgrades between minor versions (i.e., between 12.9.5 and 12.10.x and beyond). This was fixed in version 12.9.6. If you’re still on 12.9.5 and you’re reading this after we’ve moved to 12.10.0 and beyond, please stop Site.js if it’s running and [install the latest Site.js](#install) manually.

## Automatic updates in production (as of version 12.10.0)

178
[Production servers](#production) started with the `enable` command will automatically check for updates on first launch and then again at a set interval (currently every 6 hours) and update themselves as and when necessary.
179
180

This is a primary security feature given that Site.js is meant for use by individuals, not startups or enterprises with operations teams that can (in theory, at least) maintain servers with the latest updates.
181

182
183
184
185
186
## Uninstall

To uninstall the native binary (and any created artifacts, like TLS certificates, systemd services, etc.):

```shell
Aral Balkan's avatar
Aral Balkan committed
187
site uninstall
188
189
```

190
## Use
191

192
193
194
### Development (servers @localhost)

#### Regular server
195

196
Start serving the current directory at https://localhost as a regular process using locally-trusted certificates:
197
198

```shell
Aral Balkan's avatar
Aral Balkan committed
199
$ site
200
201
```

202
203
204
Note that if your current working directory is inside a special subfolder of your site (`.dynamic`, `.hugo`, `.wildcard`, `.db`) Site.js (as of version 15.4.0) magically does the right thing and serves the site root instead of the folder you’re in. If you really do want to serve one of these folders or a subfolder thereof, specifically state your intent by passing the current folder (`.`) as an argument.

The above caveat aside, the command above is a shorthand for the full form of the `serve` command:
205
206
207
208
209

```shell
$ site serve . @localhost:443
```

210
__Note:__ As of 15.4.0, Site.js will refuse to serve the root directory or your home directory for security reasons.
211

212
213
214
215
216
217
218
219
220
221
222
223
224
225
#### To serve on a different port

Just specify the port explicitly as in the following example:

```shell
$ site @localhost:666
```

That, again, is shorthand for the full version of the command, which is:

```shell
$ site serve . @localhost:666
```

226
227
228
229
230
231
232
233
234
235
236
237
238
239
#### Accessing your local server over the local area network

You can access local servers via their IPv4 address over a local area network.

This is useful when you want to test your site with different devices without having to expose your server over the Internet using a service like ngrok. For example, if your machine’s IPv4 address on the local area network is 192.168.2.42, you can just enter that IP to access it from, say, your iPhone.

To access your local machine from a different device on your local area network, you must transfer the public key of your generated local root certificate authority to that device and install and trust it.

For example, if you’re on an iPhone, hit the `/.ca` route in your browser:

```
http://192.168.2.42/.ca
```

240
The browser will download the local root certificate authority’s public key and prompt you to install the profile on your iPhone. You then have to go to Settings → Profile Downloaded → Tap Install when the Install Profile pop-up appears showing you the mkcert certificate you downloaded. Then, go to Settings → General → About → Certificate Trust Settings → Turn on the switch next to the mkcert certificate you downloaded. You should now be able to hit `https://192.168.2.42` and see your site from your iPhone.
241
242
243
244

You can also tranfer your key to your other devices manually. You can find the key at `~/.small-tech/site.js/tls/local/rootCA.pem` after you’ve created a local server at least once. For more details on transferring your key to other devices, please refer to [the relevant section in the mkcert documentation](https://github.com/FiloSottile/mkcert#mobile-devices).


245
#### Proxy server
246

247
You can use Site.js as a reverse proxy for HTTP and WebSocket connections. This is useful if you have a web app written in any language that only supports HTTP (not TLS) that you want to deploy securely.
248

249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
For example, the following is a simple HTTP server written in Python 3 (_server.py_) that runs insecurely on port 3000:

```python
from http.server import HTTPServer, BaseHTTPRequestHandler

class MyRequestHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.end_headers()
        self.wfile.write(b'Hello, from Python!')

server = HTTPServer(('localhost', 3000), MyRequestHandler)
server.serve_forever()
```

Run it (at http://localhost:3000) with:
265
266

```shell
267
$ python3 server
268
269
```

270
271
272
273
274
275
Then, proxy it securely from https://localhost using:

```shell
$ site :3000
```

276
277
278
Again, this is a convenient shortcut. The full form of this command is:

```shell
279
$ site serve :3000 @localhost:443
280
281
```

282
283
This will create and serve the following proxies:

284
285
  * http://localhost:3000 → https://localhost
  * ws://localhost:3000 → wss://localhost
286

287
288
289
290
291
292
293
294
295
296
### Testing (servers @hostname)

#### Regular server

Start serving the _my-site_ directory at your _hostname_ as a regular process using globally-trusted Let’s Encrypt certificates:

```shell
$ site my-site @hostname
```

297
Note that as of 13.0.0, Site.js will refuse to start the server if your hostname (or the domain you specified manually using the `--domain` option and any aliases you may have specified using the `--aliases` option) fails to resolve or is unreachable. This should help you diagnose and fix typos in domain names as well as DNS misconfiguration and propagation issues. As of 14.1.0, you can use the `--skip-domain-reachability-check` flag to override this behaviour and skip the pre-flight checks.
298

299
300
301
302
303
304
305
306
#### Proxy server

Start serving `http://localhost:1313` and `ws://localhost:1313` at your _hostname_:

```shell
$ site :1313 @hostname
```

307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
#### macOS notes

To set your hostname under macOS (e.g., to `example.small-tech.org`), run the following command:

```shell
$ sudo scutil --set HostName example.small-tech.org
```

#### Windows 10 notes

On Windows 10, you must add quotation marks around `@hostname` and `@localhost`. So the first example, above, would be written in the following way on Windows 10:

```shell
$ site my-site "@hostname"
```

323
Also, Windows 10, unlike Linux and macOS, does not have the concept of a hostname. The closest thing to it is your _full computer name_. Setting your full computer name is a somewhat convoluted process so we’ve documented it here for you.
324
325
326
327
328
329
330
331
332
333

##### How to set your full computer name on Windows 10

Say you want to set your hostname to `my-windows-laptop.small-tech.org`:

1. Control Panel → System And Security → System → Change Settings link (next to Computer name) → [Change…] Button
2. Under Computer name, enter your _subdomain_ (`my-windows-laptop`)
3. [More…] Button → enter your _domain name_ (`small-tech.org`) in the Primary DNS suffix of this computer field.
4. Press the various [OK] buttons to dismiss the various modal dialogues and restart your computer.

334
335
#### Making your server public

336
Use a service like [ngrok](https://ngrok.com/) (Pro+) to point a custom domain name to your temporary staging server. Make sure you set your `hostname` file (e.g., in `/etc/hostname` or via `hostnamectl set-hostname <hostname>` or the equivalent for your platform) to match your domain name. The first time you hit your server via your hostname it will take a little longer to load as your Let’s Encrypt certificates are being automatically provisioned by Auto Encrypt.
Aral Balkan's avatar
Aral Balkan committed
337

338
339
When you start your server, it will run as a regular process. It will not be restarted if it crashes or if you exit the foreground process or restart the computer.

Aral Balkan's avatar
Aral Balkan committed
340
341
### Deployment

342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
#### Pull and push

As of version 14.4.0, you can use the simplified `pull` and `push` commands if your local and remote setup adheres to the following Small Web conventions:

##### Local

  - The name of your local working folder is the same as your domain (if not, specify the domain using the `--domain` oiption)
  - Your SSH key is either found at `~/.ssh/id_{your domain}_ed25519` or you have an _id_25519_ or _id_rsa_ file in your `~/.ssh` folder. (The former is a Small Web convention, the latter is a fallback general convention.)

##### Remote

  - __Account name:__ `site`
  - __Folder being served:__ `/home/site/public`

If those requirements are met, from within your site’s folder on your local machine, you can pull (download) your site using:

```shell
site pull
```

And you can push (deploy) your site using:

```shell
site push
```

The legacy `sync` command will continue to work as before and is documented below.

Aral Balkan's avatar
Aral Balkan committed
370
#### Sync
371

Aral Balkan's avatar
Aral Balkan committed
372
Site.js can help you deploy your site to your live server with its sync feature.
Aral Balkan's avatar
Aral Balkan committed
373
374

```shell
Aral Balkan's avatar
Aral Balkan committed
375
$ site my-demo --sync-to=my-demo.site
Aral Balkan's avatar
Aral Balkan committed
376
377
```

Aral Balkan's avatar
Aral Balkan committed
378
The above command will:
Aral Balkan's avatar
Aral Balkan committed
379

Aral Balkan's avatar
Aral Balkan committed
380
381
  1. Generate any Hugo content that might need to be generated.
  2. Sync your site from the local _my-demo_ folder via rsync over ssh to the host _my-demo.site_.
Aral Balkan's avatar
Aral Balkan committed
382

383
Without any customisations, the sync feature assumes that your account on your remote server has the same name as your account on your local machine and that the folder you are watching (_my-demo_, in the example above) is located at _/home/your-account/my-demo_ on the remote server. Also, by default, the contents of the folder will be synced, not the folder itself. You can change these defaults by specifying a fully-qualified remote connection string as the `--sync-to` value.
Aral Balkan's avatar
Aral Balkan committed
384

Aral Balkan's avatar
Aral Balkan committed
385
The remote connection string has the format:
Aral Balkan's avatar
Aral Balkan committed
386

Aral Balkan's avatar
Aral Balkan committed
387
388
```
remoteAccount@host:/absolute/path/to/remoteFolder
Aral Balkan's avatar
Aral Balkan committed
389
390
```

Aral Balkan's avatar
Aral Balkan committed
391
For example:
Aral Balkan's avatar
Aral Balkan committed
392

Aral Balkan's avatar
Aral Balkan committed
393
394
395
396
```shell
$ site my-folder --sync-to=someOtherAccount@my-demo.site:/var/www
```

Aral Balkan's avatar
Aral Balkan committed
397
If you want to sync not the folder’s contents but the folder itself, use the `--sync-folder-and-contents` flag. e.g.,
Aral Balkan's avatar
Aral Balkan committed
398

Aral Balkan's avatar
Aral Balkan committed
399
```shell
Aral Balkan's avatar
Aral Balkan committed
400
$ site my-local-folder --sync-to=me@my.site:my-remote-folder --sync-folder-and-contents
Aral Balkan's avatar
Aral Balkan committed
401
402
```

Aral Balkan's avatar
Aral Balkan committed
403
The above command will result in the following directory structure on the remote server: _/home/me/my-remote-folder/my-local-folder_. It also demonstrates that if you specify a relative folder, Site.js assumes you mean the folder exists in the home directory of the account on the remote server.
Aral Balkan's avatar
Aral Balkan committed
404

405
406
(As of 15.4.0) If the sync command cannot connect in 5 seconds, it will time out. If this happens, check that you have the correct host and account details specified. If you do, there might be a problem with your connection.

407
408
(As of 16.1.0) It’s a common mistake to start the sync without specifying the `--sync-from` option when not in the root of your site but in one of the well-known subfolders (e.g., `.hugo` if you’re working on a Hugo site or your `.dynamic` folder if you happen to be in it because you’re working on a site that uses DotJS.) In these instances, Site.js will detect the mistake and understand that you want to sync the site, not the subfolder and behave accordingly. If you _really_ want to sync one of the well-known subfolders for some reason, then specifically specify it by setting `--sync-from=.`. Note that this magic rewriting of the sync path doesn’t happen any time you specify a folder explicitly using the `--sync-from` option.

Aral Balkan's avatar
Aral Balkan committed
409
410
411
412
413
414
415
#### Live Sync

With the Live Sync feature, you can have Site.js watch for changes to your content and sync them to your server in real-time (e.g., if you want to live blog something or want to keep a page updated with local data you’re collecting from a sensor).

To start a live sync server, provide the `--live-sync` flag to your sync request.

For example:
416
417

```shell
Aral Balkan's avatar
Aral Balkan committed
418
$ site my-demo --sync-to=my-demo.site --live-sync
419
420
```

Aral Balkan's avatar
Aral Balkan committed
421
422
The above command will start a local development server at _https://localhost_. Additionally, it will watch the folder _my-demo_ for changes and sync any changes to its contents via rsync over ssh to the host _my-demo.site_.

Aral Balkan's avatar
Aral Balkan committed
423

424
### Production
425

426
__Available on Linux distributions with systemd (most Linux distributions, but [not these ones](https://sysdfree.wordpress.com/2019/03/09/135/) or on macOS or Windows).__
427

428
429
__For production use, passwordless sudo is required.__ On systems where the sudo configuration directory is set to `/etc/sudoers.d`, Site.js will automatically install this rule. On other systems, you might have to [set it up yourself](https://serverfault.com/questions/160581/how-to-setup-passwordless-sudo-on-linux).

Aral Balkan's avatar
Aral Balkan committed
430
__Please make sure that you are NOT running as root.__ (As of 15.4.0, Site.js will refuse to run if launched from the root account.)
431

432
On your live, public server, you can start serving the _my-site_ directory at your _hostname_ as a daemon that is automatically run at system startup and restarted if it crashes with:
433
434

```shell
Aral Balkan's avatar
Aral Balkan committed
435
$ site enable my-site
436
437
```

438
The `enable` command sets up your server to start automatically when your server starts and restart automatically if it crashes.
439

440
For example, if you run the command on a connected server that has the ar.al domain pointing to it and `ar.al` set in _/etc/hostname_, you will be able to access the site at https://ar.al. (Yes, of course, [ar.al](https://ar.al) runs on Site.js.) The first time you hit your live site, it will take a little longer to load as your Let’s Encrypt certificates are being automatically provisioned by Auto Encrypt.
441

442
By default, the automatic TLS certificate provisioning gets certificates for your naked domain only, which it bases on your hostname.
443

444
445
446
447
448
If you want to serve your site at a domain that’s different to your hostname, specify it using the `--domain` option.

If you also want certificates for the _www_ subdomain, specify it using the `--aliases` option. You can specify multiple subdomains to provision certificates for by separating them using commas (without spaces).

__Note:__ As of 13.0.0, the `enable` will run pre-flight checks and refuse to install the service if the domain name and any aliases you have specified are not reachable. As of 14.1.0, you can use the `--skip-domain-reachability-check` flag to override this behaviour and skip the pre-flight checks. If you use this flag, the server launched by the installed service will also not check for reachability. This is useful if you want to set up a server via a script prior to DNS propagation. Just make sure you haven’t made any typos in any of the domain names as you will not be warned about any mistakes.
449

450
451
452
453
454
455
__Note:__ As of 16.2.0, you can now also run proxy servers in production. To proxy whatever is running over HTTP and WS on port 8080 at https://your.domain, do:

```shell
$ site enable :8080
```

456
When the server is enabled, you can also use the following commands:
Aral Balkan's avatar
Aral Balkan committed
457

Aral Balkan's avatar
Aral Balkan committed
458
  - `start`: Start server.
459
  - `stop`: Stop server.
Aral Balkan's avatar
Aral Balkan committed
460
  - `restart`: Restart server.
461
  - `disable`: Stop server and remove from startup.
462
463
  - `logs`: Display and tail server logs (press _Ctrl+C_ to exit).
  - `status`: Display detailed server information.
Aral Balkan's avatar
Aral Balkan committed
464

465
Site.js uses the [systemd](https://freedesktop.org/wiki/Software/systemd/) to start and manage the daemon. Beyond the commands listed above that Site.js supports natively (and proxies to systemd), you can make use of all systemd functionality via the [systemctl](https://www.freedesktop.org/software/systemd/man/systemctl.html) and [journalctl](https://www.freedesktop.org/software/systemd/man/journalctl.html) commands.
Aral Balkan's avatar
Aral Balkan committed
466

467
468
## Build and test from source

469
470
471
Site.js is built using and supports Node.js 12 LTS (currently version 12.16.2).

It has also been tested to work with the latest LTS (14.x).
Aral Balkan's avatar
Aral Balkan committed
472
473

The build is created using Nexe and our own pre-built Nexe base Node.js binaries hosted on SiteJS.org. Please make sure that the version of your Node.js runtime matches the currently supported version stated above to ensure that the correct Nexe binary build is downloaded and used by the build script.
Aral Balkan's avatar
Aral Balkan committed
474

Aral Balkan's avatar
Aral Balkan committed
475
### Install the source and run tests
Aral Balkan's avatar
Aral Balkan committed
476

477
478
```shell
# Clone and install.
Aral Balkan's avatar
Aral Balkan committed
479
mkdir site.js && cd site.js
480
git clone https://github.com/small-tech/site.js.git app
Aral Balkan's avatar
Aral Balkan committed
481
482
cd app
./install
483

484
485
486
# Make sure your computer is reachable from your
# hostname if you’re going to run the tests.
# (e.g., using PageKit or ngrok, etc.)
487

488
# Run tests.
489
npm test
Aral Balkan's avatar
Aral Balkan committed
490
```
491

492
493
494
495
496
497
498
Note that if you have a large amount of logs, the logs tests might fail due to a timeout. In this case, try clearing your journalctl logs:

```
journalctl --rotate
journalctl --vacuum-time=1s
```

Aral Balkan's avatar
Aral Balkan committed
499
500
501
502
503
504
505
506
507
508
### Install as global Node.js module

After you install the source and run tests:

```shell
# Install the binary as a global module
npm i -g

# Serve the test site locally (visit https://localhost to view).
site test/site
509
510
```

Aral Balkan's avatar
Aral Balkan committed
511
__Note:__ for commands that require root privileges (i.e., `enable` and `disable`), Site.js will automatically restart itself using sudo and Node must be available for the root account. If you’re using [nvm](https://github.com/creationix/nvm), you can enable this via:
512
513

```shell
514
# Replace v10.16.3 with the version of node you want to make available globally.
Aral Balkan's avatar
Aral Balkan committed
515
516
sudo ln -s "$NVM_DIR/versions/node/v12.16.2/bin/node" "/usr/local/bin/node"
sudo ln -s "$NVM_DIR/versions/node/v12.16.2/bin/npm" "/usr/local/bin/npm"
517
518
```

519
520
521
522
523
524
525
526
If you forget to do this and run `site enable`, you will find the following error in the systemctl logs: `/etc/systemd/system/site.js.service:15: Executable "node" not found in path`. The command itself will fail with:

```
Error: Command failed: sudo systemctl start site.js
Failed to start site.js.service: Unit site.js.service has a bad unit file setting.
See system logs and 'systemctl status site.js.service' for details.
```

Aral Balkan's avatar
Aral Balkan committed
527
528
### Native binaries

Aral Balkan's avatar
Aral Balkan committed
529
After you install the source and run tests:
Aral Balkan's avatar
Aral Balkan committed
530

Aral Balkan's avatar
Aral Balkan committed
531
```shell
532
533
# Build the native binary for your platform.
# To build for all platforms, use npm run build -- --all
Aral Balkan's avatar
Aral Balkan committed
534
535
536
npm run build

# Serve the test site (visit https://localhost to view).
Aral Balkan's avatar
Aral Balkan committed
537
538
539
# e.g., Using the Linux binary with version <binary-version>
# in the format (YYYYMMDDHHmmss).
dist/linux/<binary-version>/site test/site
540
541
```

542
543
### Build and install native binary locally

Aral Balkan's avatar
Aral Balkan committed
544
545
After you install the source and run tests:

546
547
548
549
```shell
npm run install-locally
```

550
551
552
553
554
555
556
557
558
559
560
### Update the Nexe base binary for your platform/architecture and Node version

(You will most likely not need to do this.)

```shell
npm run update-nexe
```

### Deploying Site.js itself

(You will most likely not need to do this.)
561
562

```shell
563
564
# To cross-compile binaries for Linux (x64), macOS, and Windows
# and also copy them over to the Site.js web Site for deployment.
565
npm run deploy
Aral Balkan's avatar
Aral Balkan committed
566
567
```

568
569
570
571
572
573
574
575
576
Note that the deployment script requires a modern version of git to work (e.g., will not work on elementary OS 5.1.7 which is based on Ubuntu 18.04.) To install the latest version of git on Ubuntu-esque operating systems (like elementary OS):

```
sudo apt install software-properties-common
sudo add-apt-repository ppa:git-core/ppa
sudo apt update
sudo apt install git
```

577
## Syntax
578

Aral Balkan's avatar
Aral Balkan committed
579
```shell
580
site [command] [folder|:port] [@host[:port]] [--options]
Aral Balkan's avatar
Aral Balkan committed
581
```
582

583
  - `command`: serve | enable | disable | start | stop | logs | status | update | uninstall | version | help
584
585
586
587
588
  - `folder|:port`: Path of folder to serve (defaults to current folder) or port on localhost to proxy.
  - `@host[:port]`: Host (and, optionally port) to sync. Valid hosts are @localhost and @hostname.
  - `--options`: Settings that alter command behaviour.

__Key:__ `[]` = optional &nbsp;&nbsp;`|` = or
Aral Balkan's avatar
Aral Balkan committed
589

590
591
### Commands:

592
593
594
595
596
597
598
599
600
601
602
603
604
  - `serve`: Serve specified folder (or proxy specified `:port`) on specified `@host` (at `:port`, if given). The order of arguments is:

    1. what to serve,
    2. where to serve it at. e.g.,

    ```site serve my-folder @localhost```

    If a port (e.g., `:1313`) is specified instead of my-folder, start an HTTP/WebSocket proxy.

  - `enable`: Start server as daemon with globally-trusted certificates and add to startup.

  - `disable`: Stop server daemon and remove from startup.

605
606
607
608
  - `start`: Start server as daemon with globally-trusted certificates.

  - `stop`: Stop server daemon.

Aral Balkan's avatar
Aral Balkan committed
609
610
  - `restart`: Restart server daemon.

611
  - `logs`: Display and tail server logs.
Aral Balkan's avatar
Aral Balkan committed
612

613
614
  - `status`: Display detailed server information.

615
616
617
618
619
620
  - `update`: Check for Site.js updates and update if new version is found.
  - `uninstall`: Uninstall Site.js.

  - `version`: Display version and exit.
  - `help`: Display help screen and exit.

621
If `command` is omitted, behaviour defaults to `serve`.
622
623

### Options:
Aral Balkan's avatar
Aral Balkan committed
624

Aral Balkan's avatar
Aral Balkan committed
625
626
#### For both the `serve` and `enable` commands:

627
628
  - `--domain`: The main domain to serve (defaults to system hostname if not specified).

629
  - `--aliases`: Comma-separated list of additional domains to obtain TLS certificates for and respond to. These domains point to the main domain via a 302 redirect. Note that as of 13.0.0, the _www_ alias is not added automatically. To specify it, you can use the shorthand form:`--aliases=www`
Aral Balkan's avatar
Aral Balkan committed
630

631
632
  - `--skip-domain-reachability-check`:	Do not run pre-flight check for domain reachability.

633
634
635
636
  - `--access-log-errors-only`: Display only errors in the access log (HTTP status codes _4xx_ and _5xx_). Successful access requests (_1xx_, _2xx_, and _3xx_) are not logged. This is useful during development if you feel overwhelmed by the output and miss other, non-access-related errors.

  - `--access-log-disable`: Completely disable the access log. No access requests, _not even errors_ will be logged. Be careful when using this in production as you might miss important errors.

637
#### For the `serve` command:
Aral Balkan's avatar
Aral Balkan committed
638

639
  - `--sync-to`: The host to sync to.
Aral Balkan's avatar
Aral Balkan committed
640

641
  - `--sync-from`: The folder to sync from (only relevant if `--sync-to` is specified).
Aral Balkan's avatar
Aral Balkan committed
642

Aral Balkan's avatar
Aral Balkan committed
643
  - `--live-sync`: Watch for changes and live sync them to a remote server (only relevant if `--sync-to` is specified).
Aral Balkan's avatar
Aral Balkan committed
644

645
  - `--sync-folder-and-contents`: Sync folder and contents (default is to sync the folder’s contents only).
Aral Balkan's avatar
Aral Balkan committed
646

647
648
649
#### For the `enable` command:

  - `--ensure-can-sync`: Ensure server can rsync via ssh.
Aral Balkan's avatar
Aral Balkan committed
650

651
All command-line arguments are optional. By default, Site.js will serve your current working folder over port 443 with locally-trusted certificates.
652

653
When you `serve` a site at `@hostname` or use the `enable` command, globally-trusted Let’s Encrypt TLS certificates are automatically provisioned for you using Auto Encrypt the first time you hit your hostname. The hostname for the certificates is automatically set from the hostname of your system (and the _www._ subdomain is also automatically provisioned).
654

Aral Balkan's avatar
Aral Balkan committed
655
656
## Usage examples

657
### Develop using locally-trusted TLS certificates
Aral Balkan's avatar
Aral Balkan committed
658
659
660

| Goal                                      | Command                                                       |
| ----------------------------------------- | ------------------------------------------------------------- |
661
662
663
664
665
666
667
668
669
| Serve current folder*                     | site                                                          |
|                                           | site serve                                                    |
|                                           | site serve .                                                  |
|                                           | site serve . @localhost                                       |
|                                           | site serve . @localhost:443                                   |
| Serve folder demo (shorthand)             | site demo                                                     |
| Serve folder demo on port 666             | site serve demo @localhost:666                                |
| Proxy localhost:1313 to https://localhost*| site :1313                                                    |
|                                           | site serve :1313 @localhost:443                               |
Aral Balkan's avatar
Aral Balkan committed
670
671
672
673
674
| Sync demo folder to my.site               | site demo --sync-to=my.site                                   |
| Ditto, but use account me on my.site      | site demo --sync-to=me@my.site                                |
| Ditto, but sync to remote folder ~/www    | site demo --sync-to=me@my.site:www                            |
| Ditto, but specify absolute path          | site demo --sync-to=me@my.site:/home/me/www                   |
| Live sync current folder to my.site       | site --sync-to=my.site --live-sync                            |
675
676
677
678

### Stage and deploy using globally-trusted Let’s Encrypt certificates

#### Regular process:
Aral Balkan's avatar
Aral Balkan committed
679
680
681

| Goal                                      | Command                                                       |
| ----------------------------------------- | ------------------------------------------------------------- |
682
| Serve current folder                      | site @hostname                                                |
683
| Serve current folder at specified domain  | site @hostname --domain=my.site                               |
684
| Serve current folder also at aliases	    | site @hostname --aliases=www,other.site,www.other.site        |
685
686
687
688
689
690
| Serve folder demo*                        | site demo @hostname                                           |
|                                           | site serve demo @hostname                                     |
| Proxy localhost:1313 to https://hostname  | site serve :1313 @hostname                                    |

#### Start-up daemon:

691
692
| Goal                                      | Command                                                       |
| ----------------------------------------- | ------------------------------------------------------------- |
693
| Install and serve current folder as daemon| site enable                                                   |
694
| Ditto & also ensure it can rsync via ssh  | site enable --ensure-can-sync                                 |
Aral Balkan's avatar
Aral Balkan committed
695
| Get status of daemon                      | site status                                                   |
Aral Balkan's avatar
Aral Balkan committed
696
697
698
| Start server                              | site start                                                    |
| Stop server                               | site stop                                                     |
| Restart server                            | site restart                                                  |
Aral Balkan's avatar
Aral Balkan committed
699
| Display server logs                       | site logs                                                     |
700
| Stop and uninstall current daemon         | site disable                                                  |
Aral Balkan's avatar
Aral Balkan committed
701

Aral Balkan's avatar
Aral Balkan committed
702
703
704
705
706
707
#### General:

| Goal                                      | Command                                                       |
| ----------------------------------------- | ------------------------------------------------------------- |
| Check for updates and update if found     | site update                                                   |

708
\* _Alternative, equivalent forms listed (some commands have shorthands)._
709

710
## Native support for an Evergreen Web
711

Aral Balkan's avatar
Aral Balkan committed
712
What if links never died? What if we never broke the Web? What if it didn’t involve any extra work? It’s possible. And, with Site.js, it’s effortless.
713

714
### The Archival Cascade
715

716
__(As of version 13.0.0)__ If you have static archives of previous versions of your site, you can have Site.js automatically serve them for you.
717
718
719
720
721
722
723

Just put them into folder named `.archive-1`, `.archive-2`, etc.

If a path cannot be found in your current site, Site.js will search for it first in `.archive-2` and, if it cannot find it there either, in `.archive-1`.

Paths in your current site will override those in `.archive-2` and those in `.archive-2` will, similarly, override those in `.archive-1`.

Timo Tijhof's avatar
Timo Tijhof committed
724
Use the archival  old links will never die but if you do replace them with newer content in newer versions, those will take precedence.
725

726
#### Legacy method (pre version 13.0.0)
727
728

In older versions, the convention for specifying the archival cascade was as follows:
729
730
731
732

```
|- my-site
|- my-site-archive-1
733
734
|- my-site-archive-2
|- etc.
735
736
```

737
This legacy method of specifying the archival cascade is still supported but may be removed in a future release. Please use the recommended method outlined above instead.
738
739
740
741

### Native 404 → 302 support

But what if the previous version of your site is a dynamic site and you either don’t want to lose the dynamic functionality or you simply cannot take a static backup. No worries. Just move it to a different subdomain or domain and make your 404s into 302s.
742

Aral Balkan's avatar
Aral Balkan committed
743
Site.js has native support for [the 404 to 302 technique](https://4042302.org) to ensure an evergreen web. Just serve the old version of your site (e.g., your WordPress site, etc.) from a different subdomain and tell Site.js to forward any unknown requests on your new static site to that subdomain so that all your existing links magically work.
744
745
746

To do so, create a simple file called `4042302` in the root directory of your web content and add the URL of the server that is hosting your older content. e.g.,

747
### /4042302
748
```
Aral Balkan's avatar
Aral Balkan committed
749
https://the-previous-version-of.my.site
750
```
751
752
753
754
755
756

You can chain the 404 → 302 method any number of times to ensure that none of your links ever break without expending any additional effort to migrate your content.

For more information and examples, see [4042302.org](https://4042302.org).

## Custom error pages
757

758
759
![Screenshot of the custom 404 error page included in the unit tests](images/custom-404.png)

760
761
### Custom static 404 and 500 error pages

762
763
You can specify a custom error page for 404 (not found) and 500 (internal server error) errors. To do so, create a folder with the status code you want off of the root of your web content (i.e., `/404` and/or `/500`) and place at least an `index.html` file in the folder. You can also, optionally, put any assets you want to display on your error pages into those folders and load them in via relative URLs. Your custom error pages will be served with the proper error code and at the URL that was being accessed.

764
765
766
767
768
769
770
771
772
773
774
775
If you want to display the path that could not be found in your custom 404 page, use the following template placeholder somewhere on your page and it will be automatically substituted:

```
THE_PATH
```

e.g., The example from [the test site](https://github.com/small-tech/site.js/blob/master/test/site/404/index.html) shown in the screenshot uses the following code:

```html
<p><strong>Sorry, I can’t find</strong> THE_PATH</p>
```

776
777
### Custom Hugo 404 error page

778
As of version 15.4.0, if your site uses the Hugo static site generator, you can create [a custom Hugo 404 error page](https://gohugo.io/templates/404/).
779
780
781
782
783
784
785

Put a `404.html` page in your `layouts/` folder so that it gets created in your `.generated` folder when the site is built and it will be used instead of the default 404 page.

__Note:__ If you have both a custom static 404 page (defined at `/404/index.html`) and a custom Hugo 404 page, the Hugo 404 page will take precedence.

## Default 404 and 500 error pages

786
787
If you do not create custom error pages, the built-in default error pages will be displayed for 404 and 500 errors.

788
When creating your own servers (see [API](#API)), you can generate the default error pages programmatically using the static methods `Site.default404ErrorPage()` and `Site.default500ErrorPage()`, passing in the missing path and the error message as the argument, respectively to get the HTML string of the error page returned.
789

790
791
792
793
794
795
796
797
798
799
## Ephemeral statistics

When Site.js launches, you will see a line similar to the following in the console:

```
📊    ❨site.js❩ For statistics, see https://localhost/b64bd821d521b6a65a307c2b83060766
```

This is your private, cryptographically secure URL where you can access ephemeral statistics about your site. If you want to share your statistics, link to them publicly. If you want to keep them private, keep the URL secret.

800
__Note:__ As of version 15.4.0, you can remind yourself of the statistics URL while running the Site.js daemon in production using the `site status` command while the server is active.
801

802
803
804
805
806
807
![Screenshot of the statistics page](/images/statistics.png)

The statistics are ephemeral as they are only kept in memory and they reset any time your server restarts.

The statistics are very basic and they’re there only to give an idea about which parts of your site are most popular as well as to highlight missing pages, etc., They’re not there so you can spy on people (if you want to do that, this is not the tool for you).

808
809
## Static site generation

810
As of version 13.0.0, Site.js includes the [Hugo static site generator](https://gohugo.io).
811
812
813
814

To create a new Hugo site and start serving it:

```shell
815
# Create a folder to hold your site and switch to it.
816
mkdir my-site
817
818
819
820
821
822
823
824
825
826
cd my-site

# Generate empty Hugo site.
site hugo new site .hugo

# Create the most basic layout template possible.
echo 'Hello, world!' > .hugo/layouts/index.html

# Start Site.js
site
827
828
```

829
830
831
832
833
834
When you hit _https://localhost_, you should see the ‘Hello, world!’ page.

This basic example doesn’t take advantage of any of the features that you’d want to use Hugo for (like markdown authoring, list page creation, etc.). For a slightly more advanced one that does, see the [Basic Hugo Blog example](https://github.com/small-tech/site.js/tree/master/examples/basic-hugo-blog).

Of course, if you already know how Hugo works, just [download a theme](https://themes.gohugo.io/) and [set up your configuration](https://gohugo.io/getting-started/configuration/,) and you’ll be up and running in no time. Everything in your _.hugo_ folder works exactly as it does in any other Hugo site.

835
836
837
838
839
840
__Note:__ During development, this feature uses Site.js’s live reload instead of Hugo’s. Your web page must have at least a `<body>` tag for it to work.

### How it works

If Site.js finds a folder called _.hugo_ in your site’s root, it will build it using its integrated Hugo instance (you don’t need to install Hugo separately) and place the generated files into a folder called _.generated_ in your site’s root. It will also automatically serve these files.

841
842
One difference with plain Hugo is that if you set a `baseURL` in your configuration, it will be ignored as Site.js sets the `baseURL` automatically to the correct value based on whether you are running locally in development or at your hostname during staging or production.

843
844
__Note:__ You should add `.generated` to your `.gitignore` file so as not to accidentally add the generated content into your source code repository.

845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
You can pass any command you would normally pass to Hugo using Site.js’s integrated Hugo instance:

```shell
site hugo [any valid Hugo command]
```

Please see [the Hugo documentation](https://gohugo.io/documentation/) for detailed information on how Hugo works.

### Mounting Hugo sites

Site.js will automatically mount files in the _.hugo_ directory at your site’s root.

If you want the generated Hugo site to be mounted at a different path, include the path structure you want in the name of the hugo folder, separating paths using two dashes. For example:

Folder name               | Mount path         |
------------------------- | ------------------ |
.hugo                     | /                  |
.hugo--docs               | /docs              |
.hugo--second-level--blog | /second-level/blog |

You can include any number of Hugo sites in your site and mount them at different paths and the results will be weaved together into the _.generated_ folder. We call this feature… _ahem_… Hugo Weaving (we’ll show ourselves out).

All regular Site.js functionality is still available when using Hugo generation. So you can, for example, have your blog statically-generated using Hugo and extend it using locally-hosted dynamic comments.

__Note:__ Hugo’s [Multilingual Multihost mode](https://gohugo.io/content-management/multilingual/#configure-multilingual-multihost) is _not_ supported.

Aral Balkan's avatar
Aral Balkan committed
871
872
## Dynamic sites

873
You can specify routes with dynamic functionality by specifying HTTPS and WebSocket (WSS) routes in two ways: either using DotJS – a simple file system routing convention ala PHP, but for JavaScript – or through code in a _routes.js_ file.
Aral Balkan's avatar
Aral Balkan committed
874
875
876

In either case, your dynamic routes go into a directory named _.dynamic_ in the root of your site.

877
878
### DotJS

879
DotJS maps JavaScript modules defined in `.js` and `.cjs` files within a file system hierarchy to routes on your web site in a manner that will be familiar to anyone who has ever used PHP.
Aral Balkan's avatar
Aral Balkan committed
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898

#### GET-only (simplest approach)

The easiest way to get started with dynamic routes is to simply create a JavaScript file in a folder called _.dynamic_ in the root folder of your site. Any routes added in this manner will be served via HTTPS GET.

For example, to have a dynamic route at `https://localhost`, create the following file:

```
.dynamic/
    └ index.js
```

Inside _index.js_, all you need to do is to export your route handler:

```js
let counter = 0

module.exports = (request, response) => {
  response
899
    .html(`
Aral Balkan's avatar
Aral Balkan committed
900
901
902
903
904
905
906
907
      <h1>Hello, world!</h1>
      <p>I’ve been called ${++counter} time${counter > 1 ? 's': ''} since the server started.</p>
    `)
}
```

To test it, run a local server (`site`) and go to `https://localhost`. Refresh the page a couple of times to see the counter increase.

908
Congratulations, you’ve just made your first dynamic route using DotJS.
909

Aral Balkan's avatar
Aral Balkan committed
910
911
912
In the above example, _index.js_ is special in that the file name is ignored and the directory that the file is in becomes the name of the route. In this case, since we put it in the root of our site, the route becomes `/`.

Usually, you will have more than just the index route (or your index route might be a static one). In those cases, you can either use directories with _index.js_ files in them to name and organise your routes or you can use the names of _.js_ files themselves as the route names. Either method is fine but you should choose one and stick to it in order not to confuse yourself later on (see [Precedence](#Precendence), below).
913

914
So, for example, if you wanted to have a dynamic route that showed the server CPU load and free memory, you could create a file called _.dynamic/server-stats.js_ in your web folder with the following content:
915
916
917
918

```js
const os = require('os')

Aral Balkan's avatar
Aral Balkan committed
919
function serverStats (request, response) {
920
921
922
923
924
925
926

  const loadAverages = `<p> ${os.loadavg().reduce((a, c, i) => `${a}\n<li><strong>CPU ${i+1}:</strong> ${c}</li>`, '<ul>') + '</ul>'}</p>`

  const freeMemory = `<p>${os.freemem()} bytes</p>`

  const page = `<html><head><title>Server statistics</title><style>body {font-family: sans-serif;}</style></head><body><h1>Server statistics</h1><h2>Load averages</h2>${loadAverages}<h2>Free memory</h2>${freeMemory}</body></html>`

927
  response.html(page)
928
929
930
931
932
}

module.exports = serverStats
```

Aral Balkan's avatar
Aral Balkan committed
933
Site.js will load your dynamic route at startup and you can test it by hitting _https://localhost/server-stats_ using a local web server. Each time you refresh, you should get the latest dynamic content.
934

Aral Balkan's avatar
Aral Balkan committed
935
936
__Note:__ You could also have named your route _.dynamic/server-stats/index.js_ and still hit it from _https://localhost/server-stats_. It’s best to keep to one or other convention (either using file names as route names or directory names as route names). Using both in the same app will probably confuse you (see [Precedence](#Precendence), below).

937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
##### Specifying parameters

Your DotJS routes can also define named parameters that will be passed to your routes when they are triggered.

To specify a named parameter, separate it from the rest of the route name using an underscore (`_`). At use, named parameters are provided to the route via the request path and are made available in the route callback as properties on the `request.params` object.

For example, to have a route that greets people by their first name, create a file called:

```
.dynamic/hello_name.js
```

And add the following content:

```js
module.exports = (request, response) => {
  response.html(`<h1>Hello, ${request.params.name}!</h1>`)
}
```

Now run a local server (`site`) and hit `https://localhost/hello/Laura` to see `Hello, Laura!` in the browser.

You can also specify static path fragments that must be included verbatim in between parameters. You do this by using two underscores (`__`) instead of one.

For example, to have a route that returns the author ID and book ID that it is passed in a JSON structure, create a file called:

```
.dynamic/author/index_authorId__book_bookId.js
```

(Note: you can also call it `.dynamic/author_authorId__book_bookId.js`. Just make sure you pick one convention and stick to it so you don’t confuse yourself later on.)

Then, add the following content to it:

```js
module.exports = (request, response) => {
  response.json({
    authorId: request.params.authorId,
    bookId: request.params.bookId
  })
}
```

Now run a local server (`site`) and hit:

```
https://localhost/author/philip-pullman/book/his-dark-materials
```

To see the following JSON object returned:

```json
{
  "authorId": "philip-pullman",
  "bookId": "his-dark-materials"
}
```

DotJS parameters save you from having to use [advanced routing](#advanced-routing-routesjs-file) if all you want are named parameters for your routes. The only time you should have to use the latter is if you want to use regular expressions in your route definitions.

997
##### Using node modules
998

999
1000
1001
Since Site.js contains Node.js, anything you can do with Node.js, you do with Site.js, including using node modules and [npm](https://www.npmjs.com/). To use custom node modules, initialise your _.dynamic_ folder using `npm init` and use `npm install`. Once you’ve done that, any modules you `require()` from your DotJS routes will be properly loaded and used.

Say, for example, that you want to display a random ASCII Cow using the Cows module (because why not?) To do so, create a _package.json_ file in your _.dynamic_ folder (e.g., use `npm init` to create this interactively). Here’s a basic example:
Aral Balkan's avatar
Aral Balkan committed
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013

```json
{
  "name": "random-cow",
  "version": "1.0.0",
  "description": "Displays a random cow.",
  "main": "index.js",
  "author": "Aral Balkan <mail@ar.al> (https://ar.al)",
  "license": "AGPL-3.0-or-later"
}
```

1014
Then, install the [cows node module](https://www.npmjs.com/package/cows) using npm:
Aral Balkan's avatar
Aral Balkan committed
1015
1016
1017
1018
1019

```sh
npm i cows
```

1020
This will create a directory called _node_modules_ in your _.dynamic_ folder and install the cows module (and any dependencies it may have) inside it. Now is also a good time to create a `.gitignore` file in the root of your web project and add the _node_modules_ directory to it if you’re using Git for source control so that you do not end up accidentally checking in your node modules. Here’s how you would do this using the command-line on Linux-like systems:
Aral Balkan's avatar
Aral Balkan committed
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033

```sh
echo 'node_modules' >> .gitignore
```

Now, let’s create the route. We want it reachable at `https://localhost/cows` (of course), so let’s put it in:

```
.dynamic/
    └ cows
        └ index.js
```

1034
And, finally, here’s the code for the route itself:
Aral Balkan's avatar
Aral Balkan committed
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047

```js
const cows = require('cows')()

module.exports = function (request, response) {
  const randomCowIndex = Math.round(Math.random()*cows.length)-1
  const randomCow = cows[randomCowIndex]

  function randomColor () {
    const c = () => (Math.round(Math.random() * 63) + 191).toString(16)
    return `#${c()}${c()}${c()}`
  }

1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
  response.html(`
    <!doctype html>
    <html lang='en'>
    <head>
      <meta charset='utf-8'>
      <meta name='viewport' content='width=device-width, initial-scale=1.0'>
      <title>Cows!</title>
      <style>
        html { font-family: sans-serif; color: dark-grey; background-color: ${randomColor()}; }
        body {
          display: grid; align-items: center; justify-content: center;
          height: 100vh; vertical-align: top; margin: 0;
        }
        pre { font-size: 24px; color: ${randomColor()}; mix-blend-mode: difference;}
      </style>
    </head>
    <body>
        <pre>${randomCow}</pre>
    </body>
    </html>
  `)
Aral Balkan's avatar
Aral Balkan committed
1069
1070
1071
1072
1073
}
```

Now if you run `site` on the root of your web folder (the one that contains the _.dynamic_ folder) and hit `https://localhost/cows`, you should get a random cow in a random colour every time you refresh.

1074
If including HTML and CSS directly in your dynamic route makes you cringe, feel free to `require` your templating library of choice and move them to external files. As hidden folders (directories that begin with a dot) are ignored in the _.dynamic_ folder and its subfolders, you can place any assets (HTML, CSS, images, etc.) into a directory that starts with a dot and load them in from there.
Aral Balkan's avatar
Aral Balkan committed
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158

For example, if I wanted to move the HTML and CSS into their own files in the example above, I could create the following directory structure:

```
.dynamic/
    └ cows
        ├ .assets
        │     ├ index.html
        │     └ index.css
        └ index.js
```

For this example, I’m not going to use an external templating engine but will instead rely on the built-in template string functionality in JavaScript along with `eval()` (which is perfectly safe to use here as we are not processing external input).

So I move the HTML to the _index.html_ file (and add a template placeholder for the CSS in addition to the existing random cow placeholder):

```html
<!doctype html>
<html lang='en'>
<head>
  <meta charset='utf-8'>
  <meta name='viewport' content='width=device-width, initial-scale=1.0'>
  <title>Cows!</title>
  <style>${css}</style>
</head>
<body>
    <pre>${randomCow}</pre>
</body>
</html>
```

And, similarly, I move the CSS to its own file, _index.css_:

```css
html {
  font-family: sans-serif;
  color: dark-grey;
  background-color: ${randomColor()};
}

body {
  display: grid;
  align-items: center;
  justify-content: center;
  height: 100vh;
  vertical-align: top;
  margin: 0;
}

pre {
  font-size: 24px;
  mix-blend-mode: difference;
  color: ${randomColor()};
}
```

Then, finally, I modify my `cows` route to read in these two template files and to dynamically render them in response to requests. My _index.js_ now looks like this:

```js
// These are run when the server starts so sync calls are fine.
const fs = require('fs')
const cssTemplate = fs.readFileSync('cows/.assets/index.css')
const htmlTemplate = fs.readFileSync('cows/.assets/index.html')
const cows = require('cows')()

module.exports = function (request, response) {
  const randomCowIndex = Math.round(Math.random()*cows.length)-1
  const randomCow = cows[randomCowIndex]

  function randomColor () {
    const c = () => (Math.round(Math.random() * 63) + 191).toString(16)
    return `#${c()}${c()}${c()}`
  }

  function render (template) {
    return eval('`' + template + '`')
  }

  // We render the CSS template first…
  const css = render(cssTemplate)

  // … because the HTML template references the rendered CSS template.
  const html = render(htmlTemplate)

1159
  response.html(html)
Aral Balkan's avatar
Aral Balkan committed
1160
1161
1162
}
```

Aral Balkan's avatar
Aral Balkan committed
1163
When you save this update, Site.js will automatically reload the server with your new code (version 12.9.7 onwards). When you refresh in your browser, you should see exactly the same behaviour as before.
1164

1165
As you can see, you can create quite a bit of dynamic functionality just by using DotJS with its most basic file-based routing mode. However, with this convention you are limited to GET routes. To use both GET and POST routes, you have to do a tiny bit more work, as explained in the next section.
1166

1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
#### GET and POST routes

If you need POST routes (e.g., you want to post form content back to the server) in addition to GET routes, the directory structure works a little differently. In this case, you have to create a _.get_ directory for your GET routes and a _.post_ directory for your post routes.

Otherwise, the naming and directory structure conventions work exactly as before.

So, for example, if you have the following directory structure:

```
site/
  └ .dynamic/
        ├ .get/
        │   └ index.js
        └ .post/
            └ index.js
```

Then a GET request for `https://localhost` will be routed to _site/.dynamic/.get/index.js_ and a POST request for `https://localhost` will be routed to _site/.dynamic/.post/index.js_.

These two routes are enough to cover your needs for dynamic routes and form handling.

Aral Balkan's avatar
Aral Balkan committed
1188
1189
#### WebSocket (WSS) routes

1190
1191
1192
Site.js is not limited to HTTPS, it also supports secure WebSockets.

To define WebSocket (WSS) routes alongside HTTPS routes, modify your directory structure so it resembles the one below:
Aral Balkan's avatar
Aral Balkan committed
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205

```
site/
  └ .dynamic/
        ├ .https/
        │   ├ .get/
        │   │   └ index.js
        │   └ .post/
        │       └ index.js
        └ .wss/
            └ index.js
```

1206
1207
1208
Note that all we’ve done is to move our HTTPS _.get_ and _.post_ directories under a _.https_ directory and we’ve created a separate _.wss_ directory for our WebSocket routes.

Here’s how you would implement a simple echo server that sends a copy of the message it receives from a client to that client:
1209
1210
1211
1212
1213
1214
1215
1216
1217

```js
module.exports = (client, request) => {
  client.on('message', (data) => {
    client.send(data)
  })
}
```

1218
You can also broadcast messages to all or a subset of connected clients. Here, for example, is a naïve single-room chat server implementation that broadcasts messages to all connected WebSocket clients (including the client that originally sent the message and any other clients that might be connected to different WebSocket routes on the same server):
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237

```js
module.exports = (currentClient, request) {
  ws.on('message', message => {
    this.getWss().clients.forEach(client => {
      client.send(message)
    })
  })
})
```

To test it out, run Site.js and then open up the JavaScript console in a couple of browser windows and enter the following code into them:

```js
const socket = new WebSocket('https://localhost/chat')
socket.onmessage = message => console.log(message.data)
socket.send('Hello!')
```

1238
1239
For a slightly more sophisticated example that doesn’t broadcast a client’s own messages to itself and selectively broadcasts to only the clients in the same “rooms”, see the [Simple Chat example](examples/simple-chat). And here’s [a step-by-step tutorial](https://ar.al/2019/10/11/build-a-simple-chat-app-with-site.js/) that takes you through how to build it.

1240
Here’s a simplified listing of the code for the server component of that example:
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258

```js
module.exports = function (client, request) {
  // A new client connection has been made.
  // Persist the client’s room based on the path in the request.
  client.room = this.setRoom(request)

  console.log(`New client connected to ${client.room}`)

  client.on('message', message => {
    // A new message has been received from a client.
    // Broadcast it to every other client in the same room.
    const numberOfRecipients = this.broadcast(client, message)

    console.log(`${client.room} message broadcast to ${numberOfRecipients} recipient${numberOfRecipients === 1 ? '' : 's'}.`)
  })
}
```
1259

1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
### Custom Middleware

As of version 16.5.0, you can now add any piece of standard Express middleware to your server by defining them as modules in a `.middleware` directory in your project.

For example, to have your server allow all cross-origin requests, define the following middleware in `.middleware/allow-all-cors.js`:

```js
module.exports = (request, response, next) => {
  response.header('Access-Control-Allow-Origin', '*')
  response.header('Access-Control-Allow-Headers', 'Origin, X-Requested-With, Content-Type, Accept')
  next()
}
```

This gives you full flexibility in customising your server however you like.

1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
1355
1356
1357
1358
1359
1360
### Persisting data on the server with JavaScript Database (JSDB)

The chat examples so far have been ephemeral; the chat log is not stored anywhere. While that has its uses, it does mean, for example, that someone coming into a conversation after it has already started will not see what was said. You can easily implement that feature using the bundled JavaScript Database (JSDB).

JSDB is a transparent, in-memory, streaming write-on-update JavaScript database for Small Web applications that persists to a JavaScript transaction log.

What that means in practice is that it’s very simple to use and great for storing small pieces of data on the server. (Note that whenever possible, you should store data on the client not the server for privacy reasons.)

Your Site.js server has a global database called `db` that you can use from any route.

Here’s how you would persist the data in our simple chat example using JSDB:

```js
// Ensure the messages table exists.
if (!db.messages) {
  db.messages = []
}

module.exports = function (client, request) {
  // A new client connection has been made.
  // Persist the client’s room based on the path in the request.
  client.room = this.setRoom(request)

  console.log(`New client connected to ${client.room}`)

  // Send new clients all existing messages.
  client.send(JSON.stringify(db.messages))

  client.on('message', message => {
    // Persist the message.
    db.messages.push(message)

    // A new message has been received from a client.
    // Broadcast it to every other client in the same room.
    const numberOfRecipients = this.broadcast(client, message)

    console.log(`${client.room} message broadcast to ${numberOfRecipients} recipient${numberOfRecipients === 1 ? '' : 's'}.`)
  })
}
```

Here’s a break down of the changes:

  1. You implement a global check that occurs when the module of your route is loaded to create the `messages` table (in this case, an array, although it can also be an object):

      ```js
      if (!db.messages) {
        db.messages = []
      }
      ```

  2. When a new client joins, you serialise the messages array in JSON format and send it to that client.

      ```js
      client.send(JSON.stringify(db.messages))
      ```

  3. When a message is sent by a client, you persist it in the `messages` table.

      ```js
      db.messages.push(message)
      ```

If none of this feels like you’re using a database, that’s by design. JSDB is in-process, in-memory, and JavaScript through and through. It uses proxies to make it feel like you’re just working with plain old JavaScript objects. It even persists the data as JavaScript code (not JSON) in a format called JavaScript Data Format (JSDF).

And you’re not limited to only persisting and loading data, you can also query it. You do so using the JavaScript Query Language (JSQL).

Just like the other aspects of JSDB, JSQL is designed for ease of use. For most regular use, it should feel like you’re asking a question in plain English.

For example, if you wanted to get all the messages send by the person whose nickname is `Aral`, you would write the following:

```js
db.messages.where('nickname').is('Aral').get()
```

The result would be an array of messages.

Similarly, if you wanted just the first message that contained the word `kitten`, you would write:

```js
db.messages.where('text').includes('kitten').getFirst()
```

You can learn more about JSDB in the [JSDB documentation](https://github.com/small-tech/jsdb/blob/master/README.md).

1361
### Advanced routing (routes.js file)