Verified Commit 43e5803f authored by Aral Balkan's avatar Aral Balkan
Browse files

Merge branch '14.5.0'

parents 709ac13a af23c7e3
......@@ -4,6 +4,16 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [14.5.0] - 2020-08-26
### Added
- Wildcard static route support. Any path under https://your.site/x will route to .wildcard/x/index.html if that file exists. So, for example, https://your.site/x/y, https://your.site/x/y/z, etc., will all route to the same static file. Use this if you want to allow path-style arguments in your URLs but carry out client-side processing. This saves you from having to create .dynamic routes for that use case.
### Improved
- Refactored file watching (now using a single file watcher for both dynamic and wildcard route changes. Also, upgraded to the latest Chokidar).
## [14.4.0] - 2020-08-25
The pull and push commands implement Small Web conventions to enable simple bi-directional data transfer between a local machine and a Small Web server.
......
......@@ -33,6 +33,8 @@ We exist in part thanks to patronage by people like you. If you share [our visio
- __Supports the creation of static web sites, dynamic web sites, and hybrid sites__ (via integrated [Node.js](https://nodejs.org/) and [Express](https://expressjs.com)).
- __Supports [Wildcard routes](#wildcard-routes)__ for purely client-side specialisation using path-based parameters.
- __Supports [DotJS](#dotjs) for dynamic routes.__ (DotJS is PHP-like simple routing for Node.js introduced by Site.js for quickly prototyping and building dynamic sites).
- __Includes [Hugo static site generator](#static-site-generation).__
......@@ -1205,6 +1207,66 @@ const appPath = require.main.filename.replace('bin/site.js', '')
The code within your JavaScript routes is executed on the server. Exercise the same caution as you would when creating any Node.js app (sanitise input, etc.)
## Wildcard routes
As of version 14.5.0, if all you want to do is to customise the behaviour of your pages using client-side JavaScript based on parameters provided through the URL path, you don’t have to use dynamic routes and a `routes.js` file, you can use wildcard routes instead, which are much simpler.
So say, for example, that you want your app to greet people based on the URL that’s provided:
- https://my.site/hello/aral should say “Hello, Aral”
- https://my.site/hello/laura should say “Hello, Laura”
- etc.
To do this using wildcard routes:
1. Create a folder called `.wildcard` in the root directory of your site.
2. Inside that folder, create a folder named `hello`
3. In the `hello` folder, create an `index.html` with the following code:
```html
<!doctype html>
<html lang='en'>
<head>
<meta charset='utf-8'>
<meta name='viewport' content='width=device-width, initial-scale=1.0'>
<title>Hello!</title>
</head>
<body>
<script>
function capitaliseFirstLetter (word) {
return word.split('').map((letter,index) => !index ? letter.toUpperCase() : letter).join('')
}
const name = capitaliseFirstLetter(window.arguments[0])
document.write(`<h1>Hello, ${name}</h1>`)
</script>
</body>
</html>
```
### How it works
When Site.js finds a `.wildcard` folder, it adds every first-level sub-folder in it as a route that maps to the `index.html` file in it. In the example above, `/hello/aral` and `hello/what/is/this/about` will both map to the same file.
Any path fragment after the route name itself is treated as a positional argument.
Although you could parse the `document.location` yourself to get at the arguments and the route name, Site.js makes it even easier for you by injecting a tiny bit of JavaScript at the top of your page that exposes these as:
- `window.route`: the name of your route. In the above example, this is `hello`.
- `window.arguments`: an array of arguments. For `/hello/what/is/this/about`, this would be `['hello', 'what', 'is', 'this', 'about']`
This is the JavaScript that’s injected into your page:
```html
<script>
// Site.js: add window.routeName and window.arguments objects to wildcard route.
__site_js__pathFragments = document.location.pathname.split('/')
window.route = __site_js__pathFragments[1]
window.arguments = __site_js__pathFragments.slice(2).filter(value => value !== '')
delete __site_js__pathFragments
</script>
```
## API
You can also include Site.js as a Node module into your Node project. This section details the API you can use if you do that.
......
......@@ -21,6 +21,7 @@
const fs = require('fs-extra')
const path = require('path')
const os = require('os')
const EventEmitter = require('events')
const childProcess = require('child_process')
const http = require('http')
const https = require('@small-tech/https')
......@@ -235,6 +236,8 @@ class Site {
// Introduce ourselves.
Site.logAppNameAndVersion()
this.eventEmitter = new EventEmitter()
// Ensure that the settings directory exists and create it if it doesn’t.
fs.ensureDirSync(Site.settingsDirectory)
......@@ -562,6 +565,7 @@ class Site {
this.appAddTest500ErrorPage()
this.appAddDynamicRoutes()
this.appAddStaticRoutes()
this.appAddWildcardRoutes()
this.appAddArchivalCascade()
}
......@@ -624,36 +628,43 @@ class Site {
// Enable the ability to destroy the server (close all active connections).
enableDestroy(this.server)
this.server.on('close', () => {
this.server.on('close', async () => {
// Clear the auto update check interval.
if (this.autoUpdateCheckInterval !== undefined) {
clearInterval(this.autoUpdateCheckInterval)
this.log(' ⏰ ❨site.js❩ Cleared auto-update check interval.')
}
if (this.app.__dynamicFolderWatcher !== undefined) {
this.app.__dynamicFolderWatcher.close()
this.log (` 🚮 ❨site.js❩ Removed root file watcher.`)
}
// Ensure dynamic route watchers are removed.
if (this.app.__dynamicFileWatcher !== undefined) {
this.app.__dynamicFileWatcher.close()
this.log (` 🚮 ❨site.js❩ Removed dynamic file watchers.`)
if (this.app.__fileWatcher !== undefined) {
try {
await this.app.__fileWatcher.close()
this.log (` 🚮 ❨site.js❩ Removed file watcher.`)
} catch (error) {
this.log(` ❌ ❨site.js❩ Could not remove file watcher: ${error}`)
}
}
// Ensure that the static route file watchers are removed.
if (this.app.__staticRoutes !== undefined) {
this.app.__staticRoutes.cleanUp(() => {
this.log(' 🚮 ❨site.js❩ Live reload file system watchers removed from static web routes on server close.')
await new Promise((resolve, reject) => {
this.app.__staticRoutes.cleanUp(() => {
this.log(' 🚮 ❨site.js❩ Live reload file system watchers removed from static web routes on server close.')
resolve()
})
})
}
this.log(' 🚮 ❨site.js❩ Housekeeping is done!')
this.eventEmitter.emit('housekeepingIsDone')
})
}
// Finish configuring the app. These are the routes that come at the end.
// (We need to add the WebSocket (WSS) routes after the server has been created).
endAppConfiguration () {
// Create the file watcher to watch for changes on dynamic and wildcard routes.
this.createFileWatcher()
// If we need to load dynamic routes from a routesJS file, do it now.
if (this.routesJsFile !== undefined) {
this.createWebSocketServer()
......@@ -1063,6 +1074,159 @@ class Site {
}
// Restarts the server.
restartServer () {
if (process.env.NODE_ENV === 'production') {
// We’re running production, to restart the daemon, just exit.
// (We let ourselves fall, knowing that systemd will catch us.) ;)
process.exit()
} else {
// We’re running as a regular process. Just restart the server, not the whole process.
// Do some housekeeping.
Graceful.off('SIGINT', this.goodbye)
Graceful.off('SIGTERM', this.goodbye)
if (this.hugoServerProcesses) {
this.log(' 🚮 ❨site.js❩ Killing Hugo server processes.')
this.hugoServerProcesses.forEach(hugoServerProcess => hugoServerProcess.kill())
}
// Wait until housekeeping is done cleaning up after the server is destroyed before
// restarting the server.
this.eventEmitter.on('housekeepingIsDone', () => {
// Restart the server.
this.eventEmitter.removeAllListeners()
this.log('\n 🐁 ❨site.js❩ Restarting server…\n')
const {commandPath, args} = cli.initialise(process.argv.slice(2))
serve(args)
this.log('\n 🐁 ❨site.js❩ Server restarted.\n')
})
// Destroy the current server (so we do not get a port conflict on restart before
// we’ve had a chance to terminate our own process).
this.server.destroy()
// Stop accepting new connections.
this.server.close(() => {
this.log('\n 🐁 ❨site.js❩ Server is closed.\n')
this.server.removeAllListeners('close')
this.server.removeAllListeners('error')
})
}
}
// Returns a pretty human-readable string describing the file watcher change reflected in the event.
prettyFileWatcherEvent (event) {
return ({
'add': 'file added',
'addDir': 'directory added',
'change': 'file changed',
'unlink': 'file deleted',
'unlinkDir': 'directory deleted'
}[event])
}
// Creates a file watcher to restart the server if a dynamic or wildcard route changes.
// (Changes to static files do not cause a server restart and are handled by the instant module
// with live reload.)
//
// Note: Chokidar appears to have an issue where changes are no longer picked up if
// ===== a created folder is then removed. This should not be a big problem in actual
// usage, but let’s keep an eye on this. (Note that if you listen for the 'raw'
// event, it gets triggered with a 'rename' when a removed/recreated folder
// is affected.) See: https://github.com/paulmillr/chokidar/issues/404#issuecomment-666669336
createFileWatcher () {
const fileWatchPath = `${this.pathToServe.replace(/\\/g, '/')}/**/*`
this.app.__fileWatcher = chokidar.watch(fileWatchPath, {
persistent: true,
ignoreInitial: true
})
this.app.__fileWatcher.on ('all', (event, file) => {
if (file.includes('/.dynamic')) {
//
// Dynamic route change.
//
this.log(` 🐁 ❨site.js❩ Dynamic route change: ${clr(`${this.prettyFileWatcherEvent(event)}`, 'green')} (${clr(file, 'cyan')}).`)
this.log('\n 🐁 ❨site.js❩ Requesting restart…\n')
this.restartServer()
} else if (file.includes('/.wildcard')) {
//
// Wildcard route change.
//
this.log(` 🐁 ❨site.js❩ Wildcard route change: ${clr(`${this.prettyFileWatcherEvent(event)}`, 'green')} (${clr(file, 'cyan')}).`)
this.log('\n 🐁 ❨site.js❩ Requesting restart…\n')
this.restartServer()
}
})
this.log(' 🐁 ❨site.js❩ Watching for changes to dynamic and wildcard routes.')
}
// Add wildcard routes.
//
// Wildcard routes are static routes where any path under https://your.site/x will route to .wildcard/x/index.html
// if that file exists. So, for example, https://your.site/x/y, https://your.site/x/y/z, etc., will all route to the
// same static file. Use this if you want to allow path-style arguments in your URLs but carry out client-side
// processing. This saves you from having to create .dynamic routes for that use case.
appAddWildcardRoutes () {
const wildcardRoutesDirectory = path.join(this.pathToServe, '.wildcard')
const wildcards = {}
if (fs.existsSync(wildcardRoutesDirectory)) {
fs.readdirSync(wildcardRoutesDirectory, {withFileTypes: true}).forEach(file => {
if (file.isDirectory()) {
const wildcard = file.name
const wildcardIndexFilePath = path.join(wildcardRoutesDirectory, wildcard, 'index.html')
if (fs.existsSync(wildcardIndexFilePath)) {
this.log(` 🃏 ❨site.js❩ Serving wildcard route: ${clr(`https://${this.prettyLocation()}/${wildcard}/**/*`, 'green')}${clr(`/.wildcard/${wildcard}/index.html`, 'cyan')}`)
// Read the HTML content and inject some javascript to make it easy to access the route
// name and the arguments from window.route and and window.arguments.
const wildcardIndexFilePath = path.join(wildcardRoutesDirectory, wildcard, 'index.html')
wildcards[wildcard] = fs.readFileSync(wildcardIndexFilePath, 'utf-8').replace('<body>', `
<body>
<script>
// Site.js: add window.routeName and window.arguments objects to wildcard route.
__site_js__pathFragments = document.location.pathname.split('/')
window.route = __site_js__pathFragments[1]
window.arguments = __site_js__pathFragments.slice(2).filter(value => value !== '')
delete __site_js__pathFragments
</script>
`)
this.app.use(`/${wildcard}`, (() => {
// Capture the current wildcard
const __wildcard = wildcard
return (request, response, next) => {
const pathFragments = request.path.split('/')
if (pathFragments.length >= 2 && pathFragments[1] !== '') {
// OK, we have a sub-path, so serve the wildcard.
response
.type('html')
.end(wildcards[__wildcard])
} else {
// No sub-path, ignore this request.
next()
}
}
})())
} else {
// We found a directory inside of the .wildcard directory but it doesn’t have an index.html
// file inside it with the content to serve. Warn the person.
this.log(` ❗ ❨site.js❩ Wilcard directory found at /.wildcard/${wildcard} but there is no index.html inside it. Ignoring…`)
}
}
})
}
}
// Add dynamic routes, if any, if a <pathToServe>/.dynamic/ folder exists.
// If there are errors in any of your dynamic routes, you will get 500 (server) errors.
//
......@@ -1080,90 +1244,11 @@ class Site {
// For full details, please see the readme file.
appAddDynamicRoutes () {
// Restarts the server.
const restartServer = () => {
if (process.env.NODE_ENV === 'production') {
// We’re running production, to restart the daemon, just exit.
// (We let ourselves fall, knowing that systemd will catch us.) ;)
process.exit()
} else {
// We’re running as a regular process. Just restart the server, not the whole process.
// Do some housekeeping.
Graceful.off('SIGINT', this.goodbye)
Graceful.off('SIGTERM', this.goodbye)
if (this.hugoServerProcesses) {
this.log(' 🚮 ❨site.js❩ Killing Hugo server processes.')
this.hugoServerProcesses.forEach(hugoServerProcess => hugoServerProcess.kill())
}
// Destroy the current server (so we do not get a port conflict on restart before
// we’ve had a chance to terminate our own process).
this.server.destroy()
// Stop accepting new connections.
this.server.close(() => {
// Restart the server.
this.server.removeAllListeners('close')
this.server.removeAllListeners('error')
const {commandPath, args} = cli.initialise(process.argv.slice(2))
serve(args)
this.log(' 🐁 ❨site.js❩ Restarted server.\n')
})
}
}
// Regardless of whether there is a .dynamic folder or not, add a watcher
// to watch for a change in the existence of the .dynamic folder itself. This can happen
// if it is created or deleted locally or on a remote server as the result of a sync to
// the remote server. In either case, we want to restart the server so that the new
// routes are added or removed accordingly.
//
// Note: Chokidar appears to have an issue where changes are no longer picked up if
// ===== a created folder is then removed. This should not be a big problem in actual
// usage, but let’s keep an eye on this. (Note that if you listen for the 'raw'
// event, it gets triggered with a 'rename' when a removed/recreated folder
// is affected.) See: https://github.com/paulmillr/chokidar/issues/404#issuecomment-666669336
const dynamicFolderWatchPath = `${this.pathToServe.replace(/\\/g, '/')}/**/*`
this.app.__dynamicFolderWatcher = chokidar.watch(dynamicFolderWatchPath, {
persistent: true,
ignoreInitial: true
})
this.app.__dynamicFolderWatcher.on ('addDir', file => {
if (file.endsWith('.dynamic')) {
this.log(` 🐁 ❨site.js❩ ${clr(`Dynamic folder created!`, 'green')}`)
this.log(' 🐁 ❨site.js❩ Requesting restart…\n')
restartServer()
}
})
// Initially check if a dynamic routes directory exists. If it does not,
// we don’t need to take this any further.
const dynamicRoutesDirectory = path.join(this.pathToServe, '.dynamic')
if (fs.existsSync(dynamicRoutesDirectory)) {
// Watch .dynamic directory (recursively) so we can restart server when code changes.
// Windows-style slashes are not part of the glob standard so we have to ensure all
// slashes are forward slashes to ensure correct functioning on Windows 10
// (see https://github.com/paulmillr/chokidar#api).
const watchPath = `${dynamicRoutesDirectory.replace(/\\/g, '/')}/**`
this.app.__dynamicFileWatcher = chokidar.watch(watchPath, {
persistent: true,
ignoreInitial: true
})
this.app.__dynamicFileWatcher.on ('all', (event, file) => {
this.log(` 🐁 ❨site.js❩ ${clr('Code updated', 'green')} in ${clr(file, 'cyan')}!`)
this.log(' 🐁 ❨site.js❩ Requesting restart…\n')
restartServer()
})
const addBodyParser = () => {
this.app.use(bodyParser.json())
this.app.use(bodyParser.urlencoded({ extended: true }))
......
{
"name": "@small-tech/site.js",
"version": "14.3.0",
"version": "14.5.0",
"lockfileVersion": 1,
"requires": true,
"dependencies": {
......@@ -792,9 +792,9 @@
}
},
"binary-extensions": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/binary-extensions/-/binary-extensions-2.0.0.tgz",
"integrity": "sha512-Phlt0plgpIIBOGTT/ehfFnbNlfsDEiqmzE2KRXoX1bLIlir4X/MR+zSyBEkL05ffWgnRSf/DXv+WrUAVr93/ow=="
"version": "2.1.0",
"resolved": "https://registry.npmjs.org/binary-extensions/-/binary-extensions-2.1.0.tgz",
"integrity": "sha512-1Yj8h9Q+QDF5FzhMs/c9+6UntbD5MkRfRwac8DoEm9ZfUBZ7tZ55YcGVAzEe4bXsdQHEk+s9S5wsOKVdZrw0tQ=="
},
"bl": {
"version": "3.0.0",
......@@ -1060,9 +1060,9 @@
"dev": true
},
"chokidar": {
"version": "3.3.1",
"resolved": "https://registry.npmjs.org/chokidar/-/chokidar-3.3.1.tgz",
"integrity": "sha512-4QYCEWOcK3OJrxwvyyAOxFuhpvOVCYkr33LPfFNBjAD/w3sEzWsp2BUOkI4l9bHvWioAd0rc6NlHUOEaWkTeqg==",
"version": "3.4.2",
"resolved": "https://registry.npmjs.org/chokidar/-/chokidar-3.4.2.tgz",
"integrity": "sha512-IZHaDeBeI+sZJRX7lGcXsdzgvZqKv6sECqsbErJA4mHWfpRrD8B97kSFN4cQz6nGBGiuFia1MKR4d6c1o8Cv7A==",
"requires": {
"anymatch": "~3.1.1",
"braces": "~3.0.2",
......@@ -1071,7 +1071,7 @@
"is-binary-path": "~2.1.0",
"is-glob": "~4.0.1",
"normalize-path": "~3.0.0",
"readdirp": "~3.3.0"
"readdirp": "~3.4.0"
},
"dependencies": {
"braces": {
......@@ -1090,14 +1090,6 @@
"to-regex-range": "^5.0.1"
}
},
"glob-parent": {
"version": "5.1.0",
"resolved": "https://registry.npmjs.org/glob-parent/-/glob-parent-5.1.0.tgz",
"integrity": "sha512-qjtRgnIVmOfnKUE3NJAQEdk+lKrxfw8t5ke7SXtfMTHcjsBfOfWXCQfdb30zfDoZQ2IRSIiidmjtbHZPZ++Ihw==",
"requires": {
"is-glob": "^4.0.1"
}
},
"is-number": {
"version": "7.0.0",
"resolved": "https://registry.npmjs.org/is-number/-/is-number-7.0.0.tgz",
......@@ -2320,9 +2312,9 @@
"dev": true
},
"fsevents": {
"version": "2.1.2",
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.1.2.tgz",
"integrity": "sha512-R4wDiBwZ0KzpgOWetKDug1FZcYhqYnUYKtfZYt4mD5SBz76q0KR4Q9o7GIPamsVPGmW3EYPPJ0dOOjvx32ldZA==",
"version": "2.1.3",
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.1.3.tgz",
"integrity": "sha512-Auw9a4AxqWpa9GUfj370BMPzzyncfBABW8Mab7BGWBYDj4Isgq+cDKtx0i6u9jcX9pQDnswsaaOTgTmA5pEjuQ==",
"optional": true
},
"function-bind": {
......@@ -2381,7 +2373,6 @@
"version": "5.1.1",
"resolved": "https://registry.npmjs.org/glob-parent/-/glob-parent-5.1.1.tgz",
"integrity": "sha512-FnI+VGOpnlGHWZxthPGR+QhR78fuiK0sNLkHQv+bL9fQi57lNNdquIbna/WrfROrolq8GK5Ek6BiMwqL/voRYQ==",
"dev": true,
"requires": {
"is-glob": "^4.0.1"
}
......@@ -4180,11 +4171,11 @@
}
},
"readdirp": {
"version": "3.3.0",
"resolved": "https://registry.npmjs.org/readdirp/-/readdirp-3.3.0.tgz",
"integrity": "sha512-zz0pAkSPOXXm1viEwygWIPSPkcBYjW1xU5j/JBh5t9bGCJwa6f9+BJa6VaB2g+b55yVrmXzqkyLf4xaWYM0IkQ==",
"version": "3.4.0",
"resolved": "https://registry.npmjs.org/readdirp/-/readdirp-3.4.0.tgz",
"integrity": "sha512-0xe001vZBnJEK+uKcj8qOhyAKPzIT+gStxWr3LCB0DwcXR5NZJ3IaC+yGnHCYzB/S7ov3m3EEbZI2zeNvX+hGQ==",
"requires": {
"picomatch": "^2.0.7"
"picomatch": "^2.2.1"
}
},
"referrer-policy": {
......
{
"name": "@small-tech/site.js",
"version": "14.4.0",
"version": "14.5.0",
"description": "Small Web construction set.",
"keywords": [
"web server",
......@@ -62,7 +62,7 @@
"ansi-escape-sequences": "^4.1.0",
"bent": "^7.3.4",
"body-parser": "^1.19.0",
"chokidar": "^3.3.0",
"chokidar": "^3.4.2",
"concat-stream": "^2.0.0",
"debounce": "^1.2.0",
"decache": "^4.5.1",
......
......@@ -27,6 +27,13 @@ function localhost(path) {
return `https://localhost${path}`
}
function dehydrate (str) {
if (typeof str !== 'string') {
str = str.toString('utf-8')
}
return str.replace(/\s/g, '')
}
async function secureGet (url) {
return new Promise((resolve, reject) => {
https.get(url, (response) => {
......@@ -335,6 +342,58 @@ test('[site.js] dynamic route loading from routes.js file', async t => {
})
test('[site.js] wildcard routes', async t => {
t.plan(2)
const site = new Site({path: 'test/site-wildcard-routes'})
await site.serve()
let response
try {
response = await secureGet('https://localhost/hello/there/who/is/this')
} catch (error) {
console.log(error)
process.exit(1)
}
t.strictEquals(response.statusCode, 200, 'request succeeds')
t.strictEquals(dehydrate(response.body).toLowerCase(), dehydrate(`
<!DOCTYPE html>
<html lang='en'>
<head>
<meta charset='UTF-8'>
<meta name='viewport' content='width=device-width, initial-scale=1.0'>
<title>Wildcard: hello</title>
</head>
<body>
<script>
// Site.js: add window.routeName and window.arguments objects to wildcard route.
__site_js__pathFragments = document.location.pathname.split('/')
window.route = __site_js__pathFragments[1]
window.arguments = __site_js__pathFragments.slice(2).filter(value => value !== '')
delete __site_js__pathFragments
</script>
<script>
document.write(\`<h1><em>\${window.route}</em> wildcard route</h1>\`)
document.write('<p>Called with the following arguments:</p>')
document.write('<ol>')
window.arguments.forEach(argument => {
document.write(\`<li>\${argument}</li>\`)
})
document.write('</ol>')
</script>
<script src="/instant/client/bundle.js"></script>
</body>
</html>
`).toLowerCase(), 'wildcard route body is as expected')
site.server.close()
})
test('[site.js] archival cascade', async t => {
t.plan(8)
......
<!DOCTYPE html>
<html lang='en'>
<head>
<meta charset='UTF-8'>
<meta name='viewport' content='width=device-width, initial-scale=1.0'>
<title>Wildcard: hello</title>
</head>
<body>
<script>
document.write(`<h1><em>${window.route}</em> wildcard route</h1>`)
document.write('<p>Called with the following arguments:</p>')
document.write('<ol>')
window.arguments.forEach(