Spiderable middleware

Search engines and social networks — such as Google, Facebook, Twitter, Yahoo, Bing, and many others — are constantly crawling your website to index its content. However, if your site is built using a modern JavaScript framework (for example, React, Preact, Vue, Svelte, Angular, Backbone, Ember, Meteor, Blaze), it serve only a basic HTML skeleton with script tags instead of fully rendered page content. This limitation negatively impacts website's SEO score, performance, and user experience when links shared by users around the web. The mission of spiderable-middleware and ostr.io is to enhance SEO and performance for modern JavaScript websites.

Why Pre-render?

🕸 Execute JavaScript, — get rendered HTML page and its content;
🏃‍♂️ Boost response rate and decrease response time with caching;
🚀 Optimized HTML markup for the best SEO score;
🖥 Support PWAs and SPAs;
📱 Support mobile-like crawlers;
💅 Support styled-components;
⚡️ Support AMP (Accelerated Mobile Pages);
🤓 Works with Content-Security-Policy and other complex front-end security rules;
❤️ Search engines and social network crawlers love straightforward and pre-rendered pages;
📱 Consistent link previews in messaging apps, like iMessage, Messages, Facebook, Slack, Telegram, WhatsApp, Viber, VK, Twitter, and other apps;
💻 Image, title, and description previews for posted links at social networks, like Facebook, Twitter, Instagram, and other social networks.

About Package

This package acts as middleware and intercepts requests to a Node.js application from web crawlers. All requests proxy passed to the Prerendering Service, which returns static, rendered HTML.

This is SERVER only package. For NPM make sure it's imported only in Node.js.

We made this package with developers in mind. It's well written in a very simple way, hackable, and easily tunable to meet all projects needs, it can be used to turn dynamic pages into rendered, cached, and lightweight static pages, just set botsUA to ['.*']. This is the best option to offload servers unless a website gets updated more often than once in 4 hours.

Note: This package proxies real HTTP headers and response code, to reduce overwhelming requests, try to avoid HTTP-redirect headers, like Location . Read how to return genuine status code and handle JS-redirects
Note: This is server only package. This package should be imported/initialized only within server code base

This middleware was tested and works like a charm with:

All other frameworks which follow Node's middleware convention - will work too.

This package was originally developed for ostr.io service. But it's not limited to, and can proxy-pass requests to any other rendering-endpoint.

Installation

Install spiderable-middleware package from NPM:

# using npmjs
npm install spiderable-middleware --save

# using yarn
yarn add spiderable-middleware

Usage

Get ready in a few lines of code

Basic usage

See all examples.

First, add fragment meta-tag to HTML template or page:

<html>
  <head>
    <meta name="fragment" content="!">
    <!-- ... -->
  </head>
  <body>
    <!-- ... -->
  </body>
</html>

Import or require spiderable-middleware package:

// ES6 import
import Spiderable from 'spiderable-middleware';

// or CommonJS require
const Spiderable = require('spiderable-middleware');

Register middleware handler:

import express from 'express';
import Spiderable from 'spiderable-middleware';

const spiderable = new Spiderable({
  rootURL: 'http://example.com',
  auth: 'APIUser:APIPass'
});

const app = express();
// ensure this is the most top registered handler
// to reduce response time and server load
app.use(spiderable.handler).get('/', (req, res) => {
  res.send('Hello World');
});

app.listen(3000);

We provide various options for serviceURL as "Rendering Endpoints", each endpoint has its own features to fit every project needs.

Return genuine status code

To pass expected status code of a response from front-end JavaScript framework to browser/crawlers use specially formatted HTML-comment. This comment can be placed in any part of HTML-page. head or body tag is the best place for it.

Format

html:

<!-- response:status-code=404 -->

jade:

// response:status-code=404

This package support any standard and custom status codes:

201 - 
401 - 
403 - 
500 - 
514 -  (non-standard)

Speed-up rendering

To speed-up rendering, JS-runtime should tell to the Spiderable engine when the page is ready. Set window.IS_RENDERED to false, and once the page is ready set this variable to true. Example:

<html>
  <head>
    <meta name="fragment" content="!">
    <script>
      window.IS_RENDERED = false;
    </script>
  </head>
  <body>
    <!-- ... -->
    <script type="text/javascript">
      //Somewhere deep in app-code:
      window.IS_RENDERED = true;
    </script>
  </body>
</html>

Detect request from Pre-rendering engine during runtime

Pre-rendering engine will set window.IS_PRERENDERING global variable to true. Detecting requests from pre-rendering engine are as easy as:

if (window.IS_PRERENDERING) {
  // This request is coming from Pre-rendering engine
}

Note: window.IS_PRERENDERING can be undefined on initial page load, and may change during runtime. That's why we recommend to pre-define a setter for IS_PRERENDERING:

let isPrerendering = false;
Object.defineProperty(window, 'IS_PRERENDERING', {
  set(val) {
    isPrerendering = val;
    if (isPrerendering === true) {
      // This request is coming from Pre-rendering engine
    }
  },
  get() {
    return isPrerendering;
  }
});

JavaScript redirects

Redirect browser/crawler inside application when needed while a page is loading (imitate navigation), use any of classic JS-redirects can be used, including framework's navigation, or History.pushState()

window.location.href = 'http://example.com/another/page';
window.location.replace('http://example.com/another/page');

Router.go('/another/page'); // framework's navigation !pseudo code

Note: Only 4 redirects are allowed during one request after 4 redirects session will be terminated.

API

Constructor `new Spiderable([opts])`

opts {Object} - Configuration options
opts.serviceURL {String} - Valid URL to Spiderable endpoint (local or foreign). Default: https://render.ostr.io. Can be set via environment variables: SPIDERABLE_SERVICE_URL or PRERENDER_SERVICE_URL
opts.rootURL {String} - Valid root URL of a website. Can be set via an environment variable: ROOT_URL
opts.auth {String} - [Optional] Auth string in next format: user:pass. Can be set via an environment variables: SPIDERABLE_SERVICE_AUTH or PRERENDER_SERVICE_AUTH. Default null
opts.sanitizeUrls {Boolean} - [Optional] Sanitize URLs in order to "fix" badly composed URLs. Default false
opts.botsUA {[String]} - [Optional] An array of strings (case insensitive) with additional User-Agent names of crawlers that needs to get intercepted. See default bot's names. Set to ['.*'] to match all browsers and robots, to serve static pages to all users/visitors
opts.ignoredHeaders {[String]} - [Optional] An array of strings (case insensitive) with HTTP header names to exclude from response. See default list of ignored headers. Set to ['.*'] to ignore all headers
opts.ignore {[String]} - [Optional] An array of strings (case sensitive) with ignored routes. Note: it's based on first match, so route /users will cause ignoring of /part/users/part, /users/_id and /list/of/users, but not /user/_id or /list/of/blocked-users. Default null
opts.only {[String|RegExp]} - [Optional] An array of strings (case sensitive) or regular expressions (could be mixed). Define exclusive route rules for pre-rendering. Could be used with opts.onlyRE rules. Note: To define "safe" rules as {RegExp} it should start with ^ and end with $ symbols, examples: [/^\/articles\/?$/, /^\/article\/[A-z0-9]{16}\/?$/]
opts.onlyRE {RegExp} - [Optional] Regular Expression with exclusive route rules for pre-rendering. Could be used with opts.only rules
opts.timeout {Number} - [Optional] Number, proxy-request timeout to rendering endpoint in milliseconds. Default: 180000
opts.requestOptions {Object} - [Optional] Options for request module (like: timeout, lookup, insecureHTTPParser), for all available options see http API docs
opts.debug {Boolean} - [Optional] Enable debug and extra logging, default: false

Note: Setting .onlyRE and/or .only rules are highly recommended. Otherwise, all routes, including randomly generated by bots will be subject of Pre-rendering and may cause unexpectedly higher usage.

// CommonJS
// const Spiderable = require('spiderable-middleware');

// ES6 import
// import Spiderable from 'spiderable-middleware';

const spiderable = new Spiderable({
  rootURL: 'http://example.com',
  auth: 'APIUser:APIPass'
});

// More complex setup (recommended):
const spiderable = new Spiderable({
  rootURL: 'http://example.com',
  serviceURL: 'https://render.ostr.io',
  auth: 'APIUser:APIPass',
  only: [
    /\/?/, // Root of the website
    /^\/posts\/?$/, // "/posts" path with(out) trailing slash
    /^\/post\/[A-z0-9]{16}\/?$/ // "/post/:id" path with(out) trailing slash
  ],
  // [Optionally] force ignore for secret paths:
  ignore: [
    '/account/', // Ignore all routes under "/account/*" path
    '/billing/' // Ignore all routes under "/billing/*" path
  ]
});

Configuration via env.vars

Same configuration can get achieved via setting up environment variables:

ROOT_URL='http://example.com'
SPIDERABLE_SERVICE_URL='https://render.ostr.io'
SPIDERABLE_SERVICE_AUTH='APIUser:APIPass'

alternatively, when migrating from other pre-rendering service — keep using existing variables, we support the next ones for compatibility:

ROOT_URL='http://example.com'
PRERENDER_SERVICE_URL='https://render.ostr.io'
PRERENDER_SERVICE_AUTH='APIUser:APIPass'

`spiderable.handler(req, res, next)`

Middleware handler. Alias: spiderable.handle.

// Express, Connect:
app.use(spiderable.handler);

// HTTP(s) Server
http.createServer((req, res) => {
  spiderable.handler(req, res, () => {
    // Callback, triggered if this request
    // is not a subject of spiderable pre-rendering
    res.writeHead(200, {'Content-Type': 'text/plain; charset=UTF-8'});
    res.end('Hello vanilla NodeJS!');
    // Or do something else ...
  });
}).listen(3000);

AMP Support

To properly serve pages for Accelerated Mobile Pages (AMP) we support following URI schemes:

# Regular URIs:
https://example.com/index.html
https://example.com/articles/article-title.html
https://example.com/articles/article-uniq-id/article-slug

# AMP optimized URIs (prefix):
https://example.com/amp/index.html
https://example.com/amp/articles/article-title.html
https://example.com/amp/articles/article-uniq-id/article-slug

# AMP optimized URIs (extension):
https://example.com/amp/index.amp.html
https://example.com/amp/articles/article-title.amp.html

All URLs with .amp. extension and /amp/ prefix will be optimized for AMP.

Rendering Endpoints

render (default) - https://render.ostr.io - This endpoint has "optimal" settings, and should fit 98% cases. This endpoint respects cache headers of Crawler and origin server
render-bypass (devel/debug) - https://render-bypass.ostr.io - This endpoint will bypass caching mechanisms. Use it when experiencing an issue, or during development, to make sure responses are not cached. It's safe to use this endpoint in production, but it may result in higher usage and response time
render-cache (under attack) - https://render-cache.ostr.io - This endpoint has the most aggressive caching mechanism. Use it to achieve the shortest response time, and when outdated pages (for 6-12 hours) are acceptable

To change default endpoint, grab integration examples code and replace render.ostr.io, with endpoint from the list above. For NPM integration change value of serviceURL option.

Note: Described differences in caching behavior related to intermediate proxy caching, Cache-Control header will be always set to the value defined in "Cache TTL". Cached results at the "Pre-rendering Engine" end can be purged at any time.

Debugging

Pass { debug: true } or set DEBUG=true environment variable to enable debugging mode.

To make sure a server can reach a rendering endpoint run cURL command or send request via Node.js to (replace example.com with your domain name):

curl -v "https://test:[email protected]/?url=http://example.com"

In this example we're using render-bypass.ostr.io endpoint to avoid any possible cached results, read more about rendering endpoints. As API credentials we're using test:test, this part of URL can be replaced with auth option from Node.js example. The API credentials and instructions can be found at the very bottom of Pre-rendering Panel of a particular host, — click on Integration Guide

# cURL example:
curl -v "https://test:[email protected]/?url=http://example.com"

// Node.js example:
const https = require('https');

https.get('https://test:[email protected]/?url=http://example.com', (resp) => {
  let data = '';

  resp.on('data', (chunk) => {
    data += chunk.toString('utf8');
  });

  resp.on('end', () => {
    console.log(data);
  });
}).on('error', (error) => {
  console.error(error);
});

Running Tests

Clone this package
In Terminal (Console) go to directory where package was cloned
Then run:

Node.js/Mocha

# Install development NPM dependencies:
npm install --save-dev
# Install NPM dependencies:
npm install --save
# Run tests:
ROOT_URL=http://127.0.0.1:3003 npm test

# Run same tests with extra-logging
DEBUG=true ROOT_URL=http://127.0.0.1:3003 npm test
# http://127.0.0.1:3003 can be changed to any local address, PORT is required!

Name		Name	Last commit message	Last commit date
Latest commit History 215 Commits
.github		.github
docs		docs
examples		examples
lib		lib
test		test
.eslintrc		.eslintrc
.gitignore		.gitignore
.meteorignore		.meteorignore
.npmignore		.npmignore
.versions		.versions
CHANGELOG.md		CHANGELOG.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
HISTORY.md		HISTORY.md
LICENSE		LICENSE
README.md		README.md
package-lock.json		package-lock.json
package.js		package.js
package.json		package.json

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Spiderable middleware

Why Pre-render?

ToC

About Package

Installation

Usage

Basic usage

Return genuine status code

Format

Speed-up rendering

Detect request from Pre-rendering engine during runtime

JavaScript redirects

API

Constructor `new Spiderable([opts])`

Configuration via env.vars

`spiderable.handler(req, res, next)`

AMP Support

Rendering Endpoints

Debugging

Running Tests

Node.js/Mocha

About

Releases 32

Contributors 4

Languages

License

veliovgroup/spiderable-middleware

Folders and files

Latest commit

History

Repository files navigation

Spiderable middleware

Why Pre-render?

ToC

About Package

Installation

Usage

Basic usage

Return genuine status code

Format

Speed-up rendering

Detect request from Pre-rendering engine during runtime

JavaScript redirects

API

Constructor new Spiderable([opts])

Configuration via env.vars

spiderable.handler(req, res, next)

AMP Support

Rendering Endpoints

Debugging

Running Tests

Node.js/Mocha

About

Topics

Resources

License

Code of conduct

Stars

Watchers

Forks

Releases 32

Contributors 4

Languages

Constructor `new Spiderable([opts])`

`spiderable.handler(req, res, next)`