spiderable is part of Webapp. It's
one possible way to allow web search engines to index a Meteor
application. It uses the AJAX Crawling
specification
published by Google to serve HTML to compatible spiders (Google, Bing,
Yandex, and more).
When a spider requests an HTML snapshot of a page the Meteor server runs the client half of the application inside phantomjs, a headless browser, and returns the full HTML generated by the client code.
In order to have links between multiple pages on a site visible to spiders, apps
must use real links (eg <a href="/about">) rather than simply re-rendering
portions of the page when an element is clicked. Apps should render their
content based on the URL of the page and can use HTML5
pushState
to alter the URL on the client without triggering a page reload. See the Todos
example for a demonstration.
If you deploy your application with meteor bundle, you must install
phantomjs (http://phantomjs.org) somewhere in your
$PATH. If you use meteor deploy this is already taken care of.
When running your page, spiderable will wait for all publications
to be ready. Make sure that all of your publish functions
either return a cursor (or an array of cursors), or eventually call
this.ready(). Otherwise, the phantomjs executions
will fail.
(This fork of spiderable changes URL handling so that users going to
foo.com/?spiderable-nonsense are routed to foo.com rather than ROOT_URL
in cases where foo.com isn't the ROOT_URL. There is some suggestion in
Meteor documentation that ROOT_URL intentionally has this downside, but
it prevented a product I was working on from being spiderable, so I've
made this change. I also changed the phantom_script.js file, adapting it
from [http://www.meteorpedia.com/read/spiderable](the Meteorpedia example)
because the original one didn't work for me.)