Server

[deathcap]

No description provided.

[deathcap]

Several ways to do this:

1. Dedicated server running NodeJS
   https://github.com/maxogden/voxel-server
   connect with https://github.com/maxogden/voxel-client
   adds user registration and authentication: https://github.com/chrisekelley/voxel-app-server
   client/server running much of the same code, except with and without a browser
   http://www.w3.org/TR/websockets/
2. Dedicated server running something else
   e.g., https://github.com/fogleman/Craft is mainly written in C but has a simple Python server
   or "lightweight server" concept, have most logic in client, use server to get/put voxel data, dumb storage
3. Decentralized peer-to-peer
   Using http://www.webrtc.org/, http://peerjs.com/
   https://github.com/kumavis/node-warrior added in kumavis/node-warrior@b94f9bc
   Client to "host" a server, allow other client to connect

client-only (browser)
server-only (headless)
client + server (browser hosting optional server)

[kumavis]

@deathcap would love to talk about this with you
kumavis in #voxel.js on freenode

[kumavis]

I'm maintaining the voxel-client and voxel-server modules these days, but they are due for some changes, a documentation update, and a hello-world. The modules have been reworked to be agnostic to the transport medium (webrtc, websockets, locally, or whatever you can dream up). The voxel-server can now be run in a browser.

Right now the server basically only handles managing connections, sending over initial chunks, maintaining the authoritative version of the world, and forwarding a whitelist of events (eg chat). I then extend that functionality with a project-specific wrapper of the server and client that send and react to additional events.

[kumavis]

Also: If you're planning on doing multiplayer, do it early, b/c it changes the way you do everything else

[deathcap]

Cool thanks, the new voxel-client/server looks great I'll definitely look into it. I ought to get setup with IRC too, someday..

Meanwhile, have been playing around with the Minecraft protocol (https://github.com/deathcap/voxel-clientmc), figured it could help inform what might be needed in a voxel-based game client/server protocol, or how to (or how to not) do things. Used to be much worse but they completely redesigned it for MC 1.7, not too bad.

[deathcap]

Testing running index.js through node, many browser-only APIs, will require refactoring (window, XHR, document, DOM, GUI). So far, plugin loading gets this far:

voxpopuli $ node index.js|grep Inst
Instantiated plugin: voxel-engine { isClient: false,
Instantiated plugin: voxel-registry { game: 
Instantiated plugin: craftingrecipes { recipes: [],
Instantiated plugin: voxel-carry { inventory: 
Instantiated plugin: voxel-blockdata { game: 

voxeljs/voxel-modal/index.js:14
  if (!this.element) throw 'voxel-modal requires "element" option';
                     ^
voxel-modal requires "element" option

[kumavis]

The plugins could key off the isClient property on the game object (voxel-engine) to determine if they are relevant.

[kumavis]

Hmm, I guess I don't understand how you have the plugin loading set up

[deathcap]

voxel-plugins iterates over the plugin config map, require()ing and initializing each plugin. I think I'll have to add game.isClient checks in each client/server plugin (preferably if possible having common client/server functionality together in one object, extended by a client-only object; considering a design something like this: https://github.com/MinecraftForge/FML/blob/e7512f13e4b50d29378820c9c548ef265fc4fb55/src/main/java/cpw/mods/fml/common/SidedProxy.java which has a shared CommonProxy and client-only ClientProxy extending CommonProxy), should be feasible. (Note to self: not process.browser since server may be running in the browser)

Added a few more checks, current progress:

voxpopuli $ node index.js|grep Inst
Instantiated plugin: voxel-engine { isClient: false,
Instantiated plugin: voxel-registry { game:
Instantiated plugin: craftingrecipes { recipes: [],
Instantiated plugin: voxel-carry { inventory:
Instantiated plugin: voxel-blockdata { game:
Instantiated plugin: voxel-chest { game:
Instantiated plugin: voxel-workbench { game:
Instantiated plugin: voxel-pickaxe { game:
Instantiated plugin: voxel-daylight { game:

voxeljs/voxel-land/node_modules/webworkify/index.js:10
    var cacheKeys = Object.keys(cache);
                           ^
TypeError: Object.keys called on non-object
    at Function.keys (native)
    at module.exports (voxeljs/voxel-land/node_modules/webworkify/index.js:10:28)
    at Land.enable (voxeljs/voxel-land/index.js:49:17)
    at new Land (voxeljs/voxel-land/index.js:44:8)
    at module.exports (voxeljs/voxel-land/index.js:6:10)
    at Plugins.instantiate (voxeljs/voxel-plugins/index.js:64:14)
    at Plugins.scanAndInstantiate (voxeljs/voxel-plugins/index.js:49:15)
    at Plugins.enable (voxeljs/voxel-plugins/index.js:188:19)
    at Plugins.loadAll (voxeljs/voxel-plugins/index.js:140:12)
    at main (voxeljs/voxpopuli/index.js:196:13)

[kumavis]

small correction: its just game.isClient not game.isClient()

[deathcap]

fixed, thx

as for the webworker problem, found this: https://github.com/cramforce/node-worker - could be useful for supporting server-side "webworkers"

[deathcap]

Updated all plugins (except one) to at least load on server, if applicable (not necessarily functional). Some are only intended for server, and can be tagged with clientOnly: true for voxel-plugins to skip loading if on server. Others are both client and server - example of client/sever refactoring in voxel/voxel-inventory-hotbar@c363f2d - a shared InventoryHotbarCommon class to store the selected slot index (applicable to both client and server, but maybe should move the selected index to voxel-carry instead), and InventoryHotbarClient inheriting from InventoryHotbarCommon to display the GUI.

voxel-land remains a problem due to its use of web workers. The NodeJS "web worker" (API) module node-worker has been unmaintained for several years, wouldn't run on 0.10 but updated it so at least the example works at https://github.com/deathcap/node-worker. Spawns a new process for the worker. There may be better solutions nowadays (threads-a-gogo?)

[deathcap]

https://github.com/audreyt/node-webworker-threads

[deathcap]

As of 7c2bdb9 'node index.js' now loads, though doesn't do much (using https://github.com/deathcap/unworkify instead of https://github.com/audreyt/node-webworker-threads on server-side for now)

[deathcap]

Looking into this some more, WebRTC is definitely the way to go (vs WebSockets - though they can still be used for the signaling protocol). Though mainly intended for audio/video (real-time communication), it supports "data channels" as well. Uses UDP if possible for lowest latency, STUN and TURN for compatibility with various network topologies, and is easy to use with https://github.com/rtc-io/rtc-quickconnect and https://github.com/kumavis/rtc-data-stream . References:

http://www.youtube.com/watch?v=p2HzZkd2A40 Real-time communication with WebRTC: Google I/O 2013
http://www.html5rocks.com/en/tutorials/webrtc/basics/
http://rtc.io/

Another problem: how/when to load the engine. Currently, voxel-plugins loads voxel-engine as the "master" plugin, loaded very first and passed to all plugins. However, both voxel-server and voxel-client load voxel-engine themselves.

Tentative design plan:

• in browser, by default: run both client + server; local "integrated" server (in worker?) connected to by client
• multiplayer in browser: run client only, connect to a remote server
• dedicated server: run server only, no client/browser, Node environment

[kumavis]

I've been struggling with having the server in the browser in my own projects. It works fine on its own, but when in the same window with a client, it makes that tab lag really bad. I tried to put the server in a webworker, but webRtc primitives are not yet implemented in webworkers (chrome 32.0.1700.77), so I had to pipe all connections across the window/worker divide via mux-demux and workerstream. While I'm still on the first iteration, there was no speed boost b/c the major slowdown is serialization/deserialization which actually increased. One improvement would be to pass messages to the worker and back via shared objects, but I'm not sure if it would be enough.

[deathcap]

Hmm, I'm not seeing this lag. Just got voxel-server and voxel-client running together, communicating with themselves over WebRTC locally. Not using webworkers or anything, both run together. I still have the main voxel-engine instance (3 voxel-engine instances total, I need to fix this) though, so amusingly, the stats counter shows double fps:

=) presumably since two clients are running simultaneously. Still have to refactor wrt the voxel-plugins system, later, but I'll push what I have if you want to compare. One thing I'm doing differently is passing the voxel-engine module to voxel-client/voxel-server instead of having each add their own voxel-engine dependencies, but I don't think that would matter.

Is the serialization/deserialization bottleneck in JSONStream? I noticed serious performance issues with JSON in https://github.com/deathcap/voxel-clientmc - switched to sending binary data ArrayBuffers and parsing the packets on the client. Much faster than JSON decoding large amounts of binary data, at least in that particular instance.

Another idea: avoid serialization entirely (when client + server are in same browser tab) by passing the data directly? Instead of through WebRTC or webworkers. This would mean the entire server wouldn't run concurrently from the client, but specific CPU-intensive areas of the server could still be factored out into their own webworkers. For me, the most intensive server task was terrain generation - so https://github.com/deathcap/voxel-land runs it in a worker (https://github.com/rachel-carvalho/voxel-mars also uses workers similarly)

[kumavis]

I wasn't really getting the performance hit until another player joined.

For communicating locally I was using

// create virtual connection for local player
    var clientTransport = Duplex()
    var serverTransport = Duplex()
    clientTransport.on('_data', function (data) {
      process.nextTick(function(){
        serverTransport.emit('data',data)
      })
    })
    serverTransport.on('_data', function (data) {
      process.nextTick(function(){
        clientTransport.emit('data',data)
      })
    })

I would love to take a look at your multiplayer code, b/c I think this side-by-side client-server setup is important

[deathcap]

Ah, I haven't tried multiple players joining yet. I'll probably hit the same issue then. Will have to see after I fix up voxel-plugins for multiplayer. I'm thinking I'll have voxel-cs (client/server) load as the 'master' plugin, in turn loading voxel-engine.

But yeah, agreed side-by-side client/server is crucial. The way I see it, this design will force all plugins to consider client/server separately so there will be no 'singleplayer-only' plugins; everything would have to support multiplayer by design.

[kumavis]

+1 to that. It also allows people's voxel.js hacks to be no-setup and self-scaling. As long as the CDN can handle serving the static content, the game servers are created by the players themselves so you don't have to worry about A) setting up a dedicated server or B) being overwhelmed by a sudden spike.

[kumavis]

To this end, we need to create a multiplayer hello world repo.

[deathcap]

Indeed. Running into a lot of interesting issues trying to convert everything at once ;)

Think I figured out how to load client/server plugins, though. Separate voxel-plugins instance for client and server (obvious in retrospect, but..). The 'game' variable is either the client or server voxel-engine instance, depending on if loaded from voxel-client or voxel-server. Using a new module I call voxel-fuel to setup these two engines, passed all the plugin options from the top-level application to set it all up.

Several fixes in 4ade5f2 but many issues remain:

• Not receiving server settings in client, since not all voxel-engine settings are JSON-serializable (at least 'arrayType: Uint16Array') - disabled for now and 'overriding' client settings locally
• Remote avatars aren't moving - but at least they spawn:

• Network disconnection - spams errors, should abort after X failures? - Network failure causes continuous error spam deathcap/voxel-fuel#2
• Server 'join' event (for new clients) works, but couldn't seem to trigger the 'leave' event (tested by closing browser) Network failure causes continuous error spam deathcap/voxel-fuel#2
• voxel-land was altering its 'opts' parameter, then checking if it was altered - unintentionally crossing client/server. There may be other similar bugs..
• Most/all of my plugins are not truly client/server capable yet, lots of work needed
• Probably voxel-land first
• Currently using WebRTC even locally. May or may not be a problem, could switch to your "virtual connection" circuit if it is. But I kind of like having the default configuration 'server-ready', theoretically allowing anyone to join without requiring any reconfiguring first. Support local/offline gameplay (non-RTC, client=server) deathcap/voxel-fuel#4
• Not sure about the use of JSON to serialize everything (in duplex-emitter). Took a while to realize the connection was hanging because duplex-emitter was attempting to serialize a cyclic data structure ;) (the 'settings' event). There is a ton of code involved here, including to handle JSON's quirks http://timelessrepo.com/json-isnt-a-javascript-subset , newline-delimited streaming, etc. And in voxel-client :(

      // load in chunks from the server
      connection.on('chunk', function(encoded, chunk) {
        // ensure `encoded` survived transmission as an array
        // JSON stringifies Uint8Arrays as objects
        var lastIndex = Math.max.apply(null,Object.keys(encoded).map(Number))
        encoded.length = lastIndex+1

deathcap/voxel-fuel#5

wondering if it would be faster/simpler to switch to a pure binary protocol..

• Interoperability - Chrome 31 is using non-standard audio data channels (vs true data channels) -- fixed in 32
• Chat messages - integrate with voxel-console/commands - Chat messages voxel-commands#2
• Refactor the post-load steps into their respective plugins
• Host a switchboard instead of using http://rtc.io/switchboard/?
• Client-only mode (remote server) Support client-only mode (remote server) deathcap/voxel-fuel#6
• Dedicated server mode - if run on node.js (vs browser)
• Probably more I've missed

Long way to go, but its a start.

[kumavis]

Yeah I came up with a similar conclusion regarding the plugins.
https://github.com/kumavis/node-warrior/tree/master/src/js/voxel/features
Here they haven't been broken out and published, but each of those directories would be a published module that you would consume with require('my-multi-plugin/server') and require('my-multi-plugin/client'). Then pass the instance of the server or client in.
Right now voxel-server has a whitelist of forward-able event, and the plugins should add any events they need on to that whitelist (I'm manually doing this now). On the server side, there also needs to be some standardization about access to the DB.
Example: I add a plugin that lets you instantiate and program (in game) your own NPCs; their state, position, and code need to be stored in the database for the level

[deathcap]

@kumavis Sounds awesome. Any interest in compatibility with voxel-plugins?

[deathcap]

Server now basically works as of 3259094 (can open same URL+hash in another window, client joins, can see moving around) - though there is still much to be done. Filing separate issues in: https://github.com/deathcap/voxel-fuel/issues?state=open

[kumavis]

@deathcap very interested in compatibility / migrating to your plugin system, though i haven't had a chance to explore it yet. The goal for my project is to have the player build the game as they go, making their own monsters etc, so being able to enable and disable new game plugins at runtime would be a big win.

[kumavis]

@deathcap so what is your goal with vox-populi? just a testing ground for voxel.js stuff?

[deathcap]

@kumavis

@deathcap so what is your goal with vox-populi? just a testing ground for voxel.js stuff?

Essentially, yeah, I see it as a sort of 'reference implementation' including many of the latest voxel.js plugins. Wrote some more about where I'm coming from here: voxel/voxel-land#6 - but ultimately, would like to grow the voxel.js ecosystem so its a very viable platform for building sophisticated voxel-based games

[ahdinosaur]

@deathcap, is it possible to run a multiplayer voxpopuli in the current state? if so, how?

[deathcap]

@ahdinosaur It is supposed to be possible to join a multiplayer game by loading the same URL (with the #room_identifier hash at the end, which is automatically added when 'hosting' a game - i.e., by default) in another browser, however this currently doesn't work because I swapped out the rtc.io signaller for an in-memory messenger deathcap/voxel-fuel@667c3ea for development (to avoid hitting the remote server while testing). Suppose I should restore it but multiplayer is still in a very primitive state

ref #40

[deathcap]

Note: removed server for now in 216c390

[deathcap]

After sleeping on this for a while, I'm now thinking the way forward is to work towards Minecraft-protocol server compatibility. There is an actively-maintained nodejs protocol module: https://github.com/PrismarineJS/node-minecraft-protocol and command-line api client: https://github.com/andrewrk/mineflayer for MC 1.8, and I previously prototyped a websocket/minecraft proxy (last updated for MC 1.6): https://github.com/deathcap/wsmc which worked alongside a (very incipient) voxel.js plugin: https://github.com/deathcap/voxel-clientmc

The downside of this approach is it constrains functionality to match what can be expressed in the MC protocol, so kumavis' nodejs-based voxeljs-specific voxel-client/voxel-server would still be useful, separately, but for this project at least, I think the benefits primarily of compatibility with the vast MC server ecosystem is too good to pass up.

Specifically, potential interoperability with at least some of these very interesting server projects:

Project	Language	Extensibility	License	Status
Glowstone	Java	Bukkit / Glowkit API, 1000s of plugins: BukkitDev, Spigot Resources	MIT (server), GPLv3 (API)	Active
Spigot	Java	Bukkit / SpigotAPI, 1000s of plugins: BukkitDev, Spigot Resources	GPLv3 + proprietary	Mature
MCServer	C++	Lua API	Apache	Active
PrismarineJS	JavaScript	JavaScript, from devs of node-minecraft-protocol	MIT	New
Forge, Forkage	Java	Forge, FML, 100s of mods: NotEnoughMods	MFPL + proprietary	Mature
Sponge	Java	new SpongeAPI, built with assistance from Glowstone, Cauldron, Spout, Forge community (major players combining forces)	MIT	Developing
NovaCore	Java	new cross-platform NOVA API	LGPLv3	Developing

Next steps: to dust off wsmc and voxel-clientmc, update, voxel/voxel-clientmc#5, etc.

[deathcap]

New server, been working on for a bit as a break from voxel.js: https://github.com/GlowstonePlusPlus/GlowstonePlusPlus (Glowstone++, based on Glowstone as listed above, but with enhancements including switching to the Spigot-Bukkit 1.8.3 API and new features, and compatible with Bukkit2Sponge for partially supporting SpongeAPI plugins).

Glowstone++ server works with voxel-clientmc client, built on top of PrismarineJS's node-minecraft-protocol and mineflayer. Usable, can move around, place blocks, though not much else yet. The client is more incomplete, open issues: https://github.com/deathcap/voxel-clientmc/issues and server: https://github.com/GlowstonePlusPlus/GlowstonePlusPlus/issues some in https://github.com/GlowstoneMC/Glowstone/issues

[deathcap]

Ah, this old issue again. Still I think supporting a Minecraft-compatible protocol for server interoperability is the way to go. Re-examining the current landscape, there are at least three compelling open source Minecraft-compatible server projects now:

Server	Language	Extensibility Notes	License	Status
Glowstone++	Java	Bukkit / Glowkit API, 1000s of plugins: BukkitDev, Spigot Resources	MIT (server), GPLv3 (API)	Active
Cuberite	C++	Lua API, previously "MCServer"	Apache 2	Active
flying-squid	JavaScript	JavaScript, under PrismarineJS org as is node-minecraft-protocol / mineflayer	MIT	Active
Diorite	Java	Custom plugin loader, pipelines	MIT	New
McEx	Elixir, Rust	Built on Erlang/OTP	Unknown	New

The MC-compatible server ecosystem is well-covered. Compatible clients, on the other hand...:

Client	Language	Compatibility Notes	License	Status
Steven	Go	Renders blocks; "99% of features" missing	Apache 2.0	Inactive (2015)
TrueCraft	C#	"Implementation" of beta 1.7.3	MIT	Active
voxel-clientmc	JavaScript	Worked with 1.8.3 earlier, needs update to 1.8.9/1.9	MIT

[deathcap]

Update: have since updated voxel-clientmc to 1.8.9, large refactoring, updates to wsmc as well, added Forge support and can run in a Sponge plugin in addition to Bukkit or standalone. Should be relatively straightforward to keep voxel-clientmc updated for MC updates since it largely depends on mineflayer for interpreting the protocol behavior, and node-minecraft-protocol / minecraft-data for packet serialization. As long as those projects continue to be actively maintained, should be feasible to update voxel-clientmc.

Active development in voxel-clientmc-app instead of here, but could transition voxelmetaverse to be essentially a voxel-clientmc client, once there is voxel/voxel-clientmc#29 server selection UI. Until then, developing in voxel-clientmc-app because its included server-side plugins make testing with a server more convenient.

Going forward, plugins requiring client/server support e.g. voxel-workbench could be updated to integrate with voxel-clientmc as the backend voxel/voxel-clientmc#23. Plugins like voxel-land could be transitioned to server-side, perhaps recast as flying-squid plugins.