Empty tile files for zoom 12

[speller]

When using the default config with just turned off compression tilemaker produce all tile files in the 12 folder with zero size. Look at the screenshot: (Sorry for non-English interface)

First red box: 0 bytes, second: 720 files, 40 filders.

Why???

[speller]

As I can see - the reason it that only the "pois" layer is rendered. Even when I run tilemaker with these config scripts:

{
	"layers": {
		"water": { "minzoom": 1, "maxzoom": 14 },
		"roads": { "minzoom": 10, "maxzoom": 14, "simplify_below": 13, "simplify_level": 0.0001, "simplify_ratio": 2.0 },
		"buildings": { "minzoom": 14, "maxzoom": 14 },
		"pois": { "minzoom": 13, "maxzoom": 14 },
"public_transport": {"minzoom": 12, "maxzoom": 14},
"landuse": {"minzoom": 1, "maxzoom": 14, "simplify_below": 13}
	},
	"settings": {
		"minzoom": 1,
		"maxzoom": 14,
		"basezoom": 14,
		"include_ids": false,
		"name": "Tilemaker example",
		"version": "0.1",
		"description": "Sample vector tiles for Tilemaker",
		"compress": "none",
		"metadata": {
			"json": { "vector_layers": [
						{ "id": "roads",     "description": "roads",     "fields": {"name":"String","type":"String"}},
						{ "id": "water",     "description": "water",     "fields": {}},
						{ "id": "buildings", "description": "buildings", "fields": {}},
						{ "id": "public_transport", "description": "public_transport", "fields": {}},
						{ "id": "landuse", "description": "landuse", "fields": {}}
					] }
		}
	}
} 

-- Nodes will only be processed if one of these keys is present

node_keys = { "amenity", "shop", "highway", "railway", "building", "aerialway", "aeroway", "barrier", "emergency", "historic", 
"public_transport", "tourism", "waterway", "landuse" }

-- Initialize Lua logic

function init_function()
end

-- Finalize Lua logic()
function exit_function()
end

-- Assign nodes to a layer, and set attributes, based on OSM tags

function node_function(node)
	local amenity = node:Find("amenity")
	local shop = node:Find("shop")
	if amenity~="" or shop~="" then
		node:Layer("pois", false)
		if amenity~="" then node:Attribute("type",amenity)
		else node:Attribute("type",shop) end
		node:Attribute("name", node:Find("name"))
	end
end

-- Similarly for ways

function way_function(way)
	local highway = way:Find("highway")
	local waterway = way:Find("waterway")
	local building = way:Find("building")
	local railway = way:Find("railway")
	local aeroway = way:Find("aeroway")
	local aerialway = way:Find("aerialway")
	local public_transport = way:Find("public_transport")
	local landuse = way:Find("landuse")
	if highway~="" then
		way:Layer("roads", false)
		way:Attribute("name", way:Find("name"))
		way:Attribute("type",highway)
--		way:Attribute("id",way:Id())
--		way:AttributeNumeric("area",37)
	end
	if waterway~="" then
		way:Layer("water", false)
	end
	if building~="" then
		way:Layer("buildings", true)
	end
	if railway~="" then
		way:Layer("public_transport", false)
	end
	if aeroway~="" then
		way:Layer("public_transport", false)
	end
	if aerialway~="" then
		way:Layer("public_transport", false)
	end
	if public_transport~="" then
		way:Layer("public_transport", false)
	end
	if landuse~="" then
		way:Layer("landuse", true)
	end
end 

What I do wrong?

[speller]

@systemed could you assist please?

[sfkeller]

Having same issue here...

[systemed]

It's working fine for me but I'm not using the default config. Let me have a look and see if I can reproduce it.

[systemed]

No problem for me using default config:

./tilemaker --input ~/Maps/planet/oxfordshire-latest.osm.pbf --output ~/Maps/planet/ox_test/

results in:

Can you give exact steps to reproduce? (command line invocation, pastebin/gist of config files if changed, link to the .pbf extract being used)

[mnboos]

It worked with the default config. However, there was a layer (poi) that was missing in the vector_layers array of the json row in the metadata table but was present in the data. How can this happen?

[mnboos]

Apologies, forgot to mention the data I used:

I downloaded the latest pbf for Switzerland:
http://download.geofabrik.de/europe/switzerland-latest.osm.pbf

Unfortunately, the extract has already been updated again since. In case that should be relevant, I uploaded the pbf from yesterday: https://we.tl/rftpQnTX8W

[systemed]

The metadata entry isn't parsed by tilemaker at all, it's just passed through to the output in the .mbtiles file. It's just there for clients (well, Mapbox GL) that require it. You can put anything in there you like, it won't affect what happens with the data.

Thanks for the Switzerland hint - I'll download that and take a look. @sfkeller, what extract were/are you using?

[mnboos]

I understand, that tilemaker doesn't create the metadata tables content, but in my opinion that's wrong, as there are several entries which are required in the metadata table. See the specification: https://github.com/mapbox/mbtiles-spec/blob/master/1.1/spec.md

[systemed]

I think you've slightly misunderstood things (...or maybe I have!).

tilemaker does write the required values. It writes rows like this to the SQLite metadata table:

name    | value
--------+----------------------------
name    | <whatever you put in settings.name>
version | <whatever you put in settings.version>
etc.

These are not the same as the 'metadata' hash stored in your config file as settings.metadata.

settings.metadata is a freeform hash that tilemaker simply writes into the metadata table, with json in the name column, and the JSON itself in the value column. tilemaker does not parse or act on this data in any way. This is not part of the mbtiles spec, but appears to be required by some Mapbox implementations, so tilemaker simply provides this option for compatibility. I asked for clarification on this, inter alia, in mapbox/mbtiles-spec#43 almost three years ago, but none has been forthcoming. The mbtiles spec has issues that desperately need fixing (see mapbox/mbtiles-spec#46 and mapbox/mbtiles-spec#47) and this is one of them! (FWIW, my view is strongly that this is a peculiarity of one particular implementation and should not be included in the spec.)

If you are not using a Mapbox vector tiles reader then you do not need to put anything in settings.metadata.

(If there's anything more to add, could I suggest that discussion of this would be best in another issue, and leave this one for the empty z12 tile problem?)

[sfkeller]

My comment above was related to @mboos. I'm advising his very promising "Vector Tiles Reader" QGIS plugin (http://plugins.qgis.org/plugins/vector_tiles_reader/ ) and I'm actually testing e.g. with Uster City (Switzerland) using OSMaxx: https://osmaxx.hsr.ch/media/osmaxx/outputfiles/147bebc1-593c-47d4-a958-8c655206ae9e/uster-county_wgs-84_2017-06-11_pbf_full-detail.zip (zipped PBF, 1 MB).

Our ultimate goal is to implement an MVT reader which is as compatible as possible with MVT spec as well as with existing MVT generators like tilemaker.

I see, that tilemaker just guides some settings (like JSON) through to the respective JSON entry of the mbtiles metadata table.
=> But could'nt tilemaker produce the field names entries automatically from the input source anyway?

I also realize that JSON field is implementation-specific to Mapbox' use of mbtiles, and this (specificly mapbox/mbtiles-spec#43, mapbox/mbtiles-spec#46 and mapbox/mbtiles-spec#47 ) should be really fixed by Mapbox.
=> But how else would a reader (like such a QGIS data provider) be able to prepare a layer before importing the MVT data?

(I don't mind splitting this issue/discussion)

[systemed]

I'll do another issue for the Mapbox JSON field.

On the empty z12 tiles, I can't reproduce that in any form. Creating an .mbtiles and looking at the z12 data always results in tiles with data in:

• with switzerland.osm.pbf from @mnboos' download, default config;
• with the latest oxfordshire.osm.pbf from Geofabrik, default config;
• with the uster-county .osm.pbf from @sfkeller, default config;
• with the same extract and with @speller's config.

Really happy to try and help debug this but I need a reproducible test case - .osm.pbf, Lua script, json config and command line invocation.

[TimSC]

Several robustness fixes have been added to the master branch. Could people check to see if it resolves this issue?