Spatialite and GeoPackage

So I’d like to talk about some of the decisions we made in GeoPackage, as much work went in to discussing alternatives and possibilities that is not obvious from the current document. And I’m interested in opening up the dialog around specification development. This is all written as a private individual, not representing any organization or standards working group.

One of the toughest decisions was how to design the binary format for feature data. One of the most important things for me was that it would be possible for implementors to build code to create a geopackage without requiring any special SQLite stuff. The core should just be a file format. And the file should be readable by any SQLite, not one compiled with special options or additional libraries. This means the core GeoPackage is far less than SpatiaLite. Which is one of the main ‘naive’ questions about GeoPackage – why did we not just adopt SpatiaLite? SpatiaLite as a whole does not meet my core requirement: it requires a number of libraries. Indeed it is the additional libraries, it is the full spatial database, not a core format that any software can produce.

The less naive question is why did we not just use SpatiaLite’s geometry binary format, as it’s very similar to what ended up in the spec. Both more or less use Well Known Binary for the geometry, with some header information to help in parsing. This was perhaps the only vote that was not unanimous. From my traditional outsider perspective OGC seems to have a strong case of ‘Not Invented Here’ (TMS -> WMTS to me was particular painful, as it just seemed to overcomplicate things more than it needed to). I thought it would be really great to have some compatibility with SpatiaLite, to potentially have a built-in install base.

It would have been a slam dunk to go that way if SpatiaLite’s binary was completely compatible with the widely used Well Known Binary specification. They have some extra headers, but those would have been fine, as they were nice optimizations. But the key was to be able to point any existing Well Known Binary parser at the geometry portion and have it read. Unfortunately SpatiaLite made some small changes so that as a Java programmer I can’t just use JTS‘s Well Known Binary parser, and it’s a similar story in OGR. And it’s the same case for any GIS vendor that has a WKB parser. So I felt that we should favor future implementors over backwards compatibility with a binary format that has a growing but still relatively small install base.  But I’m still not sure about the decision, and may have gotten some things wrong, so I welcome technical dialog on this.

When we were freed from backwards compatibility we also got a chance to do some small innovations on the header. Most implementations we found put some sort of bounding box in their geometry, so we did the same. Paul Ramsey had a great suggestion we incorporated to have a special flag for points so we could skip the bounds, since it requires four times as much information as the point itself, so you can save significant space.

I tried to get a ‘plugfest’ going with Luciad and Esri. Pepijn and Luciad have been amazing, and even managed to release their work open source. Our crew at OpenGeo has been busy with other projects, but got a version out in GeoTools, and Justin found it quite easy to work with. Esri has been silent about any work they may be doing, so it’s not the plugfest success I was hoping for.

I do still believe a real commitment from Esri to include GeoPackage in their products (ideally not just as a mobile transfer format, but as an input and output to ArcGIS Desktop, Server and Online) would keep me committed to the current GeoPackage binary format, so I’m hoping they step up and get at least a reader and writer for tiles and features to help further QA the specification, as a great step towards opening up. But if we get a lot of compelling arguments to just use SpatiaLite I certainly can be swayed, especially if Esri doesn’t manage to get even a prototype out. I believe it should not be hard to change the core binary format in the spec.

There’s been some good conversation on github about the technical aspects of using SpatiaLite’s binary format. I’d love to hear more from developers on if matching the binary format will actually make us backwards compatible with existing SpatiaLite data packages. And on the flip side I’d be really interested if someone from the SpatiaLite community could explore implementing GeoPackage, doing what all the other spatial database implementors will do, which is just use a compatibility layer to read in and write out the GeoPackage binary, using their existing geometry blob format internally. I definitely admit I may not understand all the issues, and while I’ve been impressed by the technical discussion of the SWG it is still a small group with a limited perspective. So please sound in on the ticket, and show what’s possible with real world implementations. And if you’re an Esri customer or employee encourage them to step up and release a GeoPackage implementation, as I’d really love to have this specification go to version 1.0 with at least 3 real world implementations. Luciad, GeoTools/GeoServer and Esri would be ideal. But Luciad, GeoTools, and SpatiaLite and/or GDAL/OGR would also be great.