Spatialite and GeoPackage

So I’d like to talk about some of the decisions we made in GeoPackage, as much work went in to discussing alternatives and possibilities that is not obvious from the current document. And I’m interested in opening up the dialog around specification development. This is all written as a private individual, not representing any organization or standards working group.

One of the toughest decisions was how to design the binary format for feature data. One of the most important things for me was that it would be possible for implementors to build code to create a geopackage without requiring any special SQLite stuff. The core should just be a file format. And the file should be readable by any SQLite, not one compiled with special options or additional libraries. This means the core GeoPackage is far less than SpatiaLite. Which is one of the main ‘naive’ questions about GeoPackage – why did we not just adopt SpatiaLite? SpatiaLite as a whole does not meet my core requirement: it requires a number of libraries. Indeed it is the additional libraries, it is the full spatial database, not a core format that any software can produce.

The less naive question is why did we not just use SpatiaLite’s geometry binary format, as it’s very similar to what ended up in the spec. Both more or less use Well Known Binary for the geometry, with some header information to help in parsing. This was perhaps the only vote that was not unanimous. From my traditional outsider perspective OGC seems to have a strong case of ‘Not Invented Here’ (TMS -> WMTS to me was particular painful, as it just seemed to overcomplicate things more than it needed to). I thought it would be really great to have some compatibility with SpatiaLite, to potentially have a built-in install base.

It would have been a slam dunk to go that way if SpatiaLite’s binary was completely compatible with the widely used Well Known Binary specification. They have some extra headers, but those would have been fine, as they were nice optimizations. But the key was to be able to point any existing Well Known Binary parser at the geometry portion and have it read. Unfortunately SpatiaLite made some small changes so that as a Java programmer I can’t just use JTS‘s Well Known Binary parser, and it’s a similar story in OGR. And it’s the same case for any GIS vendor that has a WKB parser. So I felt that we should favor future implementors over backwards compatibility with a binary format that has a growing but still relatively small install base.  But I’m still not sure about the decision, and may have gotten some things wrong, so I welcome technical dialog on this.

When we were freed from backwards compatibility we also got a chance to do some small innovations on the header. Most implementations we found put some sort of bounding box in their geometry, so we did the same. Paul Ramsey had a great suggestion we incorporated to have a special flag for points so we could skip the bounds, since it requires four times as much information as the point itself, so you can save significant space.

I tried to get a ‘plugfest’ going with Luciad and Esri. Pepijn and Luciad have been amazing, and even managed to release their work open source. Our crew at OpenGeo has been busy with other projects, but got a version out in GeoTools, and Justin found it quite easy to work with. Esri has been silent about any work they may be doing, so it’s not the plugfest success I was hoping for.

I do still believe a real commitment from Esri to include GeoPackage in their products (ideally not just as a mobile transfer format, but as an input and output to ArcGIS Desktop, Server and Online) would keep me committed to the current GeoPackage binary format, so I’m hoping they step up and get at least a reader and writer for tiles and features to help further QA the specification, as a great step towards opening up. But if we get a lot of compelling arguments to just use SpatiaLite I certainly can be swayed, especially if Esri doesn’t manage to get even a prototype out. I believe it should not be hard to change the core binary format in the spec.

There’s been some good conversation on github about the technical aspects of using SpatiaLite’s binary format. I’d love to hear more from developers on if matching the binary format will actually make us backwards compatible with existing SpatiaLite data packages. And on the flip side I’d be really interested if someone from the SpatiaLite community could explore implementing GeoPackage, doing what all the other spatial database implementors will do, which is just use a compatibility layer to read in and write out the GeoPackage binary, using their existing geometry blob format internally. I definitely admit I may not understand all the issues, and while I’ve been impressed by the technical discussion of the SWG it is still a small group with a limited perspective. So please sound in on the ticket, and show what’s possible with real world implementations. And if you’re an Esri customer or employee encourage them to step up and release a GeoPackage implementation, as I’d really love to have this specification go to version 1.0 with at least 3 real world implementations. Luciad, GeoTools/GeoServer and Esri would be ideal. But Luciad, GeoTools, and SpatiaLite and/or GDAL/OGR would also be great.

Githubbing the GeoPackage specification

In the past few months one of the main things I’ve spent time on is the new GeoPackage specification being worked on within the OGC. I was involved in the very early days of the conception, before it was picked up for the OWS-9 testbed, as I feel it has the potential to fill a big hole in the geospatial world (James Fee sums up my feelings nicely). We discussed keeping it super lean, accessible to all, and implementable by non-GIS experts.

While I got wrapped up in other things a spec for public comment popped out in January, which I felt had strayed from what I believed were the original core design goals. I sent a huge missive on all the changes I felt could get it back to that lean core. I try to never be one to just criticize though, as it’s easy to cast stones and much harder to build something that is strong enough to hold up to others pelting rocks. And my open source roots teach me that a criticism without offering to put in effort to the alternative is close to worthless. So I decided to join the GeoPackage ‘Standards Working Group’, participating in weekly (and then twice a week) calls, and trying to work with the OGC workflow of wikis and massive Word documents.

One of my main goals was to learn about how the OGC process actually works, and hopefully from a place of knowledge be able to offer some suggestions for improvement from my open source software experience. That’s worth a whole post on its own, so I’ll hold off on much of that for now.

OGC staff has also been great about being open to new ways of working. It’s had to be balanced against actually getting the specification out, as many agencies need it ‘yesterday’. But we achieved an OGC first, putting this version of the specification out on GitHub. My hope is to attract developers who are comfortable with GitHub, who won’t need to learn the whole OGC feedback process just to suggest changes and fixes.

I do believe the specification is ‘pretty good’ – it has improved in almost all the ways I was hoping it would (Pepijn and Keith drove the tech push to ‘simpler’, with Paul doing a great job as Editor and Raj from OGC helping out a bunch). I do believe GeoPackage can improve even more, primarily by feedback from real implementations. My hope is it does not go to version 1.0 without 3 full implementations. That don’t just handle one part of the document, but implement all the core options (particularly Tiles and Features).

For the OGC this is an experiment, so if you do think OGC specifications could benefit by a more open process you can help encourage them by spending some of your technical brainpower on improving GeoPackage using GitHub (big shout out to Even who already has a bunch of suggestions in). Fork the spec and make Pull Requests, on core tech decisions or on editorial tweaks that make it easier to understand. For those without GitHub experience we wrote some tips on how to contribute without needing to figure out git). We also tried to add a mechanism for ‘extensions‘ to encourage innovations on the GeoPackage base, so I’d also love to see attempts to use that and extend for things like Styles (SLD or CSS), Rasters, UTFGrids and Photo Observations.

And if you can’t fully implement it but are interested in integrating, I do encourage you to check out the Apache licensed libgpkg implementation from Luciad. Pepijn has been the technical core of GeoPackage, researching what’s come before and continually building code to test our assumptions. Libgpkg is the result of his work, and the most complete implementation. Justin also has a GeoTools implementation and GeoServer module built on it for the Java crowd. And hopefully someone will implement for GDAL/OGR and PostGIS soon.

As always I’ve written too much before I’ve even said all I wanted to. But I will try to do at least one follow up blog post to dig in to some of the details of GeoPackage. I and the SWG welcome an open dialog on the work done and where to take it next. And I’m happy to explain the thinking that got the spec to where it is, since we wanted to keep the spec itself slim and not include all that detail. We are hoping to eventually publish some ‘best practice’ papers that can dig in to ideas we considered yet discarded as not quite ready.

Opening Esri

So I’ve been meaning to write a post on this ever since I had a great talk with Andrew Turner, who recently joined Esri. He was expressing a good bit of optimism over Esri’s willingness to embrace ‘open’. My worry is that they would embrace the language of open, but not actually do the key things that would make a real open ecosystem. It’s easy to take enough action to convince more naive decision makers of their intentions, like, which looks like lots of open source projects, but is mostly code samples and api examples that are useless without pieces of closed software. So I wanted to give to Esri a measurable roadmap of actions to take that would signal to me a real commitment to ‘open’.

That was a couple months ago, but then life got in the way, as tends to happen with lots of planned blog posts. But in the meantime there’s been a fascinating flare-up around the GeoServices REST specification in the OGC (most conversation is behind OGC walls, but see open letters and lengthy discussion on OSGeo). And it similarly makes me want to address ‘Esri’ as an entity. The individuals matter less, I believe that most of them want to do the right thing. But the corporation as a whole takes action in the world, and that big picture matters more than what individuals within are trying to do. So here goes:

Dear Esri,
I imagine you’ve been pretty confused and flummoxed by all the push back to the GeoServices REST specification. It is a pretty solid piece of technology, and theoretically is a great step towards opening up Esri more – putting previously proprietary interfaces in to an open standard.
The key thing to understand though is that unfortunately very few people in this industry trust you. I am actually fairly unique in that I give you a whole lot more benefit of the doubt than most. In my life I try as hard as I can to only judge people based on their direct interactions with me, not on whatever people say about them. And though you aren’t exactly a person I do still try to judge in the same way. And I personally have never had a truly bad interaction with you.
But amazingly just about everyone that I’ve encountered in the broader geospatial industry has. When I bring up that I’ve never been mistreated by you they have the utmost confidence that I inevitably will. I have always been fascinated by this, as I believe it goes far beyond ‘typical competition’, which is one thing people at Esri have told me – something like ‘they’re just jealous’. The amount of venom that otherwise totally decent people will throw out is incredible, and it makes it hard for me to not judge harshly, when so many people whose judgement I trust absolutely have felt really screwed in the past by you. Multiple times. In really unpleasant and unexpected ways. And it’s not just competitors, but partners and clients.
So the fundamental point that needs making with regards to GeoServices REST was articulated well by one of my allies: ‘everyone in the OGC membership is objecting to the REST API because they believe ESRI to be fundamentally conniving and untrustworthy.’ He believes that if the REST interface had been proposed by anyone else then there wouldn’t be a problem – it would likely be an approved OGC standard by now. But people believe it’s a ‘cynical business play by an untrustworthy, well resourced, and predatory business interest.’
I have overall had a relatively positive take on the GeoServices REST API, because I don’t (yet?) believe you to be fundamentally conniving and untrustworthy. You’ve done an amazing amount good for this industry, but I do think you’ve just lost your way a bit. I imagine you think that putting this great interface in to OGC is a great step in the right direction, and it is, but unfortunately it’s too much too fast. It’s as if the schoolyard bully is suddenly super nice to you – you wonder what’s up his sleeve, how he’s going to screw you this time. 
But I believe there’s a path forward, to building up people’s trust so that something like GeoServices REST could be accepted in OGC. It just has to be slower and more incremental. This list is just what I can think of right now. There is a fundamental principle at work though, which is moving towards an architecture that encourages people to mix and match Esri components with other technology. Not merely implementing open standards to check the box, but building for hybrid environments: QGIS exporting to or editing ArcGIS Server, ArcGIS Desktop doing the same to GeoServer, TileMill styling a file geodatabase and publishing to ArcGIS Online, ArcGIS Desktop styling and publishing to Google Maps Engine or CartoDB, etc, etc. Each piece of Esri technology ideally could be used stand alone with other pieces. Stated another way, there should be no lock-in of anything that users create – even their cartography rules. Anything that is created with Esri software should be able to be liberated without extreme pain, in line with the Data Liberation Front  (though I think the google maps engine and earth enterprise teams also need som help from them, they too deserve a similar letter).
I realize this is a big leap, since it is not the absolutely most efficient way to solve the needs of most of your customers, since most of them use only your software. And it is a business risk, since it opens up more potential competition. But it’s also a big business opportunity if done right. And reaches beyond mere business to being a real force for good in the world, becoming a truly loved company, with lots of friends. This list is roughly ordered by easier wins first, and so later ones should probably build on them. If these actions are taken I will start to defend you to my friends a lot more.
  • Enable W*S services by default in ArcGIS Server. You’ve done a pretty great job of doing CITE certified implementations. But as the docs show they are not enabled by default, though GeoServices REST and KML are.
  • Make GeoJSON an output option for the ArcGIS Server REST implementation (and get the clients all reading it). And then get it in the next GeoServices REST specification version.
  • Publish the file formats of file geodatabase. The open API was a really great step, and indeed goes 80% of the way. But many libraries would like to not have to depend on it. We all know the format itself is likely an embarrassing mess, but everyone will forgive, as we’ve all written embarrassing code and formats.
  • Help the coming GeoPackage be the next generation of file geodatabases. Help it evolve to be the main way Esri software moves data between one another. Your team has been great on it so far, but the key step is to make it a top level format not just in mobile but also on ArcGIS Desktop (reading and writing it, like a shapefile), Server and Online (as an output format and upload configuration format for both). And I’m hoping your team can join our plug-fest this week, to get to 3 real implementations before version 1.0 of the specification.
  • Stop the marketing posts about how open you are. Open is action and conversation, not a press release or a case study. No need to talk about how we’ll be ‘surprised’ by what you open in the future. Just start opening a lot and let others do the posts for you, and count on that reaching your customers if you’re doing it well.
  • Openly publish the .lyr file format (and any other formats that define cartography and internal data structures for maps). This is probably an internal format that isn’t pretty, but opening it would enable a lot of cool hybrid architectures.
  • Support CartoCSS as an alternative to .lyr file, and collaborate around advancing it as an open standard to ideally make it the next generation .lyr file. This likely will include a number of vendor extensions as your rendering capabilities are beyond most everyone else. But we can all collaborate on a core. Supporting SLD in desktop would also be great, but we can also just look to the future.
  • Move WFS and GML support out of ‘data interoperability‘ extension and in to core format support
  • Bring support for WFS-T to ArcGIS Desktop, so people can use their desktop to directly edit a WFS-T server. The server side support in ArcGIS Server is great, but the client software also needs to implement the open standards for real interoperability. I think a similar thing might be needed for Desktop to edit directly with GeoServices REST, though maybe that is there now.
  • Support WFS-T in Javascript API and iOS SDK (and I guess flex and silverlight, since you tend to try to have all toolkits the same).
  • Become a true open source citizen. This is another large topic, and as is my tendency I’m already going on too long. So perhaps I will detail it more in another post. I have limited the above to mostly standards, but embracing open source can take you even further. But it is much more about contributing to existing open source libraries and building a real community of outside contributors on your projects, not just releasing lots of code on github. Karl Fogel wrote an excellent book on the subject. You are making some progress on this, but it is just a smidgen if you are actually serious about opening up. I will give props for the flex api, though my cynical friends would say it’s the easiest one to open source as it is dying. So please, do continue to surprise us on the open source front, you just don’t need to talk about it a bunch.
So I believe that doing a majority of these will send a strong signal that Esri is truly changing to be more open and that the submission of the GSR spec and putting code on github is more than a marketing move.
My personal recommendation for you on the GeoServices REST specification is to back down from the OGC standardization for now. Instead work to evolve the spec in the open, and try again in a year or two. I don’t think any argument is going to win over those against it, only real action will. And continuing to try to force it through will only hurt our global community. I do believe there has been really great progress made on the spec through the OGC process, and the resulting document has a much higher chance of being implemented by others. Perhaps we could make it an ‘incubating’ OGC standard, that evolves with OGC process, but does not yet get the marketing stamp. The key to me is to encourage real implementations of it, and continue to advance Esri’s implementation as well, but in dialog with other implementers. Everyone can now start with the latest version, but without the backwards compatibility requirement. Call it GeoServices REST 1.0 – it can be a standard that improved from the OGC process though needs more ‘incubation’ before it deserves the full stamp of interoperability. And aim for 1.1 or 2.0 to happen outside the OGC, and once there are 3 solid implementations all influencing its future then resubmit to OGC.
One thing that I think could help a lot is to publish and work on the spec on github, where people can more easily contribute. Take a more agile process to improving it, that doesn’t depend on a huge contentious vote and lots of OGC process. And when the trust has grown all around submit it to be an official OGC document. I believe this would also be one of the best signals to the wider world of Esri’s commitment to open, to dialog publicly and then iteratively address the concerns of all objectors in a collaborative process. If done right it will fly through the OGC process the next time, or else by then we will all be working together towards the next generation APIs that combine the best of GeoServices REST and W*S, while cutting down complexity. It will be difficult, but will ensure that the geospatial industry remains relevant as the whole world becomes geospatial. So let’s all be as open as we can to figure out how to get there together. 
Chris Holmes

(Written as a private citizen, who cares about our collective future and believes geospatial has a higher role to play than just an ‘industry’. Views expressed here do not reflect my employer or any other organizations I am associated with)


come in, we're open

Social Business thoughts

A month or two ago I came across an amazing piece by Muhammad Yunus, where he introduces the ‘Social Business’. I do hope the meme builds momentum, and I’m hoping my ‘dot-org‘ concept can grow to be the technological sector of the social business world. It’s a great narrowing of the term ‘Social Enterprise‘, which I feel has almost been too broadly adopted so as to become less meaningful. I’m looking for a more narrow focus – entities that are fundamentally both in service of a mission and operated to draw income from the market, not from charity. He articulates this better than I’ve seen before, with his full cost recovery social business – leaving behind the dependency of a charity, one enters the business world with ‘limitless possibilities’.

I especially liked his points about a Social Stock Market, that it needs to be a set of separate concepts, with different measures and media outlets. But I find his ideas on how to get there a bit weak – having a design competition that rewards the best ones with funding. He also suggests that someone soon could just hatch a Social Stock Market, and a little research found that Rockefeller is already moving to fund one. But I’m fearful that it’s too much too soon, and that a bunch of not so great social enterprises will give it a bad name.

Though maybe it would just be a sector of the Social Stock Market over time, one thing I’ve been thinking about, especially as OpenGeo progresses towards full cost recovery, is making it so money generated by social businesses is viral. The current thought with OpenGeo is that relatively soon we’ll spin it off in to its own company, fully owned by the foundation (TOPP), just like the Mozilla Corporation. Most profit will then go back to TOPP (with a portion towards employee profit sharing), which is a not for profit and by definition has to reinvest the money back in service of its mission. We’ve been thinking about what ‘outside investment’ might look like, and I’m pretty sure we’d want it to operate like the money our funder is putting in now – all returns must go back in service of our mission. But I’m thinking that outside investors could then have control over which TOPP initiatives their returns could go towards, and could choose to direct a new project, leveraging a team of programmers and designers from TOPP. If there was a social stock market, however, it would make perfect sense that their returns could go to other ventures that also guarantee to put their returns back on the alternate market.

Thus money would operate like source code with the GPL – it only helps those who agree to the same set of principles, towards a commons that can be used by others with the same values. I like this because I think it cuts a nice middle ground between non-profit charity giving and the social enterprise investment now that has little way of knowing how much investors actually do care about the profit bottom line. I suppose the money could also go back in to non-profits that are more geared towards charity. This would help foundations allow to make their endowments work for good, instead of just having the capital in traditional investments. But I’d hope that instead of just returning back to foundation endowments it boot straps social venture capital firms and incubators and more capital in service of social businesses. And thus shares in the social stock market (or at least sector thereof) is a real alternative with teeth, where success breeds further capital for more success, instead of just a nice idea.

I have not read any wider literature on these things, so this may be very naive, or an idea already tried, but it sounds potentially cool to me, and it’s really just trying to extend the proto-model we have at TOPP. But I hope to read more on where things are at with social enterprise and where they can move forward, as we’ve mostly been operating in a vacuum.

So within the GeoServer community we’re debating whether it’s kosher to do a fairly blatant ‘commercial’ announcement on the blog about But in the meantime I figured I’d just announce here, since I can do whatever I want on my personal blog 🙂

I’m really excited to present OpenGeo, the newly minted geospatial division of The Open Planning Project. Nothing much is changing internally, but we’re getting serious about our image in the world. We’ve been supporting open source geospatial projects for years, and in the past couple years we’ve offered great consulting services around the projects we work on. But it’s always been confusing for people who don’t already know our work. They might see an email address on the lists, and follow that to web tools for community organizers, click from there on The Open Planning Project logo, linking to a high tech non-profit, then maybe click projects, and see ‘GeoServer’, which they’ve been using, and from there click on ‘services’ to realize that yes, you can in fact pay us money to work on this. I’m not convinced that anyone made it that far.

So is about giving a more visible face to our services and products, so we can bring the geospatial work in TOPP to economic sustainability with full cost recovery. It also marks the launch of ‘GeoServer Enterprise‘ packages, which bundle web and telephone support, priority bug fixes, discount consulting rates, and a number of implementation hours by the experts. This is on the full OpenGeo stackOpenLayers, GeoWebCache and GeoServer. We’re hoping this makes it easier for more conservative organizations to embrace open source by establishing something much closer to a traditional vendor relationship. It should provide a clear answer to the classic question with open source ‘Who do I call when something goes wrong?’ – with GeoServer Enterprise you call us, the experts who wrote the software. We believe the total package is incredibly competitive with any offering, proprietary or open source, for the level of service and expert access provided – you’re not paying for the intellectual property of something we built in the past, you’re paying directly for our time now. We already have two clients on board, which is very exciting. The website will be improved with demos and more content, but we’re quite pleased with what we’ve got. Recently many contracts have been coming in, and the OpenGeo launch should help us grow even more. So if you’re looking for a job for a geo dot-org drop me a line, we’re definitely hiring all types of roles, and soon should update the website with specifics.

Letting everyone remix web maps

I’ve been meaning to lay down my little vision of how web mapping configuration should work for awhile.  It’s not a big idea, but a nice little way to perhaps bring more participation in to the creation of web maps, making it possible for anyone to make a mash-ups.  I believe Google Mapplets gets at a lot of this, but I’d like to add on standards based services that can bring in real GIS layers and restyle the base maps.

I should start with a view either of a decent base map (like a nice blue marble layer) or else a view that someone else composed.  I can browse around, or I can go to ‘edit mode’, to configure more of what I want to see.  From there I can specify a WMS, a WFS/WCS, a GeoRSS feed or a KML.  If there are multiple layers on the service I should be able to specify which one I want to add.  I can add in the various services, and maybe even add some annotations.  Once I compose a view that I like I can hit ‘save’, and it’ll save the map to my user account, or else export it out as an OWS Context document.  I should also be able to say ’embed’ and have it auto-create the OpenLayers javascript (pointing at the Context document created), that I can put on my blog or webpage.  My annotations will be done as inline KML in the context document.

Next, if the services are WFS/WCS or WMS that allows remote SLD I can change the styles of the layers I brought in.  I might want to emphasize one layer more than others, or even just a particular feature of the layer.  I am able to easily do thematic mapping as well, to specify what attribute to do a quantile or equal intervals on, with the colors I choose.  This remote SLD can be saved to the Context document directly, so I don’t even need permissions on a server.  If I have permissions on the WMS server I can then upload the new SLD as an option for others.

Past that I can get better performance on the map I configured by pointing the configuration at a GeoWebcache or TileCache instance that I have set up or have appropriate rights on.  I can completely drive configuration of GeoWebcache through the web UI, to set which layers to turn on and off.  I can even start it seeding the layers, or have it expire part, based on bounding box and number of zoom levels.

Then if I have a GeoServer set up, or at least layer creation rights on a server, I can start creating new data through WFS-T.  This is beyond annotations: I can create a new layer that persists on the server, centralizing its editing.  This is important because I can also set the edit permissions so other users can start editing the same layer.  I can also choose to turn on versioning, to keep a history of what users make what edits.  I can control the other user permissions, opening it up to everyone or just a select few.  All data I create is available as WMS/WFS/WCS, KML, GeoRSS, GeoJSON ect.

I can also upload a shapefile or geotiff right through the web interface.  Using the same styling utilities I can configure it to look as I like and make that the default style.  I can choose to turn caching on there too.  I can invite others to start editing the shapefile I created.  I can also always export it as a shapefile, or any other format.  All these more advanced layers are also saveable as a ‘view’, as an OWS Context document.

Whenever I put the javascript map anywhere it will always have a ‘configure the map’ button, for others to start to remix.  Once they hit the ‘configure the map’ button they get a nice listing of the layers and data that make up the map.  They can restyle and change layers to their own liking with no permissions.  And if they want to start caching or adding data they can get rights on a server, or just set up their own server.  Through this I hope we can make it incredibly easy for anyone to create their own map, and add a sort of ‘view source’ to maps that anyone can use, not just javascript programmers.  I feel this gets beyond mere ‘wizards’, which take you through a set path to add the portions you want, to a more flexible remixing environment that can hopefully encourage more lightweight maps that aren’t hard to create.  I believe this should also be the way the majority of people configure their GeoServers, though we should also have a more advanced admin interface to change all the little settings.

In terms of technology, we’re actually quite close to this vision.  It really leverages a lot of the standards architecture, utilizing OWS Context, Remote SLD and WMS/WFS.  We need to add the REST admin part of Geowebcache, but the ability to upload a shapefile or GeoTiff is now in a REST interface for GeoServer, thanks to the work of David Winslow.  We just need a front end in openlayers to edit it.  For OWS-5 we did inline KML in Context documents.  Versioning is also in place, though we need a bit more work on granular security permissions, though the framework is there. And we have funding for an ajax SLD editor, which should be the final major piece of this.   Then we just need to iterate through making the UI incredibly intuitive and easy to use.  But I’m excited for this vision, to bring the power of ‘real’ GIS – large datasets and styling of basemaps – to the easy to use style of the GeoWeb.

That’s all for me for now.  Two blog posts in the span of a less than a week, I guess this means I don’t have to post again for months 😉

Slides from GSDI-10

Just finished up a great week at GSDI-10 in Trinidad. I came down with Justin Deoliveira; he put on a great workshop on Monday, and we both had talks on Thursday. Sometime soon we should post the workshop modules on the GeoServer blog, but I wanted to get my talk online as several people asked about it. It was a fun talk, bringing together a lot of what I spent my time thinking about in Zambia around SDI’s, open source and Google Maps/Earth. The conference is great, because it gathers many of heads of national mapping agencies and other important players who work with lots of data and really want to figure out the best ways to share it. I think my talk was really well received, the timing felt right for the message: that we need to figure out how to combine what’s going on in the Google Maps/Virtual Earth world with the broader GSDI initiative in to a single GeoWeb. I hope to turn the thoughts I presented in to a paper at some point, but in the meantime my too literal slides do get across much of my argument (sometime I’ll learn how to make visually compelling presentations…)

You can also download  the powerpoint. Reading the slides you may notice the very soft launch of, which is going to be our rebranding of the geospatial division of The Open Planning Project, so it’s more clear that we offer services and support packages around GeoServer and OpenLayers. Still lots of work to be done, but we were excited to at least get a start for this conference. Once we get it ready for launch we’ll announce for real.

Oh, and the license for the slide show is Creative Commons Attribution 3.0.
Creative Commons License

Recycling to help maintain performance and stability

So I’ve rarely looked at ESRI software, but a friend has administrator rights on the new version of ArcGIS Server, and he decided to show me how it works.  There are a lot of options, and they provide some interesting configuration of how the processes work:

ArcGIS Server

They have this great ‘feature’ called Recycling – ‘Recycling shuts down the process and restarts it at regular intervals to help maintain performance and stability’.

Words escape me.

Ok, not really (when have they ever?), since I want to explain this to people who don’t write software.  This ‘feature’, to me, is their development team admitting defeat.  They couldn’t iron out all the memory leaks and little code errors that might get in the way of a well performing, stable, robust server.  When you mess up your program in that way about the only thing that can fix it is to completely restart the whole thing.  So what this ‘feature’ does is make it so your server automatically restarts itself at the interval you set, the most blunt possible approach to the problem there is.

Now I’m not saying we’ve never had memory leaks in GeoServer, on the contrary.  Indeed we had to delay 1.6.0 and do another release candidate because we found one that only built up over a long time when users were requesting really large maps.  All I’m saying is that to add something like this is to admit defeat, to say that they don’t even necessarily plan to fix all the little bugs that are causing this.  And more than anything I just love that the double speak, playing it as a feature, to ‘help maintain performance and stability’.   Say what you will about their software, they remain absolutely brilliant marketers, and I have a ton to learn from them in that realm.

Great post on GeoData licensing

I have some major posts to write, but thankfully there are others out in the world thinking about and writing about the same issues that I care about.  Today I found ‘The license: where are, where we’re going‘ on the OpenGeoData blog.  It is exactly the post I wanted to write after attending the Science Commons meeting last year, and then goes on to add even more information that I wasn’t even aware of.  It’s really great news to hear about progress being made on the Open Data Commons Database Licence.  I really hope they continue to work on it, since I had the same reaction as many OSM community members to the news of Creative Commons’ recently published “protocol” on open access data – it’s great for scientists, but doesn’t help much with what we’re doing in the geospatial community.  So thanks Richard for post and the great work you are all doing over on the OSM legal list.

Producing, not consuming, makes the GeoWeb go ’round.

Aaron on Streetsblog just passed me a link to post by Steven Johnson on what he calls ‘the pothole paradox‘. It’s a very nice introduction to GeoWeb ideas, and clearly explains to laypeople why geospatial metadata on everything could help. But for us working on building this open, interoperable GeoWeb it offers very little, except that their stovepipe system is going to attempt to solve it.

Why do I call a stovepipe? Because as it stands now it only sucks information in. He asserts that to be successful their system must be structured with two spatial properties:

1. Every single piece of data in the system has to be associated with some kind of machine-readable location, as precisely as possible.

2. Where appropriate, data should be associated with a human-readable place (a bar, school, playground, park, or even neighborhood.)

I would maintain that these are true for the broader GeoWeb, a system of systems. And I’m sure would love it if the entire web was structured that way, and they would be the one who aggregates the results. Unfortunately they are in a position to potentially do something about it, and as far as I can find they just want it all in their own system.

Why do I draw that conclusion? Mostly because there’s no way to access the geo information that their community is encoding. They have a number of easy options to tag location on blog posts, including the GeoRSS standard that many of us are promoting. And they let people submit non-located stories and easily add location. They are actively adding that meta geospatial data layer, and then keeping it all for themselves as far as I can tell. Why do I say that? Because as far as I can tell there’s no machine readable output of the geospatial locations that they’re adding. If I subscribe to a feed of stories about Brooklyn then there’s no GeoRSS tags. Which means that if I want to make a service that mashes up the geotagged stories from I have to rewrite all their mechanisms for getting that spatial information.

Yes, this is arguably their core ‘value add’, since they have the community that adds all the geospatial metadata. thus would love it if the rest of the web had GeoRSS on everything, and then their users were the ones who got the additional information added to those that the authors did not tag. But if everyone thinks in this way then we’re basically stuck in a prisoner’s dilemma.

Thankfully the big players in the Geo Web have actually been getting out of this in the last couple years. Almost two years ago at a Location Intelligence conference GeoRSS was all the buzz. Microsoft and Yahoo! both announced they would support it, and there was a lot of pressure on Google to do so as well. But all seemed utterly unaware that the key is not to be able to read GeoRSS, it’s to output it by default whenever someone uses their API. Google wasn’t yet doing anything with GeoRSS, still trying to make KML the one true format, but they suffered from the same thing – if you used the Google Maps API it would not output any machine readable format, KML or otherwise.

At the time I stood up and asked the question, and it was weird how something so obvious to me was so hard to explain to them. They thought that just being able to read a standard was the most open thing they could do. But basically none of them were offering their information up to others to read. They all had users who were creating new geospatial information through their API’s, but it wasn’t contributing to ‘the’ GeoWeb, it was just making their own little web better. Each seemed to want to ‘own’ the web, which would constrain it to a bunch of competing projects, instead of a new public infrastructure like the internet. The latter web is the one they all wanted, and why they’re investing so hugely, but none wanted to blink, to be the one to cooperate since the other might defect.

Thankfully the past couple years have seen some great progress on this. I wrote about Google’s progress after Where 2.0, which is impressive as they are the ones in the lead. Microsoft has arguably done even more, they were the first to take up my call to produce GeoRSS from their native ‘Collections’ API. They’ve also launched the ability to search KML (and GeoRSS I think/hope?) and easily import KML, now that it is becoming an open standard. This might be surprising to some, but one should remember that those who are behind always rush to standards. Microsoft IE3 actually was the first full commercial implementation of CSS2. We’ve seen a parallel in the geospatial world, with ESRI talking much more about standards, getting their services to implement OGC standards for real, instead of just paying lip service. It looks like they are also embracing KML.

On Google’s side, their GeoSearch is now crawling GeoRSS, which is great (and would be able to crawl’s aggregrated feeds if they published them). Unfortunately looking just now it appears that their push to get people to make their Google Maps using KML or GeoRSS is no longer a priority. For awhile they were pushing this in their talks, like at the last Google Dev Day. It makes a ton of sense that it’s in their interest to, since more data produced that way means more data for their GeoSearch. But the documents appear to barely mention it. On the other hand MyMaps has everything that users create as KML, and that is arguably more important because presumably people who just need to throw up some points and lines will use that. The more complicated uses of the Maps API would need more than KML or GeoRSS anyways. MyMaps could of course be more open, if it were to produce GeoRSS as well. Yahoo!’s apis seem to have made no progress towards producing GeoRSS, which is unfortunate as they were the first to support reading it.

Ok, this is longer than I had intended (but that’s why I have a blog, so I don’t have to feel bad about writing too much, dammit!).  But I hope that starts producing machine readable feeds of the awesome geospatial tags their community is gather, to help lead the growth of the geospatial web instead of just waiting to capture the hyperlocal part of it for their own sandbox.  For the GeoWeb to truly succeed we’re going to need a huge number of organizations taking that leap towards cooperation over ownership.  Thankfully the big guys seem to be providing decent leadership, I’d give them a solid ‘B’ on their progress the last few months (though Microsoft might even deserve an A).