Opening Esri

So I’ve been meaning to write a post on this ever since I had a great talk with Andrew Turner, who recently joined Esri. He was expressing a good bit of optimism over Esri’s willingness to embrace ‘open’. My worry is that they would embrace the language of open, but not actually do the key things that would make a real open ecosystem. It’s easy to take enough action to convince more naive decision makers of their intentions, like, which looks like lots of open source projects, but is mostly code samples and api examples that are useless without pieces of closed software. So I wanted to give to Esri a measurable roadmap of actions to take that would signal to me a real commitment to ‘open’.

That was a couple months ago, but then life got in the way, as tends to happen with lots of planned blog posts. But in the meantime there’s been a fascinating flare-up around the GeoServices REST specification in the OGC (most conversation is behind OGC walls, but see open letters and lengthy discussion on OSGeo). And it similarly makes me want to address ‘Esri’ as an entity. The individuals matter less, I believe that most of them want to do the right thing. But the corporation as a whole takes action in the world, and that big picture matters more than what individuals within are trying to do. So here goes:

Dear Esri,
I imagine you’ve been pretty confused and flummoxed by all the push back to the GeoServices REST specification. It is a pretty solid piece of technology, and theoretically is a great step towards opening up Esri more – putting previously proprietary interfaces in to an open standard.
The key thing to understand though is that unfortunately very few people in this industry trust you. I am actually fairly unique in that I give you a whole lot more benefit of the doubt than most. In my life I try as hard as I can to only judge people based on their direct interactions with me, not on whatever people say about them. And though you aren’t exactly a person I do still try to judge in the same way. And I personally have never had a truly bad interaction with you.
But amazingly just about everyone that I’ve encountered in the broader geospatial industry has. When I bring up that I’ve never been mistreated by you they have the utmost confidence that I inevitably will. I have always been fascinated by this, as I believe it goes far beyond ‘typical competition’, which is one thing people at Esri have told me – something like ‘they’re just jealous’. The amount of venom that otherwise totally decent people will throw out is incredible, and it makes it hard for me to not judge harshly, when so many people whose judgement I trust absolutely have felt really screwed in the past by you. Multiple times. In really unpleasant and unexpected ways. And it’s not just competitors, but partners and clients.
So the fundamental point that needs making with regards to GeoServices REST was articulated well by one of my allies: ‘everyone in the OGC membership is objecting to the REST API because they believe ESRI to be fundamentally conniving and untrustworthy.’ He believes that if the REST interface had been proposed by anyone else then there wouldn’t be a problem – it would likely be an approved OGC standard by now. But people believe it’s a ‘cynical business play by an untrustworthy, well resourced, and predatory business interest.’
I have overall had a relatively positive take on the GeoServices REST API, because I don’t (yet?) believe you to be fundamentally conniving and untrustworthy. You’ve done an amazing amount good for this industry, but I do think you’ve just lost your way a bit. I imagine you think that putting this great interface in to OGC is a great step in the right direction, and it is, but unfortunately it’s too much too fast. It’s as if the schoolyard bully is suddenly super nice to you – you wonder what’s up his sleeve, how he’s going to screw you this time. 
But I believe there’s a path forward, to building up people’s trust so that something like GeoServices REST could be accepted in OGC. It just has to be slower and more incremental. This list is just what I can think of right now. There is a fundamental principle at work though, which is moving towards an architecture that encourages people to mix and match Esri components with other technology. Not merely implementing open standards to check the box, but building for hybrid environments: QGIS exporting to or editing ArcGIS Server, ArcGIS Desktop doing the same to GeoServer, TileMill styling a file geodatabase and publishing to ArcGIS Online, ArcGIS Desktop styling and publishing to Google Maps Engine or CartoDB, etc, etc. Each piece of Esri technology ideally could be used stand alone with other pieces. Stated another way, there should be no lock-in of anything that users create – even their cartography rules. Anything that is created with Esri software should be able to be liberated without extreme pain, in line with the Data Liberation Front  (though I think the google maps engine and earth enterprise teams also need som help from them, they too deserve a similar letter).
I realize this is a big leap, since it is not the absolutely most efficient way to solve the needs of most of your customers, since most of them use only your software. And it is a business risk, since it opens up more potential competition. But it’s also a big business opportunity if done right. And reaches beyond mere business to being a real force for good in the world, becoming a truly loved company, with lots of friends. This list is roughly ordered by easier wins first, and so later ones should probably build on them. If these actions are taken I will start to defend you to my friends a lot more.
  • Enable W*S services by default in ArcGIS Server. You’ve done a pretty great job of doing CITE certified implementations. But as the docs show they are not enabled by default, though GeoServices REST and KML are.
  • Make GeoJSON an output option for the ArcGIS Server REST implementation (and get the clients all reading it). And then get it in the next GeoServices REST specification version.
  • Publish the file formats of file geodatabase. The open API was a really great step, and indeed goes 80% of the way. But many libraries would like to not have to depend on it. We all know the format itself is likely an embarrassing mess, but everyone will forgive, as we’ve all written embarrassing code and formats.
  • Help the coming GeoPackage be the next generation of file geodatabases. Help it evolve to be the main way Esri software moves data between one another. Your team has been great on it so far, but the key step is to make it a top level format not just in mobile but also on ArcGIS Desktop (reading and writing it, like a shapefile), Server and Online (as an output format and upload configuration format for both). And I’m hoping your team can join our plug-fest this week, to get to 3 real implementations before version 1.0 of the specification.
  • Stop the marketing posts about how open you are. Open is action and conversation, not a press release or a case study. No need to talk about how we’ll be ‘surprised’ by what you open in the future. Just start opening a lot and let others do the posts for you, and count on that reaching your customers if you’re doing it well.
  • Openly publish the .lyr file format (and any other formats that define cartography and internal data structures for maps). This is probably an internal format that isn’t pretty, but opening it would enable a lot of cool hybrid architectures.
  • Support CartoCSS as an alternative to .lyr file, and collaborate around advancing it as an open standard to ideally make it the next generation .lyr file. This likely will include a number of vendor extensions as your rendering capabilities are beyond most everyone else. But we can all collaborate on a core. Supporting SLD in desktop would also be great, but we can also just look to the future.
  • Move WFS and GML support out of ‘data interoperability‘ extension and in to core format support
  • Bring support for WFS-T to ArcGIS Desktop, so people can use their desktop to directly edit a WFS-T server. The server side support in ArcGIS Server is great, but the client software also needs to implement the open standards for real interoperability. I think a similar thing might be needed for Desktop to edit directly with GeoServices REST, though maybe that is there now.
  • Support WFS-T in Javascript API and iOS SDK (and I guess flex and silverlight, since you tend to try to have all toolkits the same).
  • Become a true open source citizen. This is another large topic, and as is my tendency I’m already going on too long. So perhaps I will detail it more in another post. I have limited the above to mostly standards, but embracing open source can take you even further. But it is much more about contributing to existing open source libraries and building a real community of outside contributors on your projects, not just releasing lots of code on github. Karl Fogel wrote an excellent book on the subject. You are making some progress on this, but it is just a smidgen if you are actually serious about opening up. I will give props for the flex api, though my cynical friends would say it’s the easiest one to open source as it is dying. So please, do continue to surprise us on the open source front, you just don’t need to talk about it a bunch.
So I believe that doing a majority of these will send a strong signal that Esri is truly changing to be more open and that the submission of the GSR spec and putting code on github is more than a marketing move.
My personal recommendation for you on the GeoServices REST specification is to back down from the OGC standardization for now. Instead work to evolve the spec in the open, and try again in a year or two. I don’t think any argument is going to win over those against it, only real action will. And continuing to try to force it through will only hurt our global community. I do believe there has been really great progress made on the spec through the OGC process, and the resulting document has a much higher chance of being implemented by others. Perhaps we could make it an ‘incubating’ OGC standard, that evolves with OGC process, but does not yet get the marketing stamp. The key to me is to encourage real implementations of it, and continue to advance Esri’s implementation as well, but in dialog with other implementers. Everyone can now start with the latest version, but without the backwards compatibility requirement. Call it GeoServices REST 1.0 – it can be a standard that improved from the OGC process though needs more ‘incubation’ before it deserves the full stamp of interoperability. And aim for 1.1 or 2.0 to happen outside the OGC, and once there are 3 solid implementations all influencing its future then resubmit to OGC.
One thing that I think could help a lot is to publish and work on the spec on github, where people can more easily contribute. Take a more agile process to improving it, that doesn’t depend on a huge contentious vote and lots of OGC process. And when the trust has grown all around submit it to be an official OGC document. I believe this would also be one of the best signals to the wider world of Esri’s commitment to open, to dialog publicly and then iteratively address the concerns of all objectors in a collaborative process. If done right it will fly through the OGC process the next time, or else by then we will all be working together towards the next generation APIs that combine the best of GeoServices REST and W*S, while cutting down complexity. It will be difficult, but will ensure that the geospatial industry remains relevant as the whole world becomes geospatial. So let’s all be as open as we can to figure out how to get there together. 
Chris Holmes

(Written as a private citizen, who cares about our collective future and believes geospatial has a higher role to play than just an ‘industry’. Views expressed here do not reflect my employer or any other organizations I am associated with)


come in, we're open

8 thoughts on “Opening Esri

  1. Oh, and forgot to give an important shout-out to Arc2Earth. Brian’s already built most all of the ArcGIS Desktop half of the ‘open’ story, just need to figure out how to get it shipping with every Desktop piece of software.

  2. Of course the most brilliant part was: “…they too deserve a similar letter.”. Can you write a similar one to every single GIS brand? I do believe that Open is an action more than press (I couldn’t agree more by the way), although there is no perfect world. Some of your suggestion can bring up a well known question: How to earn money with open source project? Become a consultant. Let’s move forward to build a better future.

  3. This isnt going to happen. You are barking up the wrong tree. ESRI about creating a cult not getting work done. They are still harping about location and trying to move clients into a cloud/server pay as you go model.

    The group we need to go after for standards are people who consider location as part of their everyday data anyways. We still need data standards for services and data sharing, but lets face it, the database should be your GIS for 90% of the use cases. The remaining happens with Grass/Qgis/Sextanate with R.
    Question is, how do you tell this story? How do you make someone realize that purchasing ArcView is just a G in the GIS (you have to add an additional license and extension to get the IS), and they really dont need it anyways?

  4. Pingback: Top 10 Improvements I’d Like to See in ArcGIS 10.2 for Desktop #8: Opening Esri | PolyGeo
  5. Pingback: Spatialite and GeoPackage | Into The Pudding

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s