Boundless opportunities with open and big data | NetEngine

Boundless opportunities with open and big data

We recently worked on an app which centered around data visualisation using Mapbox and Twitter APIs as described here. We were really thrilled to work on and create this tool but somehow I could not tell the reasons for this very specific enjoyment until I decided to really nail answering that question.

Stepping back

MapBox and Twitter APIs for the aforementioned project, Facebook or APIs for others… Could you see a common pattern?

Let’s step back a little to realise what’s going on.

“Traditional libraries” / APIs sitting on any computer allow us to take advantage of a software component already available rather than “reinventing the wheel”. A wise software developer generally prefers using a highly maintained (possibly open source) software component rather than trying to rewrite a code which is not his project’s main purpose. “Traditional libraries” hence provide code. They are some components of the engine you are building if you will.

Yet Facebook, MapBox, Twitter and many other cloud-based APIs not only provide code. In fact, in each of their relative segments, they provide us with something that we could not have built or reinvented by ourselves even if we dedicated our entire life to do so: they provide us with relevant data.

A new kind of dependency

When people were only using “traditional libraries” there could be difficulties: large amounts of disk space required, difficulty to locate all the dependencies… At its worst, this could be described as dependency hell. But if you had good practices in place to properly manage these situations, a “standard library” could sit in your desktop app and stays there almost forever with little to no maintenance required.

With APIs providing code as well as data, comes a new kind of dependency. For API providers it is extremely difficult to provide new API versions (to be able to enhance their product) and still support old API versions. They are indeed all simultaneously accessing the same underlying data. Even though some providers try to support several API versions (Saleforce is an example), those APIs will still have a given lifetime.

While with “traditional libraries” bug fixes and support may become unavailable as they age, those new “data-APIs” will simply not work anymore when at the end of their lives.

Using them therefore stirs up more maintenance; if your software replies heavily on this data its lifespan and functionalty is tied to that of the API’s.

A Fantastic Opportunity

But those APIs represent a fantastic opportunity! How many awesome applications would not have reached their final users without the mass of data only now available? Now I understand better why I loved working on those projects I described earlier: I (and our users, too) don’t need to wait for our apps / tools to grow, to become a great success, before seeing its true value in action. Data, the real fuel for our engine is there from the beginning.

And with the current Big Data trend, it’s very likely that more and more services will crop up; allowing us to collect, aggregate, analyse and display data in a way we had never dreamed of before. This fuel, unlike oil, is boundless.