The rise of big open data

By Chi-Loong Chan | | Data

The following article is an original unpublished piece on the rise of open data initiatives around the world. – Chi-Loong

We are living in the age of data.

In fact, we are drowning in it.

According to Digital Universe 2020, a report by analyst firm IDC that came out in December 2012, the digital universe will grow by a factor of 300 from 2005 to 2020. The data universe is the amount of data created, replicated and consumed in a single year.

The report, which was sponsored by storage company EMC, estimates that we will hit 40 ZB (Zettabytes) by 2020, a doubling of the data universe ever year from 2012, which was estimated to be around 2 ZB.

To put things in perspective, 40 ZB is 40,000,000,000,000,000,000,000 bytes, or 40 billion terabytes. If the entire digital universe was downloadable in 2020, it would still take a person over 50 million years to download (assuming fast fiber rates of 200mbps).

Of course, the pertinent question is then: What do we do with all this data?

Therein lies the burgeoning trend of Big Data, and making sense of this deluge with data science.

Data science – a big, emerging science that encompasses disparate fields like statistics, data analytics and data-modeling – hopes to use data to understand the trends of the past, make sense of the present and predict the future.

Big data holds big potential for businesses and governments. Even though there is a lot of hype about this field – inevitable for a fast, nascent booming field – there is also a lot of real potential.

According to IDC’s latest Big Data report, they expect spending in the Asia Pacific region on Big Data technologies to reach US$1.02 billion in 2014, a 36.3 percent compound growth over 2013.

IDC expects growth rates to hold strong over the next 4 to 5 years, with a compound growth rate of 34.1 percent through 2017.

The future is open

There are many reasons for Big Data’s quick ascent:

  • The rise of the always-on 24/7 consumer, where services are expected to be available at any time of the day.
  • The proliferation of mobile devices and tablets, which make it easy for these consumers to connect to the virtual world.
  • Cheaper storage and sensor technology costs, which allows more data to be tracked and stored.
  • The rise of the data platform economy, where increasingly the business’ most valuable assets is its data, even more so than its services. For example, think about all the IT powerhouses in recent years. Facebook (social media), Google (search), and Amazon (e-commerce) make their money by understanding their consumers and targeting them correctly, all through the use of data.
  • The rise of social media. Through platforms like Facebook, Twitter and YouTube, we are creating reams of content like never before. On YouTube alone we are uploading 100 hours of video every minute.
  • The ascent of the cloud, which allow us to easily share, create and procure data without caring where our information is physically located.
  • Advances in many fields of data science, from in-memory analytics, Hadoop distributed computing, to data visualization and modeling.

One interesting subtrend of the Big Data space to emerge in recent years is the rise of open data.

Open data is data that can be freely used, reused and redistributed by anyone, subject only to attribution and sharealike policies, if any.

Open data shares a similar philosophy with open-source technology and creative commons. By allowing people free access to the resource, it invites participation and involvement.

The organizations with the biggest interest in open data initiatives are governments around the world, which should come as no surprise.

Governments have some of the largest repositories of data, but most importantly, much of government data is public by law.

Open government data can be used to create value in many areas:

  • Improved transparency and democratic control
  • Increased citizen participation and self-empowerment
  • Improved efficiency and effectiveness of government services
  • Improved or new private products and services
  • Innovation
  • Impact measurement of policies

For example, the US White House in December 2013 just recently announced their continued commitment to open data initiatives for improved transparency and citizen participation.

In Singapore, the Infocomm Development Authority (IDA) is spearheading open data through its data portal data.gov.sg and geospatial data portal OneMap, which were set up in 2011.

Said Singapore Deputy Prime Minister and Minister for Finance, Mr Tharman Shanmugaratnam, at the eGOV Global Exchange Forum in June 2013: “By opening up more data, and through innovative use of technology, we can crowd-source ideas and co-create applications with the wider community.”

In fact, many governments around the world are committing to open data policies. The non-profit Open Data Index has documented 70 countries around the world with open data policies as of 2013.
Even though the trend is still new, there are already real-life examples of how open data solves difficult real world problems and even save lives.

For example, in the recent super typhoon Haiyan that struck Philippines in November 2013, more than 400 volunteers made three quarters of a million additions to free online map OpenStreetMap.

Those additions reflect the land before the storm, but this will help Red Cross workers and volunteers make critical decisions on where to send food, water and supplies, saving lives in the process.

In Africa, where food security is an issue, initiatives like Global Open Data for Agriculture and Nutrition (Godan) might just save lives. Godan was launched in October this year, and aims to share genomic data on agriculture to farmers.