By Keith Waldorf, VP Engineering
We’re heading into a world of next-generation automation and data processing. This connected world, and the data that it generates, will force us to change the way we think about and scale cloud services.
And for the record, I do understand that Big Data and the IoT are not exactly new. But soon, the amount of data produced by billions of connected Things will render our very idea of large data sets completely irrelevant. Just imagine the volumes of data that will be generated when the connected city, the connected home and the connected car come fully online. Then, reflect on the other connected Things you carry along every day, an estimated 6 devices per person. Impressive as those numbers are, they will pale in comparison to the number of Things that will be connected in automation sectors, like healthcare, manufacturing, consumer electronics, public governance, etc.
Scott Noteboom wrote about the rise of big data in his blog, “The impending insignificance of today’s Big Data.” I agree with Scott that the data to be produced will be orders of magnitude larger than our largest data sets: 43 million times larger than Twitter’s live feed today. That’s an enormous amount of real-time data, even if Scott’s estimates are very conservative.
In today’s thinking, we push data to the “cloud” and scale, compute and storage to meet that demand centrally. Yes, we can distribute the load across multiple geographies and use graph partitioning technologies (i.e. Facebook). But what happens when connected Things, at the very edge of our networks, are transmitting data every second? And what about a low-power, low-cost Thing that only wakes up to push data or receive a policy update? Or an unpowered, batteryless Thing? Or, what if the Thing needs to understand the environment around it to make decisions? Think about a set of Things that require data correlation to trigger an action. How do you make real-time decisions based on live streams of data using smart micro-services at the edge of a network, Big Data at the edge?
At iPass, we are moving into the next generation of iPass SmartConnect™ technology with data analytics. As we’re busy implementing new solutions that process real-time analytics for global Wi-Fi connectivity, we’re being forced to rethink how and where we’re processing our data and how to build a future-proof service that can meet our customers’ growing data consumption needs. We’re currently working to determine how we move policies downstream, to the edge, to make better, smarter and faster decisions – and how to leverage these edge services to participate in policies applied to other devices at the edge.
If you think Big Data challenged our ability to scale in 2015, look out for the 2020s. It’s going to be fun.