In a way, big data is exactly what it sounds like - a huge amount of data. Since the advent of the Internet, we've been producing data in staggering amounts. According to Berkeley researchers, we are now producing roughly 5 quintillion bytes (or around 4.3 exabytes) of data every two days.
There's nothing new about the notion of big data, which has been around since at least 2001. The term 'big data' is usually refer to massive, rapidly expanding, varied and often unstructured sets of digitized data that are difficult to maintain using traditional databases. It has launched a veritable industry of processes, personnel and technology to support what appears to be an exploding new field. It can include all the digital information floating around out there in the ether of the Internet, the proprietary information of companies with whom we've done business and official government records, among a great many other things. There's also the implication that the data is being analyzed for some purpose.
We've generated lots of data by making online purchases and participating in social media any other online activity. Big data can include digitized documents, photographs, videos, audio files, tweets and other social networking posts, e-mails, text messages, phone records, search engine queries, RFID tag and barcode scans and financial transaction records, though those aren't the only sources.
Besides home computers and retailers' point-of-sale systems, we have Internet connected smartphones, WiFi-enabled scales that tweet our weight, fitness sensors that track and sometimes share health related data, cameras that can automatically post photos and videos online and global positioning satellite (GPS) devices that can pinpoint our location on the globe, to name a few.
How does Big data help to solve our day to day problems?
Big data has to be collected, massaged, linked together and interpreted for it to be of any use to anyone. Companies and other entities need to filter the vast amount of available data to get to what's most relevant to them. Modern hardware and software process, store and analyze huge amounts of information and find the best solution. Some of the software is becoming more user friendly so that it doesn't necessarily take a team of programmers and data scientists to wrangle the data.
Interpretation of Big Data can bring about insights which might not be immediately visible or which would be impossible to find using traditional methods. This process focuses on finding hidden threads, trends, or patterns which may be invisible to the naked eye.
Much of this big data processing and analysis is aimed at finding patterns and correlations that provide insights that can be exploited or used to make decisions. Businesses can now mine massive amounts of data for information about consumer habits, their products' popularity or more efficient ways to do business. Big data analytics can be used to target relevant ads, products and services at the customers they believe are most likely to buy them, or to create ads that are more likely to appeal to the public at large. Companies are now even starting to do things like send real-time ads and coupons to people via their smartphones for places that are near locations where they have recently used their credit cards.
In 2012, the Wall Street journal ran an article describing how Netflix used Big Data to build out their streaming video service. They were able to analyze traffic details for various devices, spot problem areas and add network throughput to help prepare for future demand. Netflix was also able to get more insight into the type of content customers preferred, which enabled them to make more accurate suggestions as to what subscribers might like.
In essence, big data allows entities to use nearly real-time data to inform decisions, rather than relying mostly on old information as in the past. But this ability to see what's going on with us in the present, and even sometimes to predict our future behavior, can be a bit creepy.
We give up more information than we realize to companies with whom we do business, especially if we use loyalty cards or pay with credit or debit cards. Someone can learn a lot about you just from analyzing your purchases. People can tell lots about you from this data, including your age, gender, sexual orientation, marital status, income level, health status, tastes, hobbies, habits and a whole host of other things that you may or may not want to be public knowledge. They need only have the means and the will to gather and analyze it. And whether they mean well or ill, it can have unintended consequences.
Like anything, big data can be used for good, for ill, and for lots of stuff in between. It always depends on Government and company how they will use.