How do you go from raw data to actionable insights? The answer is with data processing.
The term “data processing” was first coined with the rise of computers in the 1950s. However, people have been processing data for far longer than that. From the first bookkeepers, thousands of years ago, to the the “big data” of today’s world, data is and always has been of great importance to the way our world (and our economy) is run.
But as data becomes increasingly sophisticated and complex, so do the tools, approaches, and procedures we need to process them. That’s why, in this article, I’ll explore the concept of data processing in the context of modern data analytics. We’ll go through each step of the process, from obtaining raw, unstructured data and transforming it into useful information.
If you’d like to learn more data analytics skills, try this free 5-day data short course.
We’ll cover:
- What is data processing?
- What is the data processing lifecycle? (step-by-step)
- What are some different types of data processing?
- Summary and further reading
Ready to learn everything you never knew you needed to know about data processing? Let’s dive in.
1. What is data processing?
Data processing describes the collection and transformation of raw data into meaningful information.
Once processed, this information can be used for a variety of different purposes by everyone from data scientists to business analysts, C-suite decision-makers, and IT managers, to name just a few. Regardless of the end-user or their task, the ultimate goal of data processing always remains the same. It is about turning data into information.
Within the context of modern data analytics, much of the data processing lifecycle is automated using sophisticated hardware and algorithms. Often, this is the precursor to more in-depth and hands-on data analysis, where the information gleaned is further analyzed to extract more focused and actionable insights.
Data processing is a cycle, not a once-off
Since data is constantly evolving, updating, and changing, it’s important to understand that data processing is not a standalone task. Rather, it is an iterative cycle. The cycle is constantly repeated—every time data are updated, or whenever you want to carry out a new analysis. For this reason, data processing—even using machines to streamline things—takes an awful lot of time.
Important note
It’s worth noting here that the term “data processing” is sometimes also used to describe individual steps in the overall process, as well as dedicated departments within large organizations whose function is to carry out data processing.
We’re just mentioning this in case you come across these terms in your travels. But for the sake of this article, we’ll stick to the first definition for now: data processing as a methodology.
Why is data processing important?
As we’ve already mentioned, data processing is important for transforming meaningless raw data into meaningful information for further analysis. But it has numerous other benefits, too. These include:
- More effective storage: Storing processed data in relational databases (as opposed to unstructured, text-heavy documents) makes them much easier to store, manipulate and explore using database tools like SQL.
- Easier to produce reports: Once a dataset is effectively processed, you can quickly create reports, dashboards, and other summaries of its characteristics.
- Improved productivity: By being easier to navigate, processed data saves users from having to heavily reprocess a dataset every time they want to use it.
- Sensible housekeeping: Data processing isn’t a one-off task, but an ongoing cycle. Reprocessing helps maintain order and minimizes the number of errors or mistakes that creep into your data.
- It’s more accurate: Regularly removing outliers, errors and unnecessary data points (and using clearly defined data models) increases the accuracy of your insights.
These are just a few of the reasons why data processing is important. While none of these should come as a big surprise, this hopefully illustrates just how many areas of business effective data processing can impact (beyond merely being used for data analytics tasks).
2. What is the data processing lifecycle? (step-by-step)
As we’ve already seen, data processing is an ongoing cycle, not a standalone task. In this section, we explore the different steps that make up this lifecycle. These include:
- Data collection
- Data preparation
- Data input
- Data processing
- Data output
- Data storage
Now let’s look at each of these in a bit more detail.
Data collection
The first task in data processing is to collect raw data. Straightforward as this sounds, it requires careful planning.
A common saying in data analytics (particularly in machine learning) is “garbage in, garbage out,” which means the quality of your data directly impacts the quality of your insights. You must carefully map out which data you require, where you’ll collect them from, and ensure that the source (or sources) are reliable.
It doesn’t matter how much you process erroneous data, it won’t make it any more accurate! Common sources of raw data include:
- stock market and financial data
- social media
- websites
- apps
- emails
Data preparation
Also referred to as data cleaning or data wrangling, data preparation involves tidying a raw dataset and introducing structure.
This might include removing unwanted observations, duplicates, and outliers, fixing structural and contradictory data errors, type conversion, and so on.
In reality, the exact tasks will differ depending on the nature of the data and how you intend to use them. For example, maybe you’ve collected housing data to compare house prices. If so, you might choose to remove any buyer information that isn’t directly relevant to the transaction.
Whatever your task, the ultimate goal of data preparation is to get a dataset into its best possible state before actively processing it.
To explore this step of the process in more detail, you can learn more about data cleaning in our guide.
Data input
Next up: data input. This is the process of converting raw (but tidied) data into a machine-readable format.
Once this is done, the data is then fed into a central processing unit (CPU). This could be a powerful computer with a custom-built, open-source big data architecture, or a piece of existing enterprise software.
While this step might seem straightforward, data input is as important as data collection. You should always use validated data at this stage to avoid inputting ‘garbage.’ Data input is commonly done electronically.
While it can also be carried out using scanners or manually (for smaller datasets) this is increasingly considered poor practice because it allows human error to creep in. It’s also increasingly impractical with today’s vast datasets.
Data processing
Once your data have been input into the appropriate system, they can be processed using a variety of different techniques (which we summarize in section three).
At this stage of the process, machine learning or artificial intelligence algorithms will make sense of the data, preparing to output useful information.
As a standalone task, the data processing step of the cycle can be quite time-consuming. Processing speed depends on things like your computing power, the complexity and size of the dataset, and other factors relating to the infrastructure you’re using.
Data output
Once a dataset has been processed, the results can finally be delivered to the end-user.
The format will depend on the type of data you started with and/or your preferred medium. It could include written reports, videos, images, documents, graphs, tables, and more.
At this stage, the data will have been transformed and can no longer be considered raw data. Depending on the use case, the data may be used to create dashboards, to carry out exploratory data analysis, or it could be processed further to refine relevant details.
Data storage
The final step is data storage. This is where the output is either loaded back into the system it came from or imported into another system for future use.
Storage might be in the form of a CRM or a relational database that can be queried using tools like SQL or a graphical user interface. All this depends heavily on who the data is for (e.g. is it for data scientists or corporate business leaders?) and how they intend to use or access it.
In the future, this storage facility might also be used as a source of data for another data processing cycle.
3. What are some different types of data processing?
Long ago, data processing was carried out manually without any tools (besides perhaps an abacus and wax tablet!) The introduction of early computing technologies (like calculators and bookkeeping machines) mechanized the process to a degree.
Considering the huge amounts of data that most organizations now have access to, neither of these approaches is practical today. To manually process data in the 2020s would take far too long and hugely increase the risk of errors.
Instead, the vast majority of data processing is now carried out electronically, using computers. Many high-tech systems have been designed specifically for the task (such as big data architectures like the Apache ecosystem). Compared to humans, these tools offer incredible accuracy and speed, allowing us to focus on the more important job of interpreting the data after it has been processed.
Electronic CPUs, whatever form they take, use several different techniques to process data. These are also known in computer parlance as ‘processing modes.’ While this list is not exhaustive, some common data processing modes include:
Batch processing
Batch processing is when a computer processes raw data in multiple small batches. It’s commonly used for very large, homogenous datasets.
While batches are often processed simultaneously, this approach is ideal for processing data in a cost-effective and memory-efficient way. That’s because it assigns resources as they become available. It’s often used for repetitive tasks, such as accounting, image processing, or report generation.
Real-time processing
Real-time processing is when data is processed immediately after being input into the CPU.
This is ideal when you can tolerate a short latency period (or delay) between data input and processing (the delay is usually measured in seconds or milliseconds). Real-time data processing is commonly used for smaller datasets, or when there’s a stream of ongoing data.
Examples include ATM transactions, mobile phones, and stock market data.
Online processing
Online processing is another form of real-time processing, which as the name suggests, focuses specifically on online tasks.
It describes when data accessed online are processed in real-time. This approach is commonly used for retail sales via the Internet, for example. However, it’s also often used by people throughout the retail process, such as warehouse staff.
They can use barcodes on products to track the movement of stock around a warehouse.
Distributed processing
Distributed processing is when data processing tasks are shared across a network of different computers.
In essence, this creates a single supercomputer. This is ideal for memory-intensive jobs that require the processing power and storage of an entire network.
Examples of distributed computing include things like massively multiplayer online games (MMOGs) and scientific endeavors like Folding at Home, which uses citizen computer power to unlock the secrets of protein folding in biology.
Multiprocessing
Multiprocessing (or parallel processing) is when data is processed by two or more central processors within a single computer.
The obvious benefit here is that you can carry out multiple data processing tasks concurrently. This type of data processing is commonly used by computer operating systems to carry out more than one task, e.g. playing music while running your graphic design software, as a very simple example.
While this list doesn’t cover all the processing modes, it does give a taste of them and hints at how sophisticated our use of computers for data processing has become.
The technique you use will depend on the kind of data you’re working with and the tools at your disposal. There’s no single type of data processing mode that is 100% suited to one particular task or another.
4. Summary and further reading
In this post, we’ve explored what data processing is, why it’s beneficial for the data analytics process, and what it involves.
As we’ve seen, data processing is a vital task that is as important as it is tricky to carry out! Most importantly, we’ve covered the high-level steps of the data processing lifecycle:
- Collection: Defining what raw data you require, and collecting it from a reliable source.
- Preparation: Cleaning your raw data, removing unwanted observations, and bringing structure to your dataset.
- Input: Converting the dataset into a machine-ready format and inputting it into your chosen CPU.
- Processing: Processing the data using a variety of techniques (or ‘modes’) including batch processing, real-time processing, or multiprocessing.
- Output: Outputting raw data as useful information, possibly in the form of dashboards, reports, or visualizations (or ready for further processing).
- Storage: Loading output data into your chosen storage system (e.g. your CRM) ready for end-users to access and manipulate at will.
To learn more about how data processing is important to data analytics, check out this free, five-day data analytics short course. You can also read the following introductory posts to learn more: