Processes in our Modern World
The vast majority of companies active in virtually any domain executes a process.
Whether the core business of a company is to deliver a product, e.g., manufacture a
cooking a delicious pizza, etc., or, providing a service, e.g., providing you with a
mortgage to buy your dream house, paying back your insurance claim, etc., for
efficient delivery of your product/service, processes are executed.
Hence, a natural question is: “What is a process?”.
In general, several notions of the concept of a process exist.
However, in process mining, we typically assume the following conceptual definition:
“A process represents a collection of activities that we execute to achieve a certain goal.”
As an example, consider the burger restaurant just around the corner, which also does deliveries When you call the restaurant to order your beloved burger, the first action taken by the employee, let’s call her Lucy, taking your call, is to take your order. Let’s assume you go for a tasty cheese burger, with a can of soda. After Lucy has entered your order in the cash register, she asks for your address, which she also adds to the order. Finally, she asks for your preferred means of payment, after which she provides you with a rough estimate of the time until delivery. At the moment Lucy finishes the call, she prints your order and hands it over to the chef, let’s call him Luigi. Since you’ve called relatively early, Luigi can start preparing your burger right away. At the same time, Lucy takes a can of soda out of the refrigerator and places it on the counter. At the same time, a new call comes in from a different customer, which she handles in roughly the same way as she did yours. When Luigi finishes your burger, he slides it into a carton box and hands the box over to Lucy. Lucy puts the burger in a bag and puts the can of soda next to it. She then hands the bag with your burger and soda to Mike, which uses a fancy electrical bicycle to bring your order to your home.
In this small example, let’s assume that we are interested in the process, i.e., the collection of activities performed for your order. Based on the scenario we just presented, the steps look as follows:
- Lucy takes your order
- Lucy notes down your address
- Lucy notes down your preferred payment method
- Luigi prepares your burger
- Lucy grabs your can of soda
- Luigi puts your burger in a box
- Lucy wraps your order
- Mike delivers your order
Each entry in the above list is an activity executed by an employee of the burger restaurant, in the context of your order. As such, the list represents what we typically refer to as a process instance.
Assume that, one week later, you order another burger, e.g., the BBQ-bacon burger. In this case, you just bought a back of beer, so, you decided not to order any drinks. It turns out that Lucy and Mike have a day of, hence, Randy replaces Lucy at the cash register whereas John takes care of deliveries. Let's assume that the process instance of your second order, i.e., the BBQ-bacon burger, looks as follows:
- Randy takes your order
- Randy notes down your preferred payment method
- Randy notes down your address
- Luigi prepares your burger
- Luigi puts your burger in a box
- Randy wraps your order
- John delivers your order
Do you notice some differences? Clearly, in this example, different employees (Randy/John) execute certain activities that were previously executed by Lucy. However, there are more differences. For example, Randy first asked for your preferred payment method and as a final step asked for your address. Similarly, since you did not order any soda, Randy didn't take a can of soda from the fridge.
This type of variability is naturally present in virtually any process. In some cases, variability depends on the context of a process isntance, e.g., if you do not order a can of soda, we do not need to grab it from the fridge. In other cases, variability is more natural. For example, noting down your address and/or preferred payment method can be done in any order, i.e., it does not really matter. Hence, there are two possible orders at which we are able to schedule these activities, i.e., either we note your address before your preferred payment method, or, we do it after.
Another phenomenon inherently present in processes is concurrency, i.e., the ability to execute two tasks at the same time. In our first example, note that, whilst Luigi was preparing your burger, Lucy grabbed your can of soda. Hence, these two activities are able to happen at the exact same of time. A perfect example of concurrency is a pit-stop in Formula 1. Observe that, in a pit-stop, almost every activity is literally executed at the exact same time!
In the previous section, we briefly introduced the concept of processes, variability and concurrency, using a few simple examples. Processes are virtually everywhere, i.e., almost everything we do has some notion of activities, which we execute (repetitively) to achieve some goal. Clearly, from a business perspective, understanding how processes are executed within a company is a vital first step in order to be able to get grip of the process, and, eventually, improve the process. In general, the field of Business Process Management studies methods, tools and techniques in order to achieve such aforementioned understanding of processes running at a company. So where does process mining come into play? Well, exactly at the point of understanding the process! So, what is process mining exactly?
Process mining represents a collection of tools, methods, techniques, algorithms, etc., that allows us to achieve a better understanding of the exeuction of a process, by means of analyzing the operational execution data that is generated during the execution of the process.
Beautiful definition, isn't it. Essentially, within process mining, we aim to get a better understanding of how processes are executed within a company. This is not so much different from the aforementioned field of Business Process Management, however, within process mining, a key element of gaining knowledge is the operational execution data!
Let us reconsider the previous example. Let us assume that the employees of the burger restaurant maintain a large table, e.g., an Mircosoft Excel file, in which they keep track of everything they do. Consider the following table, which represents the complete record of the first order described in the previous section.
|1337||Take Order||Lucy||April 1st 2020||1:37PM|
|1337||Note Address of Customer||Lucy||April 1st 2020||1:39PM|
|1337||Register Payment Method||Lucy||April 1st 2020||1:40PM|
|1337||Prepare Burger||Luigi||April 1st 2020||1:41PM|
|1337||Grab Soda||Lucy||April 1st 2020||1:42PM|
|1337||Put Burger in Box||Luigi||April 1st 2020||1:52PM|
|1337||Wrap Order||Lucy||April 1st 2020||1:53PM|
|1337||Deliver Order||Mike||April 1st 2020||1:55PM|
Okay, so what is captured in our table? First of all, it appears to be the case that our order got number 1337. We ordered our burger at April 1st. The first activity, i.e., taking our order was performed by Lucy, at 1:37PM. The final activity of our order, i.e., the delivery was performed around 1:55PM, quite a fast service! Whereas this example is a bit simplistic, most modern information systems actually track various parts of the execution of processes! In virtually any field, e.g., finance, health care, production technology, etc., data of the form presented are present in the underlying information systems used.
Each row in the table is referred to as an event, capturing the execution of a specific activity. We differentiate between events and activities, because in the context of some process, we are often able to repeat an activity. Hence, activities are executed, yet, events capture their execution. Within process mining, the sequence of rows depicted in the previous table, is often referred to as a trace. Hence, when we talk about traces, we talk about sequences of captured events. In the example table, all the recorded activities, i.e., the events, are listed in the context of our order. Hence, in this example, the order number allows us to identify in context of which process instance the events have been recorded. In the more general sense, the order number is referred to as the process instance identifier/case identifier. Hence, when we refer to a case (identifier), we refer to the unique way to identify in what context the events have been recorded. Examples of other process instance identifiers are a customer identifier/name, a patient identifier/name, a product ID, etc.
Imagine that we create a large table, containing all recorded traces of all orders over the past year. Such a table, e.g., containing thousands of orders, is referred to as an Event Log. Finally, there is one more difference between Table 1, and the average event log we find/extract from modern-day information systems. It is very likely that during the execution of order 1337, say at 1:42PM, another customer came in to submit an order. Hence, around 1:42PM Lucy probably started to take the order with number 1338, however, for simplicity, we have not included the activities performed for order 1338 in Table 1. In general, it is very normal that several instances of the process are executed at the same time. For example, in a bank, multiple mortgage applications run at the same point in time. Similarly, in a hospital, multiple patients are treated at the same time.
Thus far, we have seen processes, and, we have seen a simple example of an event log. However, as we have indicated, process mining is all about understanding processes. Hence, to be able to understand, and reason on, processes, we need some means to communicate about processes, i.e., means to come to a common understanding of how a specific process is expected to be executed. This is where process models come in. From an abstract point of view, a process model is simply a description of a process. The simplest form of a process model is simply a document, written in natural language, describing how the process is supposed to be executed. In the context of our burger restaurant, we could start our process description by writing “The first activity that needs to be performed for this process is the take order activity...”
The problem with describing a process in natural language is the inherent inconsistency and unclarity of natural language, i.e., different people might interpret certain statement differently. Furthermore, since process mining is a computer science discipline, we want a computer to be able to reason about and even compute with a process model. Using a process description in natural language simply does not allow us to do this. Therefore, we typically use a more mathematical notion of a process model. For example, consider the following process model, loosely describing the process executed at the burger restaurant as described earlier.
Within the figure, the leftmost circle represents the ‘start point’ of the process. Similarly, the rightmost circle represents the ‘end point’ of the process. The rectangles represent activities that can be executed in the process. The arrows in the model indicate ‘the flow’ of the process. Hence, after the starting point, the first activity to be executed is the Take Order activity. Observe that, the symbol connected to the take order activity, i.e., on the right-hand side, is a diamond shape with a +-symbol inside. This indicates the start of a parallel block of behavior, i.e., the Note Address and the Note Payment Method activities can be performed in any order, or, at the same time. Note that, the same holds for the Grab Drinks and Prepare Burger activities, however, these activities are only executed, after noting the address and noting the payment method have been completed! After this, the order is wrapped, i.e., visualized by the Wrap Order activity. After this activity, another symbol is present in the process model, i.e., a diamond shape with an x-symbol inside. This indicates that an exclusive choice needs to be made, i.e., either one of the two connecting activities needs to be executed. Hence, the order is either delivered or the customer will pick up his/her order.
Observe that, the process model does not capture the full behavior of the process as described in the first section, i.e., it is a simplified model of the process as described, just to highlight the concept of process models. Note that, using the model, we can reason what sequences of behavior, are supposed to happen in the process, i.e., as described by the model. For example, the sequence <Take order, Note Address, Note Payment Method, Grab Drinks, Prepare Burger, Wrap Order, Deliver> is a sequence that, according to the model, should be executable in the process. Similarly, the sequences <Take order, Note Address, Note Payment Method, Grab Drinks, Prepare Burger, Wrap Order, Wait for pickup > (Wait for Pickup instead of Delivery) and <Take order, Note Payment Method, Note Address, Grab Drinks, Prepare Burger, Wrap Order, Deliver > (Note Address and Note Payment are swapped), are part of the behavior of the process, i.e., according to the provided model. Consider the following image, taken from, Process Mining: Data Science in Action; Wil M.P. van der Aalst (2016), page 69, in which all the core elements of the Business and Process Modeling Notation are listed.
A detailed explanation of different (business) process modeling notations is out of the scope of this getting started page, i.e., business process modeling is an interesting field in its own right (both from an academical as well as an industry perspective). For convenience, we refer to the seminal work Fundamentals of Business Process Management, by Dumas et al. (2018), as well as The Application of Petri Nets to Workflow Management, by Wil M.P. van der Aalst (1998), highlighting the use of Petri nets (a more mathematically rigour notation compared to BPMN) in the context of business process modeling. Again, Process Mining: Data Science in Action; Wil M.P. van der Aalst (2016), serves as a perfect reference of well.
Let us briefly recapture the three concepts described in the previous three sections, and, their relation.
- Processes; The execution of activities, with some ordering among these activities, in order to achieve a (business) goal, e.g., assembly of a product, providing a service, etc.
- Event Log; A historical collection of data, capturing the execution of a process. In a way, it is a digital snapshot of what has happened in the past, i.e., how the process has been executed before.
- Process Model; A model (ranging from natural language to a mathematical model) describing how a process is supposed to behave.
Let’s get straight into it. One of the most fundamental challenges in process mining is process discovery. The goal is simple: a process discovery algorithm takes an event log as an input, and, returns a process model. Ideally, such a process discovery algorithm does this completely automatically, or, semi-automatically. In the context of our burger restaurant example, assume we have been logging all the orders over the past month, and, we have constructed a large event log. If we give this large event log to a process discovery algorithm, ideally, the algorithm discovers a model like the process model depicted in the previous section, i.e., where we introduced process models. Several process discovery algorithms have been developed, yielding various different types of process models. A few of these have been implemented in pm4py.
If you are reasonably farmiliar with Business Process Management, you might know the term 'process discovery' already. Often, BPM consultancy firms apply process discovery in a manual fashion. Various stake-holders that are responsible for certain parts of the process will be interviewed. Based on these interviews, a consultant typically manually creates a process model. However, such a process model typically describes an idealized view of the process. Typically stake-holders will (knowingly or not) describe only a small fraction of the true behavior of the process. This is where process discovery (i.e., in the process mining sense) comes in. The process model, i.e., discovered on the basis of the logged event data shows us what actually happened, rather than what is perceived to have happened.
In conformance checking, we assume that we already have a process model that accurately reflects the intended process behavior. Such a process model is either drawn by hand, i.e., by a process expert, or is the result of applying a process discovery algorithm on an event log. Note that there are various use cases in which checking whether the execution of a process is in line with a reference model. For example, the reference model expresses certain rules or regulations dictated by policy makers. Alternatively, a specific sequence of activities might be required to handle a complaint correctly.
There exist various ways to compute whether or not a given process model and event log are compliant w.r.t. each other. However, an elaborate discussion of these types of techniques is far out of scope here. The main take-away w.r.t. conformance checking is the following. Assume that we observe the trace of process behavior, depicted in Table 2, for our burger restaurant:
|1338||Take Order||Lucy||April 1st 2020||1:42PM|
|1338||Note Address of Customer||Lucy||April 1st 2020||1:44PM|
|1338||Register Payment Method||Lucy||April 1st 2020||1:45PM|
|1338||Register Payment Method||Lucy||April 1st 2020||1:46PM|
|1338||Prepare Burger||John||April 1st 2020||1:51PM|
|1338||Grab Soda||Lucy||April 1st 2020||1:47PM|
|1338||Put Burger in Box||John||April 1st 2020||1:57PM|
|1338||Deliver Order||Mike||April 1st 2020||2:00PM|
Conformance checking techniques allow us to compute that, in this example, sadly, Lucy has registered the payment method twice, and, the order was not wrapped! Additionally, conformance checking techniques allow us to quantify (numerically) the number of deviations observed in an event log.
TLDR; Key Take-Aways
Process Mining is a collection of data-driven algorithms/methods/techniques/methodologies, with the main aim to better understand processes. The core idea of process mining is to acquire said understanding by analyzing the operational event data that was generated and stored during the execution of the process under study. Such operational event data is often referred to as an Event Log. As such, an event log is one of the main input artefacts of any process mining analysis.
There is a variety of analyses one can perform, using an event log as an input. Two of the most common use cases in process mining (there are many more) are:
- Process Discovery;(Semi)Automated techniques that translate the event data into a Process Model that accurately describes the process, i.e., as captured in the event log.
- Conformance Checking;Automated techniques that allow us quantify to what degree a process, i.e., as captured in an event log, is conforming to a process model, specifying the intended behavior of the process.
Finally, what does pm4py have to do with all of this? pm4py is simply a python library that implements a wide variety of process mining algorithms!