5 ways to utilize Camunda more efficiently in your microservice architecture
Use cases for workflow engines
When talking about workflow automation, companies and business people usually think about human task management and task lists. And that makes sense since workflow engines are a great tool for that. However, workflow automation is actually much more. Modern workflow engines are lightweight and can operate high volumes at low latency. This makes them applicable for every problem that requires a state machine, whether the main driver of the requirement at hand is business or IT. Other motivating factors for using workflow engines also include graphical visibility which is important in case of long-running workflows and them being persistent state machines that help with tasks of versioning process models, scheduling or operation control.
What is Camunda
The Camunda BPM engine is an open-source framework that provides a business process management (BPM) system for any kind of organizational workflow. It is Java-based and can be embedded inside any Java application or used as a standalone process engine server. It provides REST API for non-Java developers and those who prefer it that way. The Camunda Modeler desktop application helps to model BPMN process diagrams and DMN decision tables. It also contains several web applications like those that allow end-users to work with tasks assigned to them (Tasklist) or allow developers and administrators to manage the workflows (Admin, Cockpit).
Being very lightweight and maintaining ease of use in its modeler while also providing business users with the ability to change and customize elements of the process application easily, among other things, makes Camunda both developer- and business-friendly. In the rest of the text, I will cover some of the use cases for Camunda as a lightweight workflow engine and also the problems we in Vicert encountered while using it and how we found solutions for them.
Camunda Cockpit application gives us the ability to manage workflow instances over multiple workflow definition versions
Camunda use cases/problems
Generated task view pages:
We already know that workflow engines can be used for business process automation. We also hear that Camunda is developer-friendly. Yet it is not always obvious in which form that friendliness is manifested. Let’s assume we have a few different workflows. Those workflows contain several user tasks. All those tasks require different user input in order to be completed. Finally, we do not want to expose Camunda’s Tasklist application to end-users. We want to give them the ability to work on those tasks using UI of our ReactJS-based web application.
It would be hard to maintain up-to-date task forms for all task components in the environment in which every day we may have to add or update the existing workflow. Luckily, Camunda can help us with this. Camunda can generate task forms for any task, and give us the usable HTML snippet. Of course, we can (and we will) use it in conjunction with our ReactJS components to create truly dynamic and intuitive web pages. Front End engineers at Vicert are real pros at this.
Camunda Tasklist application shows auto-generated form view for the selected task – we will use auto-generated forms to generate our customized web pages
Definition version migration
The dynamic nature of workflows can introduce another similar class of problems to our system. Let’s again picture an ecosystem with multiple different workflows. And in that system, we could have a claim handling workflow. That workflow is not necessarily unchangeable. It might very likely go through several iterations over the course of many months. We will easily end up in a situation where a workflow has several different versions and every version having several workflow instances in progress. We would, of course, update our frontend application to support the new workflow version, but how would that reflect on the user with tasks in old versions of workflow?
New UI would confuse them or even worse – cause them to inadvertently make mistakes in their workflows! We already saw a way to mitigate a similar problem. We would use Camunda’s auto-generated forms to decouple the writing of frontend code from workflow definition models. However, there may be some cases where we want to enforce moving to the newer versions of workflow even when the work is already in progress.
Once again, Camunda magic is there to aid us. Introduction process instance migration (PIMig) and process instance modification (PIMod) functionalities. In general, PIMig will move process instance from old to new version of the diagram by creating a mapping between tasks from different versions and applying it to relevant instances. This might be enough in some cases, but when it isn’t we can use PIMod which can cover the situation when an exact mapping cannot be created or additional processing needs to be done (this could be perhaps updated of some variables).
We use the microservice architecture. One of the services acts as a generic proxy to Camunda where communication between proxy and Camunda is done via REST. Other services can use the workflows by calling this proxy service and accessing workflow data they need. Having multiple services in this ecosystem increases the need for defining security policies in a more organized manner.
One way could be having the proxy service talk to Camunda as Camunda admin user, closing proxy service interface for public and open it only internally for other services. Those services would then implement facades that would wrap proxy service interface with security rules known to those services so the question of what services access which workflows and how they do it would be resolved in those services. An alternative way would be enabling Camunda security which supports concepts like tenancy, organization and group membership among others and opening proxy service interface to public and thus making the workflow engine the place where authorization is resolved.
Oh, the good old problem of synchronization of distributed transactions… It is not uncommon in a microservice architecture. One way of solving it is by adopting the Saga pattern. The basic idea is splitting the overall transaction into multiple steps, of which every step can be completed in atomic transactions and the overall consistency is maintained by the Saga. Saga has to make sure that either all steps in a transaction are completed successfully or the system is left in a consistent state. If one of the intermediate steps fails, retrying and/or compensation actions will be performed.
But, what all this have to do with Camunda? It turns out that the Saga can be implemented using Camunda. Camunda can easily be used for modeling distributed transactions. We would define a transaction as a sequence of tasks that represent transaction steps, and for each task, we would define a compensation activity. If any task fails, the compensation activities corresponding to the already completed tasks in the workflow will be executed in the reverse order which will return the system to the consistent state. We just need to make sure that that all our microservices implement idempotency and offer compensation activities.
Saga pattern implementation example – workflow shows the process of ordering an item from an online store, if problems were encountered during delivery, goods will be returned and the customer will be refunded.
What happens when a client asks for an easy way to generate a report about when the task A, which can be executed several times inside the loop of subprocess B in the workflow instance C in workflow D, has been completed for the 27th time under the condition that it did not change the value of variable E because they have to internally flag the business activity to which C is related as suspicious if that happened F days after the workflow instance has been created? We tell them, “Sure!”. We can be confident because Camunda gives us this for free. With Camunda we can find information about all completed workflow instances and tasks in the Camunda history. The great thing is that it can be customized – we can provide our own backend for storing history records since the Camunda separates runtime and history services internally.
Camunda is a great tool for automating business workflows. It can model and execute workflows that are constantly changing and evolving while at the same time providing the tools to easily track those changes and adapt to them, ranging from auto-generated forms and version migration functionalities up to good logging and auditing utilities. This can particularly become ehandy in healthcare having in mind the necessity for controlled security while being flexible to constant compliance changes that affect the workflow.
Additionally, implementing Sagas using Camunda is also a possibility. While we had many hurdles during the adoption of the Camunda engine, it turned out multiple times that we don’t have to look too far from Camunda documentation to find the answers, whether the problem was migrating workflow version or finding those magical 27th task completion dates.