In a complex environment like the one we have been describing so far in this series, one of the key factors for success is automation and auditability. This becomes even more evident when it comes to managing Apache Kafka. Kafka itself is complex, with numerous configurable options, and the thought of manual management should make you break out in a cold sweat. To address this, we chose to use a tool called JulieOPS. Disclaimer: We are aware that the project is currently dormant, but since the client has a team of skilled Java developers, we took a calculated risk knowing that if we needed to adapt the source code, we could rely on their expertise.
JulieOPS enables us to automate the creation, deletion, and updates of topics, schemas, and RBAC (Role-Based Access Control) in Kafka. Since multiple organizational units will be using Kafka either to produce or consume data, manual management would be impractical. By integrating JulieOPS into our pipelines, we can automate these tasks.
We established a central repository accessible only to Kafka administrators. There are a few reasons for this. Firstly, in order to have administrative rights over the entire environment, you need to provide your MDS (Metadata Server) user credentials to JulieOPS. Without these credentials, you won’t be able to execute all the necessary commands. Secondly, there are two main projects at work: JulieOPS and Topics. The Topics project contains the descriptor files that define the Kafka cluster(s). This project only has access to JulieOPS via pipeline tokens. Any individuals needing to interact with Kafka will make the necessary changes in the Topics project.
JulieOPS can be seen as a Java application that checks the descriptor files in predefined directories to determine the desired state of your Kafka cluster. It then compares this with the current state of Kafka and applies the necessary changes. In simpler terms, it automates the management process. The JulieOPS project includes descriptor files for topics, schemas, and plans, and it infers some RBAC rules when executing these. For example, if you create a topic and define testuser as a Producer, then testuser will be assigned the role of DeveloperWrite for that topic, but won’t have read access. These inferred rules are quite helpful. JulieOPS also has a dryRun capability, which shows you the changes that will be applied to the cluster. At the time of writing, it lacked schema validation in dryRun mode, but we consulted one of the developers to explore the possibility of incorporating it. The dryRun mode provides a complete overview of the changes, such as which users will be granted access to which topics, their roles, and the schemas attached to each topic. It’s an immensely helpful software tool, so instead of reinventing the wheel, we decided to enhance it further.
A developer makes changes to a YAML file (descriptor) within the Topics project in GitLab. This change is made in a branch they created. Once the code is committed to their branch in GitLab, a pipeline is triggered. The first step of the pipeline is to perform a dryRun of JulieOPS. If JulieOPS exits with a zero exit code, the pipeline passes. Any non-zero exit code automatically fails the pipeline. Only if this stage of the pipeline passes, the developer can open a merge request. Once the merge request is opened and assigned to a reviewer (a Kafka administrator), and the reviewer merges the code into the main branch, the next job is initiated. This job is again a dryRun to ensure that we are satisfied with the topology changes. At this point, the pipeline is paused, and the reviewer needs to manually allow the pipeline to continue, considering the potential impact on the organization if something goes wrong. Utmost care must be taken at this stage. If the pipeline is continued, it proceeds with applying the changes, followed by building an audit log.
The audit log is custom code developed by HBPS specifically for this client. It is a small Python script that examines the topology definition in the Topics project, identifies all topics, and retrieves the corresponding producers and consumers. It then writes a summary of the topic name, its consumers and its producers to a file, which is attached as an artifact to the executed pipeline. This promotes transparency and ensures the entire process is auditable, making compliance much easier.
Although JulieOPS is currently a dormant project, it remains the most powerful tool for automating the management of Apache Kafka. It is well-suited for incorporation into CI/CD pipelines, requires minimal adaptation to be functional, and offers the added advantage of being customizable due to its availability on GitHub. We believe that any organization using Apache Kafka in their environment would benefit from using JulieOPS to automate its management.