Several years in production with AWS MSK

After deploying AWS MSK to production Dec 2021, has it been worth it?

Deploying and supporting AWS MSK (Managed Streaming for Kafka) for a year in production can be a challenging but rewarding experience. AWS MSK is a fully managed service that makes it easy to build and run applications that use Apache Kafka to process streaming data. In this blog post, we will discuss some of the key considerations when deploying and supporting AWS MSK for one year in production.

Deployment:

Deploying AWS MSK requires careful planning and coordination. Here are some key considerations when deploying AWS MSK:

1. Choose the right instance type: AWS MSK offers different instance types, each with different capabilities and costs. You need to choose the right instance type based on your workload's requirements and budget.
2. Choose the right number of brokers: The number of brokers you need depends on the size of your cluster and the number of partitions you need to support. You need to choose the right number of brokers to ensure optimal performance and availability.
3. Configure security: AWS MSK integrates with AWS security services like IAM, AWS KMS, and VPC to ensure that your data is secure. You need to configure security settings based on your organization's security policies and compliance requirements.

Support:

Supporting AWS MSK requires ongoing monitoring and maintenance to ensure optimal performance and availability. Here are some key considerations when supporting AWS MSK:

1. Monitor performance: You need to monitor your cluster's performance regularly to detect and troubleshoot any performance issues. AWS MSK provides metrics and logs that you can use to monitor your cluster's performance.
2. Apply updates: AWS MSK releases updates regularly to improve performance, fix bugs, and address security vulnerabilities. You need to apply updates to your cluster regularly to ensure that you are running the latest version of AWS MSK.
3. Manage data retention: AWS MSK provides different data retention options that you can use to manage your data. You need to choose the right data retention option based on your organization's data retention policies and compliance requirements.

Cost

In total we're paying around £1000 per month for 3 MSK instances for development, staging and production. Each cluster has 3 brokers, so there's room for optimisation. On production we rearly exceed 5% CPU so finding more cost effective testing environments will be on the cards soon

Schemas

We decided to try the Kafka Schema Registry project since our system was already using AJV which meant generating schemas from our Typescript code was easy. The main reason for using Kafka Schema Registry was to avoid poisoned messages in the event log and to encourage engineers to mindfully adjust schemas by enforcing compatibility modes.

Conclusion:

Deploying and supporting AWS MSK for one year in production requires careful planning, coordination, and ongoing maintenance. By following the key considerations discussed above, you can ensure that your AWS MSK cluster is deployed and supported properly, ensuring optimal performance, availability, and security. With AWS MSK, you can process streaming data at scale with ease, enabling your organization to derive valuable insights from your data.