By Tim Smith — 06 Jun 2023

Kafka JSON Schema Registry Compatibility Modes

Apache Kafka is a popular distributed streaming platform that has gained immense popularity in recent years due to its scalability, reliability, and ability to handle large amounts of data. One of the key components of Kafka is the Schema Registry, which is used to manage schemas for data stored in Kafka. The Schema Registry ensures that all data written to Kafka conforms to a specified schema, which makes it easy to maintain consistency across different applications.

When it comes to managing schemas in the Schema Registry, there are four different compatibility modes: backward compatibility, forward compatibility, full compatibility, and no compatibility. Each of these modes is designed to address different use cases and provide different levels of schema compatibility between different versions of the same schema. In this blog post, we will explore each of these compatibility modes in more detail to highlight some of the things that may catch you out.

Backward compatibility is the default mode for the Schema Registry. It ensures that any changes made to a schema are backward compatible with existing data stored in Kafka. In other words, when you make changes to a schema, the new schema must be able to read data that was written using the old schema. This mode is useful when you want to make non-breaking changes to a schema, such as adding new fields or changing the default value of a field.

describe('BackwardTransitive Compatible Schemas Evolution', () => {
  it('will ALLOW new optional fields in CLOSED schema mode', () => ... )
  it('will DENY new optional fields in OPEN schema mode', () => ... )
  it('will DENY new required fields in OPEN schema mode', () => ... )
  it('will DENY new required fields in CLOSED schema mode', () => ... ) 
  it('will DENY deletion of required field in OPEN schema mode', () => ... ) 
  it('will ALLOW deletion of required field in CLOSED schema mode', () => ... ) 
  it('will ALLOW deletion of optional field in OPEN schema mode', () => ... ) 
  it('will DENY deletion of optional field in CLOSED schema mode', () => ... ) 
  it('will DENY deletion of optional field in CLOSED schema mode', () => ... ) 
  it('will ALLOW adding value to ENUM field in CLOSED schema mode', () => ... ) 
  it('will DENY removing value from ENUM field in OPEN schema mode', () => ... ) 
  it('will DENY removing value from ENUM field in CLOSED schema mode', () => ... )
})

Forward compatibility ensures that any changes made to a schema are forward compatible with future versions of the schema. This means that data written using the new schema must be readable by applications using the old schema. This mode is useful when you want to make changes to a schema that may break existing applications that are using the old schema.

describe('ForwardTransitive Compatible Schemas Evolution', () => {
  it('will ALLOW new optional fields in OPEN schema mode', () => ... )
  it('will DENY new optional fields in CLOSED schema mode', () => ... )
  it('will ALLOW new required fields in OPEN schema mode', () => ... )
  it('will DENY new required fields in CLOSED schema mode', () => ... )
  it('will DENY deletion of required field in OPEN schema mode', () => ... )
  it('will DENY deletion of required field in CLOSED schema mode', () => ... )
  it('will DENY deletion of optional field in OPEN schema mode', () => ... )
  it('will ALLOW deletion of optional field in CLOSED schema mode', () => ... )
  it('will DENY adding value to ENUM field in OPEN schema mode', () => ... )
  it('will DENY adding value to ENUM field in CLOSED schema mode', () => ... )
  it('will ALLOW removing value from ENUM field in OPEN schema mode', () => ... )
  it('will ALLOW removing value from ENUM field in CLOSED schema mode', () => ... )
})

Full compatibility ensures that any changes made to a schema are both backward and forward compatible. This means that the new schema can read data written using the old schema, and data written using the new schema can be read by applications using the old schema. This mode is useful when you want to make significant changes to a schema that may break existing applications, but still want to ensure that data is readable across different versions of the schema.

No compatibility provides the freedom to make all changes to the schema which might include modifying the field type from a String to a Number.

In conclusion, the compatibility modes provided by the Kafka Schema Registry make it easy to manage schemas and ensure schema compatibility across different versions of an application. Backward compatibility is the default mode, and it is useful when you want to make non-breaking changes to a schema. Forward compatibility is useful when you want to make changes to a schema that may break existing applications, and full compatibility is useful when you want to make significant changes to a schema while still ensuring that data is readable across different versions of the schema.

Subscribe to Bear Bytes