Immutable solution version
Using Kafka as a (CQRS) Eventstore. Good idea?
Complete solution
What a coding agent needs to know about Apache Kafka here: Using Kafka as a (CQRS) Eventstore. Good idea?.
Kafka is meant to be a messaging system which has many similarities to an event store however to quote their intro:
The Kafka cluster retains all published messages-whether or not they have been consumed- for a configurable period of time . For example if the retention is set for two days, then for the two days after a message is published it is available for consumption, after which it will be discarded to free up space. Kafka's performance is effectively constant with respect to data size so retaining lots of data is not a problem.
So while messages can potentially be retained indefinitely, the expectation is that they will be deleted. This doesn't mean you can't use this as an event store, but it may be better to use something else. Take a look at EventStoreDB for an alternative. UPDATE Kafka documentation :
Event sourcing is a style of application design where state changes are logged as a time-ordered sequence of records. Kafka's support for very large stored log data makes it an excellent backend for an application built in this style.
UPDATE 2 One concern with using Kafka for event sourcing is the number of required topics. Typically in event sourcing,
Evidence
Asked on Stack Overflow so:17708489: Stack Overflow #17708489 (score 327, accepted answer, 163 votes): Although I've come across Kafka before, I just recently realized Kafka may perhaps be used as (the basis of) a CQRS , eventstore . One of the main points that Kafka supports:
Event capturing/storing, all HA of course. Pub/sub architecture Ability to replay the event log which allows the ability for new subscribers to register with the system after the fact.
Admittedly I'm not 100% versed in CQRS / Event sourcing but this seems pretty close to what an events tore should be. The funny thing is: I really can't find th The verified fix/answer: Kafka is meant to be a messaging system which has many similarities to an event store however to quote their intro:
The Kafka cluster retains all published messages-whether or not they have been consumed- for a configurable period of time . For example if the retention is set for two days, then for the two days after a message is published it is available for consumption, after which it will be discarded to free up space. Kafka's performance is effectively constant with respect to data size so retaining lots of data is not a problem.
So while messages can potentially be retained indefinitely, the expectation is that they will be deleted. This doesn't mean you can't use this as an event store, but it may be better to use something else. Take a look at EventStoreDB for an alternative. UPDATE Kafka documentation :
Event sourcing is a style of application design where state changes are logged as a time-ordered sequence of records. Kafka's support for very large stored log data makes it an excellent backend for an application built in this style.
UPDATE 2 One concern with using Kafka for event sourcing is the number of required topics. Typically in event sourcing, Source: https://stackoverflow.com/questions/17708489/using-kafka-as-a-cqrs-eventstore-good-idea
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_ttMV2Wy9JnW1emYRxusdfg" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Using Kafka as a (CQRS) Eventstore. Good idea? <span>— Vectle</span></a>