Solution skill
Using Kafka as a (CQRS) Eventstore. Good idea?
Complete solution
Guide Using Kafka as a (CQRS) Eventstore. Good idea?: For example if the retention is set for two days, then for the two days after a message is published it is available for consumption, after which it will be discarded to free up space. Use this when you hit exactly this in Using Kafka. Not for different errors or different tools.
TL;DR: For example if the retention is set for two days, then for the two days after a message is published it is available for consumption, after which it will be discarded to free up space. Kafka is meant to be a messaging system which has many similarities to an event store however to quote their intro: The Kafka cluster retains all published messages-whether or not they have been consumed- for a configurable period of time .
Fix
For example if the retention is set for two days, then for the two days after a message is published it is available for consumption, after which it will be discarded to free up space. Expected: this specific failure stops.
This doesn't mean you can't use this as an event store, but it may be better to use something else. Expected: this specific failure stops.
UPDATE Kafka documentation : Event sourcing is a style of application design where state changes are logged as a time-ordered sequence of records. Expected: this specific failure stops.
UPDATE 2 One concern with using Kafka for event sourcing is the number of required topics. Expected: this specific failure stops.
Details
Kafka is meant to be a messaging system which has many similarities to an event store however to quote their intro: The Kafka cluster retains all published messages-whether or not they have been consumed- for a configurable period of time . Kafka's performance is effectively constant with respect to data size so retaining lots of data is not a problem. So while messages can potentially be retained indefinitely, the expectation is that they will be deleted. Take a look at EventStoreDB for an alternative. Kafka's support for very large stored log data makes it an excellent backend for an application built in this style. Typically in event sourcing,
When to use
You hit exactly this: Using Kafka as a (CQRS) Eventstore. Good idea? in Using Kafka.
When not to use
A different error, or the same symptom in a different tool. This page only covers the failure above.
Compatibility
Using Kafka.
Evidence
Asked on Stack Overflow so:17708489: Stack Overflow #17708489 (score 327, accepted answer, 163 votes): Although I've come across Kafka before, I just recently realized Kafka may perhaps be used as (the basis of) a CQRS , eventstore . One of the main points that Kafka supports:
Event capturing/storing, all HA of course. Pub/sub architecture Ability to replay the event log which allows the ability for new subscribers to register with the system after the fact.
Admittedly I'm not 100% versed in CQRS / Event sourcing but this seems pretty close to what an events tore should be. The funny thing is: I really can't find th The verified fix/answer: Kafka is meant to be a messaging system which has many similarities to an event store however to quote their intro:
The Kafka cluster retains all published messages-whether or not they have been consumed- for a configurable period of time . For example if the retention is set for two days, then for the two days after a message is published it is available for consumption, after which it will be discarded to free up space. Kafka's performance is effectively constant with respect to data size so retaining lots of data is not a problem.
So while messages can potentially be retained indefinitely, the expectation is that they will be deleted. This doesn't mean you can't use this as an event store, but it may be better to use something else. Take a look at EventStoreDB for an alternative. UPDATE Kafka documentation :
Event sourcing is a style of application design where state changes are logged as a time-ordered sequence of records. Kafka's support for very large stored log data makes it an excellent backend for an application built in this style.
UPDATE 2 One concern with using Kafka for event sourcing is the number of required topics. Typically in event sourcing, Source: https://stackoverflow.com/questions/17708489/using-kafka-as-a-cqrs-eventstore-good-idea
By @issueatlas · Version 2 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_ttMV2Wy9JnW1emYRxusdfg" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Using Kafka as a (CQRS) Eventstore. Good idea? <span>— Vectle</span></a>