1. What is Kafka?
Apache Kafka is developed in Scala and started out at LinkedIn as a way to connect different internal systems. At the time, LinkedIn was moving to a more distributed architecture and needed to reimagine capabilities like data integration and realtime stream processing, breaking away from previously monolithic approaches to these problems. Kafka is well adopted today within the Apache Software Foundation ecosystem of products and is particularly useful in event-driven architecture.
2. Structure
- Topics and partitions
- Message offset
- Consumer groups
3. Kafka transactions
Exactly-once Support in Apache Kafka
4. Python 与 Kafka
- producer
from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers='localhost:1234')
producer.send('foobar', b'some_message_bytes')
- consumer
from kafka import KafkaConsumer
consumer = KafkaConsumer('my_favorite_topic')
for msg in consumer:
print (msg)
5. 与 RabbitMQ 的区别
Understanding When to use RabbitMQ or Apache Kafka
Message-oriented Middleware for Scalable Data Analytics Architectures
Kafka vs RabbitMQ
