Apache Kafka 2.7.0 has arrived! ✨

What is new explained by Bill Bejeck, author of📚Kafka Streams in Action📖 [II Ed.📗 now available and newly titled as 📚Event Streaming with Kafka Streams and ksqlDB📖 (MEAP)]:
https://blogs.apache.org/kafka/entry/what-s-new-in-apache4.

List of the KIPs:

Kafka broker, producer, and consumer
- KIP-654: Aborted transaction with non-flushed data should throw a non-fatal exception;
- KIP-651: Support PEM format for private keys and SSL certificates and private key;
- KIP-612: Ability to limit connection creation rate on broker;
- KIP-599: Throttle create topic, create partition, and delete topic operations;
- KIP-584: Versioning scheme for features;
- KIP-554: Add broker-side SCRAM configuration API;
- KIP-497: Add inter-broker API to alter ISR;
- KIP-431: Print additional fields from records with the ConsoleConsumer.

Kafka Connect
- KIP-632: Add DirectoryConfigProvider

Kafka Streams
- KIP-662: Throw exception when source topics of Kafka Streams application is deleted;
- KIP-648: Renaming getter method for interactive queries;
- KIP-617: Allow Kafka Streams state stores to be iterated backwards;
- KIP-616: Rename implicit SerDes instances in kafka-streams-scala;
- KIP-613: Add end-to-end latency metrics to Kafka Streams;
- KIP-607: Add metrics to Kafka Streams to report properties of RocksDB;
- KIP-450: Sliding window aggregations in the DSL.

For further details, read the Apache blog post (link above).

Data Streaming IT in Italian: @kafkastreamingit

#ApacheKafka #Kafka #KafkaStreams #streamprocessing #datastreamingit

Apache Kafka 2.8.0 ✨, Kafka Streams improvements:

- KIP 418 repleces 'branch()' with 'split()' ... ;
- KIPs 663 / 671 / 696 improve "fatal error" handling and recovery;
and much more!

See kafkastreams's Twitter 🧵: https://twitter.com/kafkastreams/status/1360653644568272897?s=12.

DataStreamingIT in Italian: @kafkastreamingit

#ApacheKafka #Kafka #KafkaStreams #KStreams #streamprocessing #datastreamingit

ksqlDB 0.15 ✨ is out!

Expanded support for message keys:
- JSON, AVRO, PROTOBUF, DELIMITED, KAFKA and JSON_SR;
- ARRAY, STRUCT and nested combinations.

Also PARTITION BY and GROUP BY clauses with multiple partitioning or grouping expressions.

And more,see: https://twitter.com/ksqldb/status/1362087629718650882?s=21.

#ApacheKafka #Kafka #ksqlDB #streamprocessing #datastreamingit

Apache Spark 3.1.1 ✨ is here!

The changes are mainly about:
- Core
- Spark SQL
- Structured Streaming
- PySpark
- MLlib
- SparkR
and more.

For further details: https://spark.apache.org/releases/spark-release-3-1-1.html.

DataStreamingIT in Italian: @kafkastreamingit

#ApacheSpark #SparkStreaming #PySpark #Python #streamprocessing #datastreamingit

21-23 SEP ApacheCon @ Home 2021. CfP are opened: https://blogs.apache.org/conferences/entry/call-for-presentations-for-apachecon.

#ApacheCon #ASF #ApacheKafka #ApacheFlink #ApacheBeam #ApacheSpark #datastreamingit

Litestream (Real-time S3 replication for SQLite) v0.3.3 ✨ is out!

Released: https://litestream.io/blog/litestream-v0.3.3-released.

Why I [Ben Johnson] Built Litestream: https://litestream.io/blog/why-i-built-litestream.

DataStreamingIT in Italian: @kafkastreamingit

#Litestream #databases #S3 #SQLite #datastreamingit

Apache Kafka 2.8 ✨ without ZooKeeper (KIP-500) first look:

- Quorum Controller
- KRaft aka Kafka RAFT Metadata mode,

explained by Ben Stopford, author of 📚 Designing Event-Driven Systems 📖: https://www.confluent.io/blog/kafka-without-zookeeper-a-sneak-peek.

DataStreamingIT in italiano: @kafkastreamingit

#ApacheKafka #Kafka #KRaft #streamprocessing #distributedsystems #datastreamingit

What is a Streaming Database? 🤔

Brief summary from VentureBeat: https://venturebeat.com/2021/04/04/what-is-a-streaming-database.

Data Streaming IT in Italian: @kafkastreamingit

#ApacheKafka #Kafka #ksqlDB #databases #streamprocessing #datastreamingit

21-23 SEP ApacheCon @ Home 2021.

Registration now opened: https://hopin.com/events/apachecon-2021-home.

Data Streaming IT in italiano: @kafkastreamingit

#ApacheCon #ASF #ApacheKafka #ApacheFlink #ApacheBeam #ApacheSpark #opensource #datastreamingit

Beam College 🎓

Apache Beam 🐝 free training organized by the Community: https://beamcollege.dev.

Data Streaming IT in italiano: @kafkastreamingit

#ApacheBeam #BeamCollege #Beam #streaminganalytics #streamprocessing #dataprocessing #datastreamingit

Apache Flink 🐿 Stateful Functions aka StateFun 3.0.0 ✨ is out!

New features:
- Unified Languages SDKs;
- Cross-Language Type System;
- Dynamic Registration of State and Functions
and more.

For further details: https://flink.apache.org/news/2021/04/15/release-statefun-3.0.0.html.

DataStreamingIT in italiano: @kafkastreamingit

#ApacheFlink #Flink #StatefulFunctions #StateFun #streamprocessing #datastreamingit

Apache Kafka 2.8.0 ✨ is finally out!

- KIP-500: Replace ZooKeeper with a self-managed quorum;
- KIP-700: Add Describe Cluster API;
- KIP-684: Support mutual TLS authentication on SASL_SSL listeners;
- KIP-676: Respect logging hierarchy;
- KIP-673: Emit JSONs with new auto-generated schema;
- KIP-612: Limit broker connection creation rate;
- KIP-516: Topic identifiers.

Kafka Connect
- KIP-661: Expose task configurations in Connect REST API.

Kafka Streams
- KIP-696: Update Streams FSM to clarify ERROR state meaning;
- KIP-689: Extend StreamJoined to allow more store configs;
- KIP-680: TopologyTestDriver should not require a properties argument;
- KIP-671: Introduce Kafka-Streams-specific uncaught exception handler;
- KIP-663: API to start and shut down Streams threads;
- KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size;
- KIP-572: Improve timeouts and retries in Kafka Streams.

For further details, read the Apache blog post: https://blogs.apache.org/kafka/entry/what-s-new-in-apache5.

Data Streaming IT in Italian: @kafkastreamingit

#ApacheKafka #Kafka #KafkaStreams #streamprocessing #datastreamingit

NATS 2.2 ⚡️ JetStream release is out!

JetStream features:
- At-least-once delivery and exactly once within a window;
- Store data and replay by time or sequence;
- Cleanse specific messages (GDPR);
- Horizontal scalability;
- Persist Streams and replay via Consumers
and more.

For further details: https://nats.io/blog/nats-whats-new-22.

DataStreamingIT in italiano: @kafkastreamingit

#NATS #JetStream #CNCF #golang #streamprocessing #datastreamingit

ksqlDB 0.17 🚀 is here!

CI/CD features, new TIMESTAMP data type and much more.

See: https://twitter.com/ksqldb/status/1386729481507770369?s=21.

DataStreamingIT in italiano: @kafkastreamingit

#ApacheKafka #Kafka #ksqlDB #databases #DevOps #streamprocessing #datastreamingit

Apache Druid 0.21.0 🧙‍♂️is here!

New features:
- Leader election based on Kubernetes (without ZooKeeper);
- New grouping aggregator function;
- Old Kafka version support
and much more.

For further details: https://github.com/apache/druid/releases/tag/druid-0.21.0.

DataStreamingIT in italiano: @kafkastreamingit

#ApacheDruid #Druid #realtimeanalytics #databases #datastreamingit

Apache Flink 1.13.0 🐿 is out!

New features:
- Reactive scaling;
- Flame Graphs;
- Machine Learning Library moved to flink-ml;
- Windows via Time-valued functions;
- PyFlink improvements
and much more.

For further details: https://flink.apache.org/news/2021/05/03/release-1.13.0.html.

Data Streaming IT in italiano: @kafkastreamingit

#ApacheFlink #Flink #streamprocessing #SQL #PyFlink #Python #datastreamingit

Kafka Summit Europe 2021 📅 11-12 MAY 📆 is happening now!

✨ kafka-summit.org

#ApacheKafka #Kafka #KafkaSummit #streamprocessing #datastreamingit

Apache Beam 🐝

Beam Summit 2021 📅 04-06 AUG 📆

Free registration and CfP: https://beam-summit-2021-digital.web.app.

Data Streaming IT in italiano: @kafkastreamingit

#ApacheBeam #Beam #BeamSummit #streaminganalytics #dataprocessing #datastreamingit

Just won 📚Mastering Kafka Streams and ksqlDB: Building Real-Time Data Systems by Example📖 @ 📅 Kafka Summit Europe 2021 📆 Raffle: https://twitter.com/bsak/status/1395095120450568195?s=21!

Looking forward to reading this highly anticipated book!

Data Streaming IT in italiano: @kafkastreamingit

#ApacheKafka #Kafka #KafkaSummit #ksqlDB #KStreams #streamprocessing #datastreamingit

Apache Spark ⭐️

DATA + AI Summit 2021 📅 24-28 MAY 📆

Free General Admission: https://databricks.com/dataaisummit/north-america-2021.

Data Streaming IT in italiano: @kafkastreamingit

#ApacheSpark #Spark #databricks #MLFlow #AI #Python #SQL #datastreamingit

ksqlDB 0.18.0 🚀 is out!

New features:
- Materialize Table-Table join results;
- support DEFINE and UNDEFINE statements in migrations tool
and much more.

See: https://twitter.com/ksqldb/status/1397600801581789185?s=21.

Data StreamingI T in italiano: @kafkastreamingit

#ApacheKafka #Kafka #ksqlDB #databases #streamprocessing #datastreamingit