developers working on the computer
Apache Kafka Articles Data Engineering Data Pipeline Data Pipelines Microservice Architecture Microservices

Implementing MongoDB to Elastic Search 7.X Data Pipeline


In this article, we will see how to implement a data pipeline from an application to Mongo DB database and from there into an Elastic Search keeping the same document ID using Kafka Connect in a Microservice Architecture. In recent days and years, all the microservices architectures are asynchronous in

computer and coding screen kafka logo
Articles Learning & Development Popular

Enriching a Kafka Connect to replace traditional batches and codes


Kafka Connect is a technology for linking Kafka with other services. One of the main challenges of an IT architecture is the batch process and a bunch of codes that transport or align data from one to another application. A small failure can cause ultimate data and business losses with

women holding a phone with technology graphics and kafka and flink logo
Articles Popular

Real-time processing and aggregation of Call Detail Records (CDRs) using Kafka and Flink


With the exponential growth in data, the biggest problem plaguing the telecom industry is how to process and aggregate huge volumes of call detail records (CDRs). Telcos are takingĀ  fresh approaches to get the real-time value of streaming data. They are shifting from heavy batch processing systems to faster, scalable

computer with charts and Kibana logo
Articles Popular

Kibana: Data visualization made easier


Kibana makes data analytics easy. It is one of the elites in providing visualization solutions for enterprises for data sets including runtime data, server logs, market statistics, geo-spatial data mappings, density graphs. It works with Elasticsearch and supports machine learning. The tool can unravel secrets such as trends/patterns from the

Real time data streaming and Analytics
Articles Popular

Failure detection and alert from real-time data flow using Kafka and KSQL


Apache Kafka ecosystem is relevant to detect anomalies (failure detection) in the huge volume of data sets. These data sets can come from various industries including IoT, banking or insurance. In this article, we present a use case on how to use the cloud-native ecosystem with Kafka, MQTT, Kafka Connect,

graphic computer and mobile screen with graphs and charts

Real-Time Data Streaming, Kafka and Analytics, Part 2: Going Beyond Pure Streaming


Data transaction streaming is managed through many platforms, with one of the most common being Apache Kafka. In our first article in this data streaming series, we delved into the definition of data transaction and streaming and why it is critical to manage information in real-time for the most accurate