Data pipeline on cloud
Articles Case Studies Cloud Cloud Computing Confluent Kafka Data Engineering Data Pipeline Data Pipelines Event Streaming Kafka Real time data streaming Real Time Streaming

CASE STUDY: Drug discovery with new data pipelines based on Confluent Cloud


Recursion is a biotechnology company founded in 2013, headquartered in Salt Lake City, Utah. It accelerates drug discovery by combining experimental biology, artificial intelligence, automation, and real-time event streaming. It has built a system that processes over three petabytes of biological image data generated on Recursion’s robotic platform. The company

How To Build a Scalable Big Data Pipeline
Data Engineering Data Pipeline ETL Pipeline Real time data Tutorials Videos

VIDEO: How to design an efficient Big Data Pipeline. ETL vs Real-time pipelines


A data pipeline architecture is complex. With the continuous advancement, it becomes even more challenging to comprehend the structure and be decisive about the tools and technologies for the data pipelines. However, it is easy to understand it when you connect it with the day-to-day scenario. For example, a data

How To Build a Scalable Big Data Pipeline
Articles Data Engineering Data Pipeline Data Pipelines

How To Build a Scalable Big Data Pipeline


When you deploy machine learning, big-data analytics, and data science in real-time, you need to remember that model training and analytics tuning occupies only a portion of the work. Around 50% of the effort is dependent upon grooming the data for Machine Learning and Analytics. The rest of the effort

Data Pipeline definition
Articles Data Pipeline Data Pipelines ETL Pipeline Real time data

What is a Data Pipeline? Definition & Examples


Have you ever star-gazed? Let’s imagine that you are counting the number of stars in the sky. Would you be able to count all the stars? You can categorize them for sure. That’s exactly how abundant data is nowadays. When you allow data flow from one location to the next,

Real time data streaming and Analytics
Articles Case Studies Data Pipeline Data Pipelines Real time data Real time data streaming Real Time Streaming

CASE STUDY: Real time analytics & Data management at Charter


Charter Communications, Inc. is a leading broadband connectivity company and cable operator. Through their brand, Spectrum, they offer a full range of state-of-the-art residential and business services including Spectrum Internet®, TV, Mobile, and Voice in 41 states for more than 30 million customers. As customers always require better reliability, competitive

Real time Data Streaming
Apache Kafka Apache Nifi Articles Data Ingestion Tools Data Pipeline Data Pipelines Kafka Architecture Kafka as a Service Kafka Use Cases

Data Ingestion Pipelines & use cases


What is a Data pipeline? A data pipeline is a system where data is transferred in chunks in a serial and systematic manner (Messages, records) between systems. These flows are well defined, audited and might contain sensitive information, which needs to be secured.  These pipelines can be application queues, transfers

Real time Data Streaming
Articles Data Engineering Data Pipeline Data Pipelines Kafka Streams Real Time Streaming

What Is Streaming Data? Guide To Real-time Data And Stream Processing


Data is indeed the new oil, and real-time data processing & real time stream processing is the one that unlocks its potential to drive businesses in this technologically advanced era. Now, businesses need high-speed data pipelines more than before, to streamline their processes and match the customer’s expectations. A report

Articles Data Engineering Data Pipeline Infographic Learning & Development

What are streaming data pipelines?


Articles Data Engineering Data Pipeline

Why Scala and Apache Spark are key skills for Big data engineering?


Apache Spark is one of the most popular frameworks for big data analysis. This framework is written in Scala, because it is functional language and very scalable. It also can be quite fast because it's statically typed, and it compiles in a known way to the JVM. Hence, most of

developers working on the computer
Apache Kafka Articles Data Engineering Data Pipeline Data Pipelines Microservice Architecture Microservices

Implementing MongoDB to Elastic Search 7.X Data Pipeline


In this article, we will see how to implement a data pipeline from an application to Mongo DB database and from there into an Elastic Search keeping the same document ID using Kafka Connect in a Microservice Architecture. In recent days and years, all the microservices architectures are asynchronous in