About Apache Kafka & Apache Airflow Kafka is a popular messaging platform based on pub-sub mechanism. It is a highly available, fault tolerant & distributed system. Most organisations are using Kafka in different use-cases, but the best part of Kafka is it’s service oriented architecture, it makes it language agnostic,
Company: Alight Solutions is a leading technology-enabled health, wealth, and human capital management solutions company in the US. The company is providing its services in over 100 countries across five continents and serves over 4,300 clients. Alight has recently taken initiatives to align its internal organization with the next-generation digital
If dealing with data was not difficult, many of us would be out of the job, literally. But, on a serious note, even as real-time data analytics becomes one of the key drivers of revenues for many businesses, implementing a robust data IT architecture is quite challenging. To extract the
A data pipeline is defined commonly as a set of bulk actions through which we extract data from a number of sources. This standardized, robotic process takes place when the system takes columns from different databases. They mould them with different columns with API as the target source. The combination
What Is an ETL Pipeline? Extract, Transform and Load, significantly known as ETL pipeline is a set of processes where you extract data from a particular source, and then transform it before you load it into the destination. The source can derive itself from anywhere, be it business systems or
Kafka is a popular messaging platform based on pub-sub mechanism. It is a highly available, fault tolerant & distributed system. Most organisations are using Kafka in different use-cases, but the best part of Kafka is it’s service oriented architecture, it makes it language agnostic, and gives it wide usability. One
Recursion is a biotechnology company founded in 2013, headquartered in Salt Lake City, Utah. It accelerates drug discovery by combining experimental biology, artificial intelligence, automation, and real-time event streaming. It has built a system that processes over three petabytes of biological image data generated on Recursion’s robotic platform. The company
A data pipeline architecture is complex. With the continuous advancement, it becomes even more challenging to comprehend the structure and be decisive about the tools and technologies for the data pipelines. However, it is easy to understand it when you connect it with the day-to-day scenario. For example, a data
When you deploy machine learning, big-data analytics, and data science in real-time, you need to remember that model training and analytics tuning occupies only a portion of the work. Around 50% of the effort is dependent upon grooming the data for Machine Learning and Analytics. The rest of the effort
Have you ever star-gazed? Let’s imagine that you are counting the number of stars in the sky. Would you be able to count all the stars? You can categorize them for sure. That’s exactly how abundant data is nowadays. When you allow data flow from one location to the next,
Charter Communications, Inc. is a leading broadband connectivity company and cable operator. Through their brand, Spectrum, they offer a full range of state-of-the-art residential and business services including Spectrum Internet®, TV, Mobile, and Voice in 41 states for more than 30 million customers. As customers always require better reliability, competitive
What is a Data pipeline? A data pipeline is a system where data is transferred in chunks in a serial and systematic manner (Messages, records) between systems. These flows are well defined, audited and might contain sensitive information, which needs to be secured. These pipelines can be application queues, transfers
