A modern enterprise Data Streaming Architecture will have a lot of communications, these can include application queues, streaming data, control data, data transfers, for backup, syncing, updates, IOT sensor data, system snapshots and other never ending use cases. We would be discussing about some commonly used use-cases in most enterprises.
What is a Data pipeline? A data pipeline is a system where data is transferred in chunks in a serial and systematic manner (Messages, records) between systems. These flows are well defined, audited and might contain sensitive information, which needs to be secured. These pipelines can be application queues, transfers