Components
Components are the building blocks of flows. Each flow uses one input, zero or more processors, and one output.
Inputs
Inputs define where data enters the pipeline.
| Component | Description |
|---|---|
| Generate | Generates synthetic messages on a schedule — for testing or recurring job triggers |
| HTTP Client | Pulls data via HTTP requests |
| HTTP Server | Accepts incoming HTTP requests (webhooks) |
| Kafka | Consumes messages from Kafka topics |
| Broker | Combines multiple inputs into one flow |
| CDC MySQL | CDC from MySQL/MariaDB binlog |
| Shopify | Fetches data from Shopify stores |
Processors
Processors transform, validate, or route messages within the pipeline.
| Component | Description |
|---|---|
| Mapping | Bloblang transformations |
| JSON Schema | Validates messages against a JSON schema |
| Catch | Error handling — runs processors on failure |
| Switch | Conditional processing based on message content |
| Schema Registry Decode | Decodes Avro messages via Schema Registry |
Outputs
Outputs define where data is delivered.
| Component | Description |
|---|---|
| HTTP Client | Sends data via HTTP requests |
| Kafka | Produces messages to Kafka topics |
| SQL Insert | Inserts rows into SQL databases |
| Sync Response | Returns response to HTTP Server input |
| Switch | Conditional routing to different outputs |
| Broker | Routes messages to multiple outputs |
Other Components
Caches
Caches provide key-value state storage for components like CDC MySQL (binlog position) and Shopify (pagination cursor).
| Cache | Description |
|---|---|
| Memory | In-memory with TTL |
| Redis | Distributed caching |
| Memcached | Distributed memory caching |
| File | File-based persistence |
| LRU | Least Recently Used with capacity limits |
| TTLRU | TTL-aware LRU |
| Ristretto | High-performance cache |
| Noop | No-op cache for testing |
See the Caches page for detailed configuration of each type.
Files
The File Manager lets you upload files that components can reference at runtime — for example, JSON Schema files for validation. Files are stored centrally and distributed to workers automatically.
Rate Limits
| Rate Limit | Description |
|---|---|
| Coordinator | Distributed rate limiting across workers |
Buffers
| Buffer | Description |
|---|---|
| Memory | In-memory buffering |
| SQLite | Persistent at-least-once delivery |
| System Window | Time-based windowing |