Central is a Streams-as-a-Service platform that serves as the primary source for real-time data generated by products and services. It functions as a centralized hub for managing and distributing real-time data streams, enabling efficient data sharing and processing across various applications and teams.

The key features of Central include:

• Data ingestion: Products and services can publish message payloads to Central's Collector API using a defined schema.
• Topic routing: The Central Collector determines the appropriate Kafka topic to store the data based on the type of payload.
• Data consumption: Consumers can identify relevant topics based on the payload type and consume messages from those topics. This streamlined process allows for seamless data flow between producers and consumers, facilitating real-time data analysis, event-driven architectures, and various other applications.

Central provides two primary integration methods for platform services and product teams within a region:

• Method 1: Using the Collector Endpoint

  This method involves leveraging Central's Collector API to send data to Kafka.

• Method 2: Direct Publishing to Kafka

  This method enables producers to publish data directly to Kafka topics, bypassing the Collector API.

To learn the details, see Integrate with Central.

To consume data, teams must:

1. Perform VPC peering with Central's Kafka VPC.
2. Associate their VPC with Central's Route53 hosted zone.
3. Generate SSL credentials and have their consumer group whitelisted to access the data streams.

Central has a dedicated team responsible for monitoring and maintaining the platform. Their responsibilities include:

• Monitoring Slack channel: Addressing issues and responding to help requests in designated Slack channels.
• Managing support portal: Handling customer-reported tickets on the Central Support Portal based on priority and service-level agreement (SLA).

The Central team is exploring ways to implement a fair usage policy to prevent the "noisy neighbor" phenomenon and ensure equitable resource allocation. Proposed solutions include:

• Rate limiting: Utilizing services like Fluffy to control the rate at which tenants send data through the Collector API.
• Kafka quotas: Leveraging Kafka's built-in quota mechanisms to manage client usage, though these mechanisms may lack granularity.
• Kafka proxy: Introducing a proxy between tenants and Kafka brokers to enable more granular monitoring and rate limiting.

Central is exploring strategies to cater to different user tiers (Free, Trial, Active, Premium) to ensure optimal performance and efficient resource allocation:

• Dedicated topics: Creating separate Kafka topics for each tier to enable customized resource allocation and scaling.
• Tier information in headers: Using a single Kafka topic for all users while embedding tier information in headers to support dynamic scaling based on tier.
• Dedicated clusters: Deploying separate Kafka clusters for each tier to provide maximum isolation and performance optimization.