Day 7: System Design(Message Queues and Event Driven)

Modern applications need to process large volumes of requests asynchronously to improve scalability, reliability, and performance. Message queues and event-driven architecture help in handling tasks efficiently in distributed systems.

What is a Message Queue?

A message queue is a system that stores and forwards messages between different services asynchronously.

How It Works:

1. Producers (senders) generate messages and push them to the queue.
2. The message queue temporarily stores the messages until they are processed.
3. Consumers (receivers) pick up messages from the queue and process them.

Example:

• E-commerce systems use message queues to handle order processing asynchronously. When a user places an order, the system adds a message to a queue, and another service processes it without making the user wait.

Why Use Message Queues?

1. Asynchronous Processing

• Users don’t have to wait for tasks to complete.
• Example: When a user uploads a video to YouTube, the system queues the video for processing while allowing the user to continue browsing.

2. Load Balancing & Scalability

• Spreads workloads across multiple consumers to prevent a single system from getting overloaded.
• Example: Netflix uses message queues to distribute streaming requests across different servers.

3. Fault Tolerance & Reliability

• If a consumer fails, messages remain in the queue until another service picks them up.
• Example: If a payment service crashes, pending payments remain in the queue until the system recovers.

4. Microservices Communication

• Decouples services, so they don’t need to wait for each other.
• Example: WhatsApp processes messages via queues, so even if the server is busy, messages get delivered once resources are available.

Types of Message Queues

1. Point-to-Point Queue (FIFO — First In, First Out)

• Messages are delivered to one consumer only, ensuring order is maintained.
• Example: A banking system ensures that transaction logs are processed sequentially to prevent inconsistencies.

2. Publish-Subscribe (Pub/Sub) Model

• Multiple consumers subscribe to the same message, and each gets a copy.
• Example: Facebook’s news feed pushes updates to millions of users using Pub/Sub.

3. Priority Queues

• Messages have different priority levels, so urgent tasks are processed first.
• Example: Ride-hailing apps (Uber, Lyft) prioritize emergency rides over regular bookings.

Popular Message Queue Technologies

1. Apache Kafka

• Best for high-throughput real-time data streaming.
• Used by LinkedIn, Uber, and Netflix for processing millions of events per second.

2. RabbitMQ

• Lightweight and flexible, supports Pub/Sub and direct messaging.
• Used in e-commerce platforms to handle order placements and notifications.

3. Amazon SQS (Simple Queue Service)

• Fully managed queue service that scales automatically.
• Used by AWS Lambda and serverless applications.

4. Google Cloud Pub/Sub

• Ideal for event-driven applications.
• Used in Google Ads and analytics systems.

Event-Driven Architecture

What is an Event-Driven System?

An event-driven system reacts to events in real time, rather than following a fixed process.

How It Works:

1. An event occurs (e.g., a user makes a purchase).
2. The system captures the event and sends it to the relevant services.
3. Different services process the event independently.

Example:

• E-commerce checkout: When a user buys a product, different services handle payment, inventory updates, and shipping asynchronously without blocking each other.

Use Cases of Event-Driven Architecture

1. Real-Time Analytics (Kafka, Spark Streaming)

• Stock market trading systems process millions of price changes per second.

2. Fraud Detection in Banking

• Banks use event-based AI models to detect fraudulent transactions in real-time.

3. Social Media Notifications

• Instagram uses event-driven architecture to send push notifications when someone likes your post.

4. IoT (Internet of Things)

• Smart homes (Google Nest, Alexa) trigger automation events when sensors detect motion.

Challenges in Message Queues & Event-Driven Systems

1. Message Ordering Issues

• If multiple consumers pick up messages out of order, data consistency can break.
• Solution: Use Kafka’s ordering guarantees or distributed locks.

2. Duplicate Message Delivery

• Some queues may send the same message multiple times.
• Solution: Implement idempotent processing to prevent duplicate actions.

3. Eventual Consistency

• Since events are processed asynchronously, data may take time to update across services.
• Solution: Use distributed transactions (Sagas, Two-Phase Commit).

4. Debugging & Monitoring Complexity

• Tracing events across multiple microservices is challenging.
• Solution: Use distributed tracing tools like OpenTelemetry.

Message queues and event-driven architectures enable scalable, fault-tolerant, and real-time applications. Advanced topics like event sourcing, distributed transactions, and event replay are essential for designing large-scale, mission-critical systems used by Netflix, Uber, Twitter, and banking institutions.

I’ll be posting daily to stay consistent in both my learning followed by daily pushups. Thank you!

Follow my journey:
Medium: https://ankittk.medium.com/
Instagram: https://www.instagram.com/ankitengram/