Distributed Image Processor using Apache Kafka

📖 Project Overview

Welcome to the Distributed Image Processor. This project demonstrates a robust, horizontally scalable architecture for parallel image processing using Apache Kafka, Python, and Flask.

In modern distributed systems, handling large payloads synchronously is a bottleneck. This system solves that by employing a master-worker architecture where a heavy workload (processing a high-resolution image) is broken down into independent chunks, processed in parallel by distributed workers, and seamlessly reconstructed.

🎯 What Exactly Is It?

At its core, this is a distributed computing pipeline designed for image manipulation:

1. Master Node (Flask Web App & Kafka Producer/Consumer):

   • Accepts an uploaded image via a web interface.
   • Splits the large image into smaller, manageable chunks (e.g., 512x512 tiles).
   • Publishes these tiles as independent tasks to a Kafka tasks topic.
   • Listens on a results topic for processed tiles and reconstructs the final image in real-time.
   • Provides a live dashboard to monitor job progress and worker node health (heartbeats).

2. Worker Nodes (Kafka Consumers/Producers):

   • Run independently (on the same machine or completely different machines).
   • Consume raw image tiles from the tasks topic.
   • Apply a processing algorithm (e.g., Grayscale conversion via OpenCV).
   • Publish the processed tile back to the results topic.
   • Send regular heartbeats to the heartbeats topic to inform the master of their active status.

🌐 The Network Layer (Crucial Setup Step)

Important Note on Networking: For this distributed system to function, the Master and all Worker nodes must be able to communicate with the Kafka Broker as if they are on the same Local Area Network (LAN).

Since workers might be distributed across different physical networks or even different countries, we utilized a Virtual LAN tool to bridge the network gap. Based on the configuration (e.g., zt_ip and the 172.x.x.x subnet), ZeroTier was used to achieve this.

If you are setting this up across multiple machines, you must use a Virtual LAN network tool like ZeroTier, Tailscale, or LogMeIn Hamachi to ensure all nodes share the same virtual subnet and can route traffic to the Kafka broker seamlessly.

🚀 How to Use / Run This Project

Prerequisites

• Python 3.8+
• Apache Kafka & Zookeeper (running locally or remotely)
• A Virtual LAN tool (like ZeroTier) if running across multiple physical machines.

1. Install Dependencies

Run the following command on both the Master and Worker machines to install the required Python packages:

pip install -r requirement.txt

2. Configure the Network & Broker

1. Ensure your Kafka broker is running and accessible on your Virtual LAN (e.g., ZeroTier network).
2. Open master.py, Worker1/worker1.py, and Worker2/worker2.py.
3. Locate the BOOTSTRAP_SERVER / BROKER configuration variable and update it with the Virtual LAN IP address of your Kafka Broker (e.g., 172.22.161.228:9092).

3. Start the Master Node

On the main machine, start the Flask application:

python app.py

This will spin up the web server on http://0.0.0.0:5000 and start the heartbeat listener.

4. Start the Worker Nodes

On your worker machines (or in separate terminal windows on the same machine), start the worker scripts:

# Terminal 1
cd Worker1
python worker1.py

# Terminal 2
cd Worker2
python worker2.py

5. Process an Image

1. Open your web browser and navigate to http://<MASTER_IP>:5000.
2. Upload a large image using the web interface.
3. Watch the live dashboard as the Master splits the image, sends it to the Kafka broker, and the Workers process the tiles in parallel.
4. Once completed, the reconstructed processed image will be displayed on the screen!

---

Built with scalability, fault-tolerance, and distributed computing principles in mind.