Smart India Hackathon 2026 | Problem Statement 145
Aegis is a passive, AI/ML-powered network threat detection platform designed to observe network traffic without introducing an active control path into the monitored network. It combines Zeek network telemetry, feature engineering, Kafka streaming, machine learning, anomaly detection, SHAP explainability, FastAPI, Elasticsearch, and a real-time React dashboard into a single end-to-end detection pipeline.
The system is designed for controlled, isolated experimentation with realistic attack and benign traffic scenarios.
- Overview
- Why Aegis
- Core Capabilities
- Detection Scenarios
- System Architecture
- Data Flow
- Feature Engineering
- ML Detection
- Explainability
- Streaming Pipeline
- Alert Correlation and Kill Chains
- Dashboard
- Project Structure
- Team Responsibilities
- Getting Started
- Configuration
- Testing
- Evaluation
- Security and Isolation
- Development Workflow
- Project Status
- Future Extensions
Traditional network monitoring systems often generate large volumes of low-level telemetry that must be manually inspected or processed by separate security tools.
Aegis creates a unified pipeline:
┌─────────────────────┐
│ Network Traffic │
│ PCAP / Live Traffic │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Zeek │
│ Network Telemetry │
└──────────┬──────────┘
│
┌───────────────┼────────────────┐
▼ ▼ ▼
conn.log dns.log ssl.log
│
quic.log
└───────────────┬────────────────┘
▼
┌─────────────────────┐
│ Feature Engineering │
│ Pipeline │
└──────────┬──────────┘
│
FeatureRow
│
▼
┌─────────────────────┐
│ Staging Queue │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Kafka │
│ raw-features │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Kafka Consumer │
│ → FastAPI /infer │
└──────────┬──────────┘
│
┌──────────┴──────────┐
▼ ▼
Inference Result Pipeline Metrics
│ │
▼ ▼
Elasticsearch Kafka /metrics
│ │
└──────────┬──────────┘
▼
┌─────────────────────┐
│ React Dashboard │
│ Alerts / Stats / │
│ Metrics / Kill Chain│
└─────────────────────┘
The project specification defines this as a multi-stage pipeline connecting Docker-based traffic generation, Zeek, feature engineering, Kafka, ML inference, Elasticsearch, FastAPI, and the frontend dashboard.
Aegis focuses on passive detection rather than active network intervention.
The system:
- Observes traffic and network metadata.
- Converts network telemetry into ML-ready feature rows.
- Streams features through Kafka.
- Combines supervised classification with unsupervised anomaly detection.
- Produces human-readable evidence using SHAP.
- Stores positive detections for investigation.
- Correlates multiple detections into possible attack kill chains.
- Exposes live operational metrics for throughput and latency.
- Presents security events through a centralized dashboard.
Zeek processes captured traffic and produces structured logs such as:
conn.logdns.logssl.loghttp.logquic.log
The feature pipeline derives:
- Flow-level traffic statistics
- Source/destination fan-out and fan-in
- Inter-arrival-time statistics
- Connection frequency
- Destination entropy
- DNS characteristics
- TLS fingerprints
- JA3 / JA3S / JA4 fingerprints
- QUIC metadata
Aegis combines:
- XGBoost for known threat classification
- Isolation Forest for anomaly detection outside the known classes
SHAP is used to identify important features behind individual predictions and convert them into human-readable evidence.
Kafka decouples feature extraction, inference, and downstream processing.
Positive detections are written to Elasticsearch for querying, filtering, aggregation, and correlation.
The frontend provides:
- Live alert feed
- Alert details
- Threat statistics
- Throughput metrics
- Kill-chain visualization
- API/service status
The project defines seven scenarios:
Scenario Detection Class
SYN flood ddos
UDP flood ddos
C2 beaconing c2_beaconing
DGA traffic dns_anomaly
DNS tunneling dns_anomaly
Port scan port_scan
Data exfiltration exfiltration
Benign traffic benign
The specification groups the scenarios into seven labeled classes, with SYN/UDP floods representing the DDoS class and DGA/DNS tunneling representing DNS anomalies.
Docker containers provide controlled traffic generation and packet capture.
The intended environment contains:
- Attacker container
- Victim container
- Benign traffic container
- Zeek container
- Kafka + Zookeeper
- Elasticsearch
- Backend services
- Frontend
The lab is intended to use an isolated Docker network with no external routing.
Zeek converts PCAP/network traffic into structured network telemetry.
Important logs include:
conn.log
dns.log
ssl.log
http.log
quic.log
TLS telemetry includes JA3 and JA3S, with JA4 added as an extended fingerprinting capability.
The feature pipeline consumes a Zeek log directory and creates one
feature row per network flow using Zeek's uid as the join key.
Output is stored as scenario-specific Parquet data.
Example:
data/features/
├── syn_flood.parquet
├── udp_flood.parquet
├── c2_beacon.parquet
├── dga.parquet
├── dns_tunnel.parquet
├── port_scan.parquet
├── exfiltration.parquet
└── benign.parquet
Kafka provides asynchronous communication between feature extraction and inference.
The main topics are:
raw-features
inference-results
alerts
pipeline-metrics
The backend provides the inference and monitoring API.
Core endpoints include:
POST /infer
GET /alerts
GET /alerts/{id}
GET /stats
GET /metrics
GET /kill-chains
GET /health
Elasticsearch stores confirmed positive detections and supports:
- Filtering
- Pagination
- Threat-class aggregation
- Time-based statistics
- Source-IP analysis
- Kill-chain correlation
The frontend consumes the FastAPI APIs and displays security events and operational metrics.
A typical feature row follows this path:
PCAP
│
▼
Zeek
│
├── conn.log
├── dns.log
├── ssl.log
└── quic.log
│
▼
features/pipeline.py
│
▼
FeatureRow
│
▼
Staging Queue
│
▼
Kafka: raw-features
│
▼
kafka/consumer.py
│
▼
POST /infer
│
├── XGBoost
├── Isolation Forest
└── SHAP
│
▼
Inference Result
│
├── inference-results
├── alerts
└── Elasticsearch
│
▼
FastAPI
│
▼
React Dashboard
Aegis uses multiple feature families.
From conn.log:
packets_per_sec
bytes_per_sec
outbound_inbound_ratio
fan_out
fan_in
unique_dst_ips
unique_dst_ports
iat_mean
iat_std
iat_min
iat_max
connection_frequency
periodicity_score
The periodicity score is intended to highlight highly regular communication patterns that can indicate beaconing.
src_ip_entropy
dns_query_entropy
High DNS query entropy can be useful as a DGA indicator.
domain_length_mean
domain_length_max
subdomain_count
dns_record_type_a_ratio
dns_record_type_txt_ratio
dns_query_count
ja3_hash
ja3s_hash
ja4_hash
tls_version
cipher_suite_enc
is_tls
is_quic
quic_0rtt
quic_pkt_size_mean
quic_pkt_size_std
Non-QUIC flows are intended to receive zero-valued QUIC features.
XGBoost performs multi-class threat classification.
The training pipeline is intended to:
- Combine feature datasets.
- Join features with labels.
- Handle class imbalance.
- Split data into training, validation, and test sets.
- Train a multi-class XGBoost classifier.
- Tune hyperparameters with Optuna.
- Evaluate precision, recall, and F1.
- Save the trained model.
Initial target:
F1 > 0.85 per class
The specification proposes a starting configuration around:
n_estimators = 200
max_depth = 6
learning_rate = 0.1
subsample = 0.8
eval_metric = mlogloss
Isolation Forest is trained primarily on benign traffic.
Its purpose is to identify anomalous behavior that may not belong to the known supervised classes.
The contamination parameter is tuned against benign validation data.
An inference result contains:
{
"threat_class": "c2_beaconing",
"confidence": 0.94,
"anomaly_score": 0.71,
"evidence_features": []
}The final implementation may enrich the evidence field using SHAP output.
Aegis uses SHAP to explain individual model predictions.
The top contributing features are converted into human-readable evidence.
Examples include:
Feature Human-readable evidence
periodicity_score Repeated communication at fixed intervals
fan_out Single source contacted many ports or hosts
outbound_inbound_ratio Unusually large outbound data volume
dns_query_entropy High-entropy domain names (DGA indicator)
ja3_hash Unusual TLS client fingerprint (JA3)
ja4_hash Unusual TLS client fingerprint (JA4)
quic_pkt_size_std Irregular QUIC packet sizing
quic_0rtt QUIC 0-RTT resumption detected
src_ip_entropy Source IP scanning a wide range of destinations
The intended inference response contains the strongest evidence features rather than exposing raw model internals to the dashboard.
Kafka separates the high-frequency feature generation stage from inference.
The producer:
- Reads feature rows from the staging queue.
- Serializes them as JSON.
- Publishes them to
raw-features. - Uses short batching windows to avoid waiting indefinitely for a large batch.
The consumer:
- Reads from
raw-features. - Deserializes the message into a
FeatureRow. - Records the arrival time.
- Sends the feature vector to
/infer. - Measures inference pipeline latency.
- Publishes the inference result.
- Publishes pipeline metrics.
- Handles malformed messages without crashing.
Malformed messages are written to a dead-letter log.
The intended metrics stream contains values such as:
{
"ts": "<ISO>",
"flows_per_sec": 0,
"bytes_per_sec": 0,
"kafka_queue_depth": 0,
"pipeline_latency_ms": 0
}These metrics power the live throughput/latency view.
Aegis can correlate multiple alerts from the same source into a possible multi-stage attack.
The correlator is designed to:
- Examine recent alerts.
- Group alerts by source IP.
- Identify sources producing multiple threat classes.
- Assign a shared
kill_chain_id. - Update the corresponding Elasticsearch documents.
- Expose the resulting chains through
/kill-chains.
The dashboard represents each kill chain as a timeline of attack stages.
The React dashboard is designed around four major operational views.
Displays recent alerts with:
- Timestamp
- Source IP
- Destination IP
- Threat class
- Severity
- Confidence
Displays:
- Complete flow tuple
- Threat class
- Confidence
- Anomaly score
- Evidence
- Kill-chain relationship
Displays:
- Threat-class distribution
- Alert volume over time
- Top source IPs
Displays:
- Flows/sec
- MB/sec
- Kafka queue depth
- Pipeline latency
Displays:
- Source IP
- Attack stages
- Timestamp
- Confidence
- Severity
The target repository structure is:
Aegis/
│
├── docker-compose.yml
├── .env
├── .env.example
├── Makefile
├── LAB.md
├── INTEGRATION.md
├── API_CONTRACT.md
├── README.md
│
├── attacker/
│ ├── Dockerfile
│ └── scripts/
│ ├── c2_beacon.py
│ ├── exfiltration.py
│ ├── dga.py
│ ├── syn_flood.py
│ ├── udp_flood.py
│ ├── port_scan.py
│ └── dns_tunnel.py
│
├── victim/
│ └── Dockerfile
│
├── benign/
│ ├── Dockerfile
│ └── scripts/
│ └── benign_traffic.py
│
├── data/
│ ├── raw/
│ │ └── *.pcap
│ ├── labels/
│ │ └── *.json
│ └── features/
│ └── *.parquet
│
├── zeek/
│ └── scripts/
│ ├── main.zeek
│ ├── ja3.zeek
│ └── quic.zeek
│
├── features/
│ ├── schema.py
│ ├── pipeline.py
│ └── watcher.py
│
├── kafka/
│ ├── producer.py
│ ├── consumer.py
│ ├── metrics_producer.py
│ └── health_check.py
│
├── ml/
│ ├── train.py
│ ├── evaluate.py
│ ├── inference.py
│ ├── explainability.py
│ ├── model.joblib
│ └── tests/
│ └── test_explainability.py
│
├── api/
│ ├── main.py
│ └── correlator.py
│
├── es/
│ ├── init_index.py
│ └── health_check.py
│
└── dashboard/
└── src/
├── api/
│ ├── alerts.ts
│ ├── stats.ts
│ ├── metrics.ts
│ └── killChains.ts
└── views/
├── LiveFeed.tsx
├── AlertDetail.tsx
├── Stats.tsx
├── Metrics.tsx
└── KillChains.tsx
Team Member Role Main Responsibility
Nived DevOps / Pipeline Docker lab, Kafka, Elasticsearch, integration
Ebin Backend Shared schema, watcher/Kafka glue, FastAPI, APIs, correlation
Aadi ML Traffic generation, Zeek/QUIC work, Isolation Forest, inference
Gowri ML Feature engineering, JA3/JA4/QUIC features, SHAP
Sanjay ML Labels, dataset preparation, XGBoost training/evaluation
The project is designed around:
- Docker and Docker Compose
- Python 3
- Node.js/npm
- Zeek
- Kafka + Zookeeper
- Elasticsearch
- A modern web browser
The exact versions and installation commands should be kept in the repository's environment/setup documentation.
git clone <repository-url>
cd AegisCreate the local environment file from the example:
cp .env.example .envOn Windows PowerShell:
Copy-Item .env.example .envDo not commit secrets or machine-specific credentials.
Use the project's Docker/Makefile commands once configured:
docker compose up -dVerify the isolated network and service health before generating traffic.
Run the scenario scripts in the controlled lab.
Expected raw captures include:
data/raw/
├── syn_flood.pcap
├── udp_flood.pcap
├── c2_beacon.pcap
├── dga.pcap
├── dns_tunnel.pcap
├── port_scan.pcap
└── exfiltration.pcap
Benign traffic should also be generated for the benign dataset.
The DNS-tunnel scenario uses iodine in the isolated Docker lab. The attacker
container (10.10.0.2) runs iodined; the victim container (10.10.0.3)
runs the iodine client. Both containers need /dev/net/tun.
The validated lab parameters are:
domain: tunnel.lab
password: test
tunnel-side server address: 192.168.99.1
final victim capture: /pcaps/dns_tunnel.pcap
Run these commands in separate terminal sessions:
# Attacker (10.10.0.2)
iodined -f -P test 192.168.99.1 tunnel.lab
# Victim (10.10.0.3), start capture before the client
tcpdump -i eth0 -U -w /pcaps/dns_tunnel.pcap
# Victim (10.10.0.3)
iodine -f -P test 10.10.0.2 tunnel.labUse python attacker/scripts/dns_tunnel.py --help to print the same workflow
with configurable server, domain, password, tunnel-side address, and capture
path. The final capture is an authentic iodine session: direct client-to-server
operation exposes initial DNS negotiation and may then use raw UDP transport.
The -r DNS-forced experiment did not complete query-type autodetection and is
not the final workflow.
This hackathon lab simulates the authoritative DNS relationship within its
isolated Docker network. A real deployment would require DNS delegation of the
tunnel domain to the iodined endpoint; no public DNS delegation exists in
this lab.
Run Zeek against the captured traffic and produce the required logs.
The feature pipeline consumes the Zeek log directory rather than individual feature files.
The feature engineering pipeline produces scenario-specific Parquet files under:
data/features/
Train the XGBoost classifier and Isolation Forest using the prepared feature datasets.
The final artifacts are intended to include:
ml/model.joblib
ml/optuna_study.pkl
ml/classification_report.txt
ml/confusion_matrix.png
Start the FastAPI service and verify:
GET /health
The health endpoint is intended to report service dependencies, model state, and uptime.
From the dashboard directory:
npm install
npm run devThe frontend connects to the FastAPI service and displays live detections and metrics.
Configuration should be supplied through environment variables rather than hardcoded values.
Important variables include:
KAFKA_BOOTSTRAP
KAFKA_TOPIC_FEATURES
KAFKA_TOPIC_RESULTS
KAFKA_TOPIC_METRICS
KAFKA_CONSUMER_GROUP
INFER_URL
INFER_TIMEOUT
ES_HOST
ES_PORT
Keep local configuration in .env and provide safe
defaults/documentation in .env.example.
Aegis uses unit tests for individual pipeline components and integration testing for the complete system.
Run the Python test suite with:
pytest -vUnit testing should cover areas such as:
- Feature/schema validation
- File watcher behavior
- Feature serialization
- Kafka producer behavior
- Kafka consumer deserialization
- Inference request handling
- Rolling pipeline statistics
- Explainability output
Integration testing should verify:
Traffic
→ Zeek
→ Feature Pipeline
→ Kafka
→ FastAPI
→ Elasticsearch
→ Dashboard
The integration environment should also verify that all required services and Kafka topics are available.
The ML pipeline is intended to use:
70% Training
15% Validation
15% Test
with stratification by class.
Evaluation includes:
- Precision
- Recall
- F1 score
- Confusion matrix
- Classification report
- Anomaly-score analysis
- Hyperparameter tuning results
The project specification sets a target of F1 > 0.85 per class.
Model evaluation should be performed on held-out data that is not used during training.
Aegis's traffic-generation environment is designed specifically for controlled testing.
The Docker lab should:
- Use a private isolated network.
- Prevent external routing/NAT from the experiment network.
- Keep attack tooling inside the lab.
- Use dedicated attacker, victim, and benign containers.
- Avoid exposing unnecessary victim ports.
- Keep credentials and secrets outside source control.
Attack-generation scripts are intended for the isolated SIH test environment only.
Aegis is developed as a team project with component ownership.
A typical feature workflow is:
git switch main
git pull --rebase origin main
git switch -c feature/<feature-name>
# make changes
pytest -v
git add .
git commit -m "Describe the change"
git push -u origin feature/<feature-name>Create a Pull Request against main after testing.
Before merging a feature branch:
- Pull the latest
main. - Rebase the feature branch when appropriate.
- Run the test suite.
- Review the changed files.
- Push the final branch.
- Open the Pull Request.
- Merge after review.
- Delete the feature branch after successful merge if it is no longer needed.
The architecture and component responsibilities are organized around a staged build:
1. Docker + isolated traffic lab
↓
2. Traffic generation + PCAP capture
↓
3. Zeek telemetry
↓
4. Feature engineering
↓
5. Kafka streaming
↓
6. ML training + inference
↓
7. FastAPI + Elasticsearch
↓
8. React dashboard
↓
9. Full integration test + demo rehearsal
The project specification identifies the full integration test and PCAP replay demonstration as the final validation stage.
Potential extensions that fit naturally into the architecture include:
- Additional network protocols and fingerprints
- More anomaly-detection strategies
- Online/continual model updates
- Distributed Kafka deployments
- More sophisticated alert correlation
- Additional kill-chain stages
- Historical performance analytics
- Model drift monitoring
- Automated dataset validation
- Additional explainability methods
Project: Aegis
Theme: Passive AI/ML Network Threat Detection
Problem Statement: 145
Focus: Network telemetry → ML detection → explainable alerts →
real-time monitoring
Built as a collaborative SIH 2026 project with a focus on an end-to-end, reproducible, isolated network-threat detection pipeline.