GitHub PR Scraper - Production-Grade Dataset Builder

A complete, production-ready Python scraper for collecting GitHub pull request review comments to build high-quality datasets for fine-tuning code review LLMs.

🎯 Project Overview

Objective: Create a structured dataset of real GitHub PR review comments with automatic language detection and severity labeling.

Dataset Format:

{
  "pr_id": "12345",
  "repo_name": "psf/requests",
  "file_path": "requests/models.py",
  "line_number": 456,
  "code_diff": "def prepare_request(...)",
  "review_comment": "This could cause a memory leak if not properly handled.",
  "language": "python",
  "severity_label": "critical"
}

📁 Project Structure

github_pr_scraper/
│
├── scraper/
│   ├── __init__.py              # Package initialization
│   ├── github_client.py         # GitHub API client with rate limiting
│   ├── repository_scraper.py    # PR and file fetching
│   ├── review_extractor.py      # Comment extraction & cleaning
│   ├── dataset_builder.py       # Dataset export & statistics
│   └── utils.py                 # Utility functions & logging
│
├── data/                        # Output directory
│   ├── raw_reviews.csv
│   ├── cleaned_reviews.csv
│   └── dataset.jsonl
│
├── logs/                        # Scraper logs
│
├── config.py                    # Configuration settings
├── main.py                      # Main orchestrator
├── requirements.txt             # Python dependencies
├── .env.example                 # Environment variables template
├── .env                         # (Create this - DO NOT COMMIT)
└── README.md                    # This file

🚀 Setup Instructions

Step 1: Clone or Create Project

# Create project directory
mkdir github_pr_scraper
cd github_pr_scraper

# Initialize git (optional)
git init

Step 2: Set Up Python Environment

# Create virtual environment (Python 3.10+)
python -m venv venv

# Activate virtual environment
# On Windows:
venv\Scripts\activate
# On macOS/Linux:
source venv/bin/activate

Step 3: Install Dependencies

pip install -r requirements.txt

Step 4: Configure GitHub Token

1. Generate GitHub Personal Access Token:

   • Go to https://github.com/settings/tokens
   • Click "Generate new token" → "Generate new token (classic)"
   • Select scopes:
     • ✅ repo (Full control of private repositories)
     • ✅ read:user (Read user profile data)
   • Copy the token

2. Create .env file:

   cp .env.example .env

3. Edit .env:

   GITHUB_TOKEN=ghp_your_actual_token_here
   TARGET_REPOSITORY=psf/requests
   MAX_PRS_TO_SCRAPE=1000
   LOG_LEVEL=INFO
   

Step 5: Run the Scraper

# Run with default configuration
python main.py

# Or run with custom settings
python main.py --max-prs 50 --repo owner/repo

📊 Features

✅ Data Collection

• Pagination Support: Handles large datasets efficiently
• Rate Limit Management: Automatic retry with exponential backoff
• Progress Tracking: Real-time progress bars with tqdm
• Resumable: Save checkpoints every 100 records

✅ Data Cleaning

Filters out low-quality comments:

• Bot comments (dependabot, github-actions, codecov, etc.)
• Low-quality reactions (LGTM, Thanks, +1, Approved)
• Empty or too-short comments (< 20 characters)
• Duplicate records

✅ Language Detection

Automatically detects from file extensions:

.py → Python
.js → JavaScript
.ts → TypeScript
.java → Java
.cpp → C++
... and 20+ more languages

✅ Severity Labeling

Automatic classification based on keywords:

Critical (Security/Bugs):

• Keywords: security, vulnerability, injection, crash, bug, exploit

Warning (Performance/Optimization):

• Keywords: performance, optimize, memory, scalability, refactor

Info (Default):

• Everything else

✅ Output Formats

1. CSV Format (raw_reviews.csv)

pr_id,repo_name,file_path,line_number,code_diff,review_comment,language,severity_label
12345,psf/requests,requests/models.py,456,"def prepare_request(...)","Check error handling",python,warning

2. Cleaned CSV (cleaned_reviews.csv)

• Same format as above, with low-quality records removed

3. JSONL Format (dataset.jsonl)

{"instruction": "Review the following pull request diff.", "input": "def prepare_request(...)", "output": "Check error handling"}
{"instruction": "Review the following pull request diff.", "input": "class Session(...)", "output": "Consider thread safety"}

📈 Performance & Scalability

Current Configuration

• Test Mode: 100 PRs (configured in .env)
• Sample Size: ~1,000-5,000 review comments
• Execution Time: ~10-30 minutes depending on network

Scale to Production

Modify config.py to handle 50,000+ samples:

# In config.py
MAX_PRS_TO_SCRAPE=5000          # Increase from 100
MAX_COMMENTS_PER_PR=50          # Adjust as needed
BATCH_SIZE=100                  # Keep for memory efficiency
CHECKPOINT_INTERVAL=500         # Increase for efficiency

Estimated Resources for 50K+ Records

• Time: 4-8 hours (depends on network and GitHub rate limits)
• Storage: ~100-200 MB for CSV, ~80-150 MB for JSONL
• Memory: ~1-2 GB (managed by checkpointing)

🔧 Configuration Guide

Main Settings (config.py)

# API Configuration
BATCH_SIZE = 100                # Records per batch
REVIEWS_PER_PAGE = 30          # Reviews per API call
PRS_PER_PAGE = 30              # PRs per API call
MAX_RETRIES = 3                # Retry attempts on failure
RETRY_DELAY = 5                # Delay between retries (seconds)

# Data Collection
MAX_PRS_TO_SCRAPE = 1000        # Test with 100, scale to 50K+
MAX_COMMENTS_PER_PR = 50       # Max reviews per PR
MIN_COMMENT_LENGTH = 20        # Minimum characters

# Output
CHECKPOINT_INTERVAL = 100      # Save progress every N records

📝 Output Examples

Sample Dataset Statistics

======================================================
DATASET STATISTICS
======================================================
Total Records: 1,247
Total Pull Requests: 87
Total Files: 342
Total Repositories: 1

Languages:
  python: 892
  javascript: 185
  other: 170

Severity Distribution:
  info: 612
  warning: 445
  critical: 190

Comment Statistics:
  - Average Length: 156 characters
  - Min Length: 21 characters
  - Max Length: 892 characters

Date Range:
  - Earliest: 2023-01-15T10:30:45
  - Latest: 2024-12-10T18:22:33
======================================================

Sample Records

Record 1 - Critical Severity

{
  "pr_id": 12345,
  "repo_name": "psf/requests",
  "file_path": "requests/models.py",
  "line_number": 456,
  "code_diff": "def prepare_headers(self):\n    return self._headers or {}",
  "review_comment": "This could cause a security vulnerability. Consider validating headers against known injection patterns.",
  "language": "python",
  "severity_label": "critical",
  "comment_author": "security-reviewer",
  "created_at": "2024-06-15T10:30:45"
}

Record 2 - Warning Severity

{
  "pr_id": 12347,
  "repo_name": "psf/requests",
  "file_path": "requests/adapters.py",
  "line_number": 289,
  "code_diff": "def get_connection(self, url):\n    # Connection logic",
  "review_comment": "Performance concern: consider caching the parsed URL to avoid repeated parsing in loops",
  "language": "python",
  "severity_label": "warning",
  "comment_author": "performance-team",
  "created_at": "2024-06-20T14:22:10"
}

🐛 Troubleshooting

Issue: "Invalid GitHub Token"

Error: GitHub authentication failed: Invalid GitHub Token

Solution:

• Verify token at https://github.com/settings/tokens
• Ensure it has repo scope
• Regenerate if necessary

Issue: "Rate Limit Exceeded"

Error: Rate limit exceeded while fetching pull requests

Solution:

• Automatic retry with exponential backoff handles this
• For aggressive scraping, use GitHub App instead of personal token
• Check rate limit status: https://api.github.com/rate_limit

Issue: "Repository Not Found"

Error: Failed to fetch repository: {404, ()}

Solution:

• Check repository format: owner/repo (not owner-repo)
• Verify repository is public or token has access
• Try: curl https://api.github.com/repos/psf/requests

Issue: "Empty Dataset"

Warning: No review comments were extracted

Possible Causes:

• Repository has no closed PRs with reviews
• All comments were filtered (bots, low quality)
• Try a different repository

🔐 Security & Best Practices

✅ DO

• Store token in .env (never commit to git)
• Use .gitignore to exclude .env
• Rotate token regularly
• Use fine-grained tokens with minimal scopes
• Monitor API usage for anomalies

❌ DON'T

• Commit .env with actual tokens
• Share tokens in code or documentation
• Use tokens in logs
• Skip rate limiting checks
• Ignore errors in production

.gitignore Setup

.env
.env.local
*.pyc
__pycache__/
venv/
logs/
data/*.csv
data/*.jsonl
.DS_Store

📚 API Reference

GitHubClient

from scraper.github_client import GitHubClient

client = GitHubClient(token="ghp_...")
repo = client.get_repository("psf/requests")
prs = client.get_pull_requests(repo, state="closed", max_prs=100)
comments = client.get_pull_request_review_comments(pr)
status = client.get_rate_limit_status()

RepositoryScraper

from scraper.repository_scraper import RepositoryScraper

scraper = RepositoryScraper(github_client, config)
pr_data = scraper.scrape_repository("psf/requests")
comments = scraper.get_pr_review_comments(pr)
stats = scraper.get_repository_stats("psf/requests")

ReviewExtractor

from scraper.review_extractor import ReviewExtractor

extractor = ReviewExtractor(config)
reviews = extractor.extract_reviews_from_pr(pr_data, comments)
language = extractor.detect_language("src/main.py")  # "python"
severity = extractor.label_severity("Security vulnerability...")  # "critical"
cleaned = extractor.remove_duplicates(records)

DatasetBuilder

from scraper.dataset_builder import DatasetBuilder

builder = DatasetBuilder(config)
builder.add_records(records)
builder.save_raw_csv()
builder.save_cleaned_csv(cleaned_records)
builder.save_jsonl(cleaned_records)
stats = builder.get_statistics(records)
builder.print_statistics(records)

🎓 Use Cases

1. Train Code Review LLM

# Load JSONL dataset for fine-tuning
import json
training_data = []
with open("data/dataset.jsonl") as f:
    for line in f:
        training_data.append(json.loads(line))

# Use with your favorite framework
# model.finetune(training_data)

2. Code Review Benchmark Dataset

# Use cleaned CSV for analysis
import pandas as pd
df = pd.read_csv("data/cleaned_reviews.csv")
print(df.groupby('severity_label').size())

3. Multi-Repository Analysis

# Scrape multiple repositories
repositories = [
    "psf/requests",
    "torvalds/linux",
    "facebook/react"
]

for repo in repositories:
    scraper.run(repo)

📈 Expected Dataset Size

For 100 PRs from psf/requests:

• Raw Records: 1,200-1,500
• After Cleaning: 900-1,200
• Removed Records: ~20-30% (bots, duplicates, low-quality)
• File Size (JSONL): 80-120 MB

🤝 Contributing

Improvements welcome! Consider:

• Supporting GraphQL API for better performance
• Adding sentiment analysis
• Implementing additional severity labels
• Supporting multiple repository scraping
• Adding data augmentation

📄 License

MIT License - See LICENSE file for details

🆘 Support

For issues or questions:

1. Check troubleshooting section
2. Review logs in logs/ directory
3. Check GitHub API status: https://www.githubstatus.com
4. Review PyGithub documentation: https://pygithub.readthedocs.io

🎯 Next Steps

1. ✅ Set up environment and install dependencies
2. ✅ Generate and configure GitHub token
3. ✅ Test with a small repository
4. ✅ Monitor logs and output
5. ✅ Scale up to larger repositories
6. ✅ Export cleaned dataset for model training

---

Built with ❤️ for the ML/Data Engineering community

Version: 1.0.0 | Last Updated: 2024-12 | Python: 3.10+