Wikipedia to Wikidata Link Resolver

This project extracts unique Wikipedia URLs from a local knowledge graph repository (such as the MaRDI Knowledge Graph) and maps them directly to their corresponding Wikidata IDs (QIDs) using the official English Wikipedia Action API.

To maximize throughput and prevent hitting standard API request throttles, the pipeline is fully split into two independent, isolated execution steps and supports OAuth 2.0 authentication.

---

Architecture Overview

1. getUrls.py: Queries the MaRDI SPARQL endpoint to fetch all unique Wikipedia links present in the dataset. It isolates the heavy graph operation and saves a clean, raw text cache to wikipedia_urls.txt.
2. resolveUrls.py: Reads your text cache locally, contacts the English Wikipedia Action API (https://wikipedia.org) in a strict 1-by-1 operation, and logs results directly to wikidata_mapping.csv.

---

Fast Mode Integration via OAuth 2.0 (Required for Speed)

Anonymous IP addresses querying the Wikipedia endpoints frequently hit heavy 429 Client Error: Too Many Requests traffic blocks. To unlock high-volume speeds (up to hundreds of requests per minute), you must use an official Owner-Only OAuth 2.0 Access Token.

How to generate your high-speed token:

1. Log into your standard account on Wikipedia.
2. Go to meta:Special:OAuthConsumerRegistration/propose/oauth2.
3. Complete the required field forms (Application Name, Description).
4. Crucial: Ensure you check the box titled "This consumer is for use only by user" to instantly self-approve the configuration without waiting for administrator review.
5. Under requested grants/permissions, select at least "Basic Rights (Read)".
6. Save the application and copy your generated Access Token string.

---

Setup & Running the Project

1. Installation

Install the necessary networking dependency inside your virtual environment setup:

pip install requests

2. Step 1: Populate the URL Cache

Run the first step to fetch everything from your repository. This writes directly into wikipedia_urls.txt:

python step1_fetch.py

3. Step 2: High-Speed Link Resolution

You can feed your generated token to the script using two different secure approaches. The script will dynamically process outstanding items 1-by-1 and automatically skip records that are already successfully cached inside wikidata_mapping.csv.

Option A: Automatic Pipeline Integration (Recommended)

Export your secret token to your environment layer before execution. This entirely bypasses interactive terminals:

export WIKIPEDIA_TOKEN="your_copied_oauth2_access_token"
python resolveUrls.py

Option B: Secure Terminal Prompt

If the environment variable is missing, the program will securely request your credentials via a masked interface:

python resolveUrls.py
# The program will output:
# WIKIPEDIA_TOKEN environment variable not found.
# Please paste your Wikipedia OAuth2 Access Token: [Your input stays hidden]

---

Output Formatting

The mapped matrix is updated in real-time after every single lookup transaction inside wikidata_mapping.csv:

qid,original_link
Q12345,https://wikipedia.org
NO_WIKIDATA_LINK,https://wikipedia.org

Import to MaRDI

Copy file to importer pod:

kubectl config set-context mardi --namespace=production
kubectl cp input.txt $(kubectl get pod -l app=importer -o jsonpath='{.items[0].metadata.name}'):/app/input.txt -c importer

On the pod

 kubectl exec -it deployments/importer -- bash

On th run

while IFS= read -r qid || [ -n "$qid" ]; do
  qid=$(echo "$qid" | tr -d "\r\n ")
  if [ -n "$qid" ]; then
    echo "----------------------------------------"
    echo "Processing item token: $qid"
    echo "----------------------------------------"
    python -m cli.importer_cli import-wikidata --qids "$qid"
  fi
done < /app/input.txt

Merging zbMATH Open concepts

The zbm_concepts.py helper prepares zbMathConcepts.json for the MaRDI import flow. It reuses the repository's existing wikipedia_urls.txt and wikidata_mapping.csv workflow, then writes a quasi QuickStatements CSV together with bookkeeping outputs that show what is already reflected in the MaRDI knowledge graph.

extract-urls collects all Wikipedia links from zbMathConcepts.json into the existing cache. build-csv consumes wikidata_mapping.csv and generates the quasi QuickStatements file zbmath-quasi-quickstatements.csv plus the preprocessing report zbmath-preprocess-report.csv. bookkeep-zbmath-concepts reads a first-pass import CSV and writes zbmath-bookkeeping.csv, zbmath-bookkeeping-report.csv, and zbmath-bookkeeping-stats.csv.

python3 zbm_concepts.py extract-urls ~/Documents/zbMathConcepts.json
python3 resolveUrls.py
python3 zbm_concepts.py build-csv ~/Documents/zbMathConcepts.json
python3 zbm_concepts.py update-stats
python3 -m unittest test_preprocess_zbmath_concepts
export JOBRUNNER_INSTANCE=`kubectl get pod -l app=wikibase-jobrunner -o jsonpath='{.items[0].metadata.name}'`
kubectl cp zbmath-wikidata-first-pass.csv $JOBRUNNER_INSTANCE:/var/www/html/w/extensions/MathSearch/maintenance/zbMATH.csv -c mediawiki
kubectl exec -it $JOBRUNNER_INSTANCE -- bash
cd maintenance
php ImportQuickStatements.php zbMATH.csv 

The unit test module test_preprocess_zbmath_concepts.py covers Wikipedia URL extraction, source metadata bookkeeping, quasi QuickStatements row generation, and the storage stats denominator logic.

The following items are problematic and need to be manually resolved:

{
  "qids": [
    "Q1064218",
    "Q110535082",
    "Q115489271",
    "Q12018378",
    "Q120756609",
    "Q121224584",
    "Q1568269",
    "Q165498",
    "Q173817",
    "Q1756677",
    "Q179916",
    "Q187672",
    "Q190524",
    "Q1974553",
    "Q19979500",
    "Q2206263",
    "Q339444",
    "Q3489473",
    "Q3650511",
    "Q37970",
    "Q38433",
    "Q677864",
    "Q704747",
    "Q7756337",
    "Q88997156",
    "Q932001",
    "Q9378773"
  ],
  "count": 27,
  "results": {
    "Q1064218": {
      "qid": "Q6674535",
      "status": "success"
    },
    "Q110535082": {
      "qid": "Q7278657",
      "status": "success"
    },
    "Q115489271": {
      "qid": "Q7273191",
      "status": "success"
    },
    "Q12018378": {
      "qid": "Q6480612",
      "status": "success"
    },
    "Q120756609": {
      "qid": "Q7266378",
      "status": "success"
    },
    "Q121224584": {
      "qid": "Q7283136",
      "status": "success"
    },
    "Q1568269": {
      "qid": null,
      "status": "error",
      "error": "'NoneType' object is not subscriptable"
    },
    "Q165498": {
      "qid": "Q6534305",
      "status": "success"
    },
    "Q173817": {
      "qid": "Q6673728",
      "status": "success"
    },
    "Q1756677": {
      "qid": "Q6674576",
      "status": "success"
    },
    "Q179916": {
      "qid": null,
      "status": "error",
      "error": "'NoneType' object is not subscriptable"
    },
    "Q187672": {
      "qid": null,
      "status": "error",
      "error": "'NoneType' object is not subscriptable"
    },
    "Q190524": {
      "qid": "Q6534306",
      "status": "success"
    },
    "Q1974553": {
      "qid": null,
      "status": "error",
      "error": "'NoneType' object is not subscriptable"
    },
    "Q19979500": {
      "qid": "Q7279180",
      "status": "success"
    },
    "Q2206263": {
      "qid": "Q6675169",
      "status": "success"
    },
    "Q339444": {
      "qid": "Q6534320",
      "status": "success"
    },
    "Q3489473": {
      "qid": "Q6673996",
      "status": "success"
    },
    "Q3650511": {
      "qid": "Q7273795",
      "status": "success"
    },
    "Q37970": {
      "qid": null,
      "status": "error",
      "error": "'NoneType' object is not subscriptable"
    },
    "Q38433": {
      "qid": null,
      "status": "error",
      "error": "'NoneType' object is not subscriptable"
    },
    "Q677864": {
      "qid": "Q6673747",
      "status": "success"
    },
    "Q704747": {
      "qid": null,
      "status": "error",
      "error": "'NoneType' object is not subscriptable"
    },
    "Q7756337": {
      "qid": "Q7274372",
      "status": "success"
    },
    "Q88997156": {
      "qid": "Q7271941",
      "status": "success"
    },
    "Q932001": {
      "qid": null,
      "status": "error",
      "error": "'NoneType' object is not subscriptable"
    },
    "Q9378773": {
      "qid": "Q7275171",
      "status": "success"
    }
  },
  "all_imported": false
}

Undid Import260709060709, Quickstatements260709120750

./run rollbackEdits.php --user Import260709060709