From 36cf054eea1ed71f19fa145176a5a1e6f6231220 Mon Sep 17 00:00:00 2001
From: Ming-Jer Lee <mingjerli@gmail.com>
Date: Thu, 23 Jul 2026 19:26:43 -0700
Subject: [PATCH] docs: add SQL RAG schema-context application to README

Position clgraph as the context layer for retrieval-augmented
text-to-SQL systems: a new section after Text-to-SQL shows how to
build PII-filtered, description-rich retrieval documents from the
lineage graph and expand retrieved context with backward tracing.
Also adds SQL RAG to the applications list, LLM feature bullets,
and Use Cases. Both code examples verified against the current API.
---
 README.md | 48 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 48 insertions(+)

diff --git a/README.md b/README.md
index 320243e..60b42d3 100644
--- a/README.md
+++ b/README.md
@@ -13,6 +13,7 @@ The column lineage graph captures how data flows through your SQL: every column
 - **PII compliance?** Mark nodes, propagate through edges.
 - **DAG construction?** Project to table-level dependencies.
 - **Documentation?** Describe nodes and their relationships.
+- **SQL RAG?** Schema context is graph export.
 - **AI applications?** Context engineering is graph traversal.
 
 **One graph. Many applications.**
@@ -45,6 +46,7 @@ Traditional tools reverse-engineer lineage from query logs and database metadata
 ### LLM-Powered Features
 - **Lineage Agent** — Natural language interface to query lineage ("Where does revenue come from?")
 - **Text-to-SQL** — Schema-aware SQL generation with column descriptions as context
+- **SQL RAG context** — Lineage-aware, PII-filtered schema context for retrieval pipelines
 - **Programmatic tools** — 11 built-in tools for lineage, schema, and governance queries
 
 ### MCP Server (AI Integration)
@@ -616,6 +618,51 @@ print(result.data["sql"])
 # LIMIT 10
 ```
 
+### SQL RAG: Schema Context for Retrieval Pipelines
+
+Text-to-SQL RAG systems live or die on the quality of the schema context they retrieve. Raw DDL tells an LLM that `lifetime_value` is a FLOAT—the lineage graph tells it the column is derived from order amounts, which tables it flows through, and whether it is safe to expose.
+
+clgraph is the context layer for these systems: it does not do retrieval or embedding, but it produces lineage-aware, description-rich, PII-filtered documents that any retrieval stack (LangChain, LlamaIndex, or your own) can index:
+
+```python
+from clgraph import Pipeline
+
+queries = [
+    ("customers", """
+        CREATE TABLE analytics.customers AS
+        SELECT
+            customer_id,       -- Unique customer identifier
+            email,             -- Customer email address [pii: true]
+            lifetime_value     -- Total revenue from this customer in USD
+        FROM raw.customers
+    """),
+]
+pipeline = Pipeline(queries, dialect="bigquery")
+
+# Build one retrieval document per table, filtering out PII columns
+documents = []
+for table_name in pipeline.table_graph.tables:
+    lines = [f"Table: {table_name}"]
+    for col in pipeline.get_columns_by_table(table_name):
+        if col.pii:
+            continue  # keep sensitive columns out of the retrieval store
+        lines.append(f"  {col.column_name}: {col.description or 'no description'}")
+    documents.append("\n".join(lines))
+
+print(f"Built {len(documents)} documents for embedding")
+```
+
+At question time, expand the retrieved context with lineage so the LLM sees where a column actually comes from:
+
+```python
+# The retriever matched analytics.customers.lifetime_value —
+# pull in its upstream sources as additional context
+sources = pipeline.trace_column_backward("analytics.customers", "lifetime_value")
+print(f"lifetime_value is derived from: {[s.full_name for s in sources]}")
+```
+
+For columns without inline comments, `pipeline.generate_all_descriptions()` fills the gaps with lineage-aware LLM descriptions, and `pipeline.propagate_all_metadata()` ensures PII flags flow through every transformation before you filter. The full graph is also available as a single JSON document via `pipeline.to_json()` if you prefer to chunk it yourself.
+
 ### Lineage Tools (Programmatic Access)
 
 Use tools directly without the agent for programmatic access:
@@ -1171,6 +1218,7 @@ print(f"Dialect: snowflake, Tables: {list(pipeline.table_graph.tables.keys())}")
 - **Pipeline Optimization**: Identify unused columns and redundant transformations
 - **Data Quality**: Trace data issues back to their source
 - **Documentation**: Auto-generate data flow diagrams and column descriptions
+- **SQL RAG**: Feed lineage-aware, PII-filtered schema context to retrieval-augmented text-to-SQL systems
 
 ## Development