Azure-Based External Knowledge Ingestion Pipeline
An event-driven backend that turns documents from internal service portals and SharePoint into RAG knowledge with no manual work
To consolidate documents scattered across internal service portals and SharePoint into RAG knowledge, I built an event-driven ingest pipeline: Azure Logic Apps → Blob Storage → an App Service API. Logic Apps catches change events from each source and calls the embedding/ingest API deployed on Azure App Service, which runs per-type parsing/chunking/embedding into a Vector DB and uses safe-replace indexing — swapping the old index only after the new one succeeds — to prevent data loss.
Using documents scattered across internal service portals and SharePoint in RAG previously meant manually downloading, extracting, chunking, and embedding each time, and edits mixed old and new content unless vectors were safely replaced. So I deployed the heavy parsing, embedding, and storage work as an API on Azure App Service, and had Logic Apps — triggered by changes from each source — call it through Blob to automate ingestion. With per-type routing, token-limit re-validation, domain classification of external-dependency errors (document parser/OpenAI/Vector DB), and replacing the index only after new processing succeeds, knowledge expands safely while the service runs.
- Auto-reflects document changes from internal service portals and SharePoint into the RAG knowledge base with no manual work
- Prevented data loss and failure propagation during reindexing with safe-replace indexing and retry/error handling