Agentic AI-Driven Data Engineering: A Framework for Self-Directed Enterprise Data Pipelines

Anil Kumar Kolla 1, * and Ameya Kokate 2

1 Department of Information Technology, HonorHealth, Scottsdale, Arizona, USA.
2 Department of Data Science and Analytics, HonorHealth, Scottsdale, Arizona, USA.
* Corresponding Author
ORCID Details
 
Research Article
International Journal of Frontline Research in Engineering and Technology, 2026, 06(01), 001–008.
Article DOI: 10.56355/ijfret.2026.6.1.0015
Publication history: 
Received on 14 August 2026; revised on 22 September 2026; accepted on 24 September 2026
 
Abstract: 
Enterprise data pipelines have grown in scale and interdependency faster than the operational practices used to maintain them. Pipeline failures are still predominantly diagnosed and remediated by human engineers reading logs, tracing lineage, and applying tacit knowledge, which produces long mean-time-to-recovery and concentrates institutional knowledge in a small number of individuals. This paper proposes an agentic architecture in which large language model (LLM) agents, constrained by explicit tool interfaces and policy guardrails, perform continuous monitoring, root-cause triage, and bounded remediation of production data pipelines. The framework specifies five cooperating agent roles — Observer, Diagnostician, Remediator, Verifier, and Orchestrator — operating over a shared state store containing pipeline metadata, lineage graphs, data-quality expectations, and incident history. We define a capability-tiering model that maps each remediation action to one of four autonomy levels (observe, recommend, act-with-approval, act-autonomously), and we describe a human-in-the-loop governance layer providing approval routing, immutable action logging, and rollback. We further propose an evaluation protocol covering task success rate, mean-time-to-detection, mean-time-to-recovery, false-remediation rate, and escalation precision, together with a reference workload derived from common ingestion, transformation, and schema-evolution failure modes. The paper concludes by analysing failure modes specific to agentic operation — hallucinated root causes, unsafe tool invocation, reward-free drift, and accountability gaps — and argues that autonomy must be earned incrementally through measured reliability rather than granted by architectural default.

 

Keywords: 
Agentic Systems, Large Language Models, Data Engineering, DataOps, Pipeline Orchestration, Human-in-the-Loop, Data Reliability, Lakehouse.
 
Full text article in PDF: 
 
Copyright information: 
Copyright © 2026 Author(s) retain the copyright of this article. This article is published under the terms of the Creative Commons Attribution Liscense 4.0