Budgeted Two-Stage Pre-Validation for Retrieval-Grounded Machine Learning Pipelines: A Group-Held-Out Evaluation
DOI:
https://doi.org/10.54097/m33p3g15Keywords:
Machine learning pipelines, retrieval-guided generation, budgeted pre-validation, resource prediction, dependency satisfiability, uncertainty under shiftAbstract
Retrieval-grounded machine-learning pipeline synthesis must reconcile code templates with package versions and resource budgets. We study a two-stage pre-validator adapted from concrete dependency-satisfiability and resource-prediction mechanisms: an inexpensive retrieval-profile margin ranks candidate manifests; a budgeted second stage invokes a gradient-boosted peak-memory predictor and applies bounded repairs. Actual execution of 48 public-dataset learning workflows supplies memory profiles, while eight controlled configuration mutations per workload define explicit local feasibility outcomes. Four group-disjoint test folds provide 384 held-out candidates. At an exact 50% second-stage invocation budget, retrieval-prioritized validation yields 372/384 feasible configurations (96.9%), matching exhaustive checking, versus 281/384 (73.2%) for one matched random schedule. Across 300 random schedules the mean is 74.2%. A 16-group paired bootstrap puts the advantage over that seeded random baseline at 19.8–27.1 percentage points. Uncertainty offsets and semantic novelty are not independently beneficial on this coarse-fault benchmark: point prediction and a simple memory-margin ranking tie the proposed method. Calibration also deteriorates on a shifted fold. These findings support auditable, low-cost screening for synthetic configuration faults, without treating local feasibility as actual cloud deployment success or deterministic template synthesis as LLM generation.
Downloads
References
[1] Sculley, D., et al. (2015). Hidden technical debt in machine learning systems. In Advances in Neural Information Processing Systems (Vol. 28, pp. 2503–2511). https://research.google/pubs/hidden-technical-debt-in-machine-learning-systems/
[2] Polyzotis, N., Roy, S., Whang, S. E., & Zinkevich, M. (2017). Data management challenges in production machine learning. In Proceedings of the ACM SIGMOD International Conference on Management of Data (pp. 1723–1726). https://doi.org/10.1145/3035918.3054782
[3] Baylor, D., et al. (2017). TFX: A TensorFlow-based production-scale machine learning platform. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1387–1395). https://doi.org/10.1145/3097983.3098021
[4] Paleyes, A., Urma, R.-G., & Lawrence, N. D. (2023). Challenges in deploying machine learning: A survey of case studies. ACM Computing Surveys, 55(6). https://doi.org/10.1145/3533378
[5] Lewis, P., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems (Vol. 33, pp. 9459–9474). https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html
[6] Chen, J., Zhang, X., & Han, X. (2023). Explainable identification of tax-related financial reporting anomalies in U.S. public companies: Cross-fitted tax-consistency residuals and Owen-value Shapley propagation. World Journal of Economics and Business Research, 1(1), 33–45.
[7] Zhang, F., et al. (2023). RepoCoder: Repository-level code completion through iterative retrieval and generation. In Proceedings of EMNLP (pp. 2471–2484). https://doi.org/10.18653/v1/2023.emnlp-main.151
[8] Zhou, S., Alon, U., Xu, F. F., Wang, Z., Jiang, Z., & Neubig, G. (2023). DocPrompting: Generating code by retrieving the docs. In Proceedings of the International Conference on Learning Representations (ICLR). shturl.cc/nCJPRmwXG7DDrhA0eQTWT0odvp9BSm
[9] Wang, J., Ge, Y., Liu, J., & Jiang, B. (2024). Early warning of liquidity stress in U.S. banks with seasonally aligned frailty hazards and delayed-feedback conformal alarms. Journal of Trends in Finance and Economics, 1(1), 85–100.
[10] Wang, Y., Le, H., Gotmare, A., Bui, N., Li, J., & Hoi, S. (2023). CodeT5+: Open code large language models for code understanding and generation. In Proceedings of EMNLP (pp. 1069–1088). https://doi.org/10.18653/v1/2023.emnlp-main.68
[11] Le, H., Wang, Y., Gotmare, A. D., Savarese, S., & Hoi, S. C. H. (2022). CodeRL: Mastering code generation through pretrained models and deep reinforcement learning. In Advances in Neural Information Processing Systems (Vol. 35). https://doi.org/10.52202/068431-1549
[12] Asai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2024). Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In Proceedings of the International Conference on Learning Representations (ICLR). https://proceedings.iclr.cc/paper_files/paper/2024/hash/25f7be9694d7b32d5cc670927b8091e1-Abstract-Conference.html
[13] Schelter, S., Lange, D., Schmidt, P., Celikel, M., Biessmann, F., & Grafberger, A. (2018). Automating large-scale data quality verification. Proceedings of the VLDB Endowment, 11(12), 1781–1794. https://doi.org/10.14778/3229863.3229867
[14] Kuchnik, M., Klimovic, A., Simsa, J., Smith, V., & Amvrosiadis, G. (2022). Plumber: Diagnosing and removing performance bottlenecks in machine learning data pipelines. Proceedings of Machine Learning and Systems, 4. https://proceedings.mlsys.org/paper_files/paper/2022/hash/d0e90e9a9310570dfa643aa3b2da6e89-Abstract.html
[15] Husain, H., Wu, H.-H., Gazit, T., Allamanis, M., & Brockschmidt, M. (2019). CodeSearchNet challenge: Evaluating the state of semantic code search (Tech. Rep. MSR-TR-2019-45). Microsoft Research. https://arxiv.org/abs/1909.09436
[16] Robertson, S., & Zaragoza, H. (2009). The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 4(1–2), 1–174. https://doi.org/10.1561/1500000019
[17] Zhou, F. (2026). Moment-constrained side-force allocation for wind-assisted merchant ships: Hydrodynamic performance of retractable anti-leeway fins and flap rudders. Innovation and Technology Studies, 3(1), 27–41.
[18] Karpukhin, V., et al. (2020). Dense passage retrieval for open-domain question answering. In Proceedings of EMNLP (pp. 6769–6781). https://doi.org/10.18653/v1/2020.emnlp-main.550
[19] Zhao, W., Chen, T., Yang, J. S., & Qiu, L. (2026). AutoML-Pipeline: A RAG-enhanced code generation framework with pre-validation for cloud-native machine learning workflows. IEEE Access, 14, 41932–41945. https://doi.org/10.1109/ACCESS.2026.3673923
[20] Friedman, J. H. (2001). Greedy function approximation: A gradient boosting machine. Annals of Statistics, 29(5), 1189–1232. https://doi.org/10.1214/aos/1013203451
[21] Vovk, V., Gammerman, A., & Shafer, G. (2005). Algorithmic learning in a random world. Springer. https://doi.org/10.1007/b106715
[22] Angelopoulos, A. N., & Bates, S. (2023). Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning, 16(4), 494–591. https://doi.org/10.1561/2200000101
[23] Geifman, Y., & El-Yaniv, R. (2017). Selective classification for deep neural networks. In Advances in Neural Information Processing Systems (Vol. 30). https://proceedings.neurips.cc/paper/2017/hash/4a8423d5e91fda00bb7e46540e2b0cf1-Abstract.html
[24] Niculescu-Mizil, A., & Caruana, R. (2005). Predicting good probabilities with supervised learning. In Proceedings of the International Conference on Machine Learning (pp. 625–632). https://doi.org/10.1145/1102351.1102430
[25] Teng, D. (2025). TRIAD: Deadline- and reliability-aware scheduling for post-earthquake structural assessment workflows. World Journal of Engineering Research, 3(6), 55–66.
[26] Pedregosa, F., et al. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825–2830. https://www.jmlr.org/papers/v12/pedregosa11a.html
[27] Zhao, W., Chen, T., Wang, B., & Shen, Z. (2026). PIPE-RAG: An Execution-Guided Retrieval and Reuse Framework for Reliable Machine Learning Pipeline Code Generation. WEP Transactions on Computer Science and Intelligent Systems Research, 13, 286–297.
[28] Shang, W., Yang, J.-S., Ding, G., & Yang, S. (2026). ArchMigrate-Swift: A Compiler-in-the-Loop Retrieval-Augmented Multi-Agent Framework for Objective-C-to-Swift Migration. WEP Transactions on Computer Science and Intelligent Systems Research, 13, 275–285.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Academic Journal of Applied Sciences
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
