Towards Better Data Discovery and Collection with Flow-Based Programming

Andrei Paleyes; Christian Cabrera; Neil D. Lawrence

Back to publications

Towards Better Data Discovery and Collection with Flow-Based Programming

Andrei Paleyes, Christian Cabrera, Neil D. Lawrence

Neurips Data-Centric AI Workshop (DCAI), 2026.

Abstract

Despite huge successes reported by the field of machine learning, such as voice assistants or self-driving cars, businesses still observe very high failure rate when it comes to deployment of ML in production. We argue that part of the reason is infrastructure that was not designed for data-oriented activities. This paper explores the potential of flow-based programming (FBP) for simplifying data discovery and collection in software systems. We compare FBP with the currently prevalent service-oriented paradigm to assess characteristics of each paradigm in the context of ML deployment. We develop a data processing application, formulate a subsequent ML deployment task, and measure the impact of the task implementation within both programming paradigms. Our main conclusion is that FBP shows great potential for providing data-centric infrastructural benefits for deployment of ML. Additionally, we provide an insight into the current trend that prioritizes model development over data quality management.

Links

Cite this Paper

BibTeX


@InProceedings{publications/towards-better-data-discovery-and-collection-with-flow-based-programming,
  title = 	 {Towards Better Data Discovery and Collection with Flow-Based Programming},
  author = 	 {Paleyes, Andrei and Cabrera, Christian and Lawrence, Neil D.},
  booktitle = 	 {Neurips Data-Centric AI Workshop (DCAI)},
  year = 	 {2026},
  url = 	 {/publications/towards-better-data-discovery-and-collection-with-flow-based-programming.html},
  abstract = 	 {Despite huge successes reported by the field of machine learning, such as voice assistants or self-driving cars, businesses still observe very high failure rate when it comes to deployment of ML in production. We argue that part of the reason is infrastructure that was not designed for data-oriented activities. This paper explores the potential of flow-based programming (FBP) for simplifying data discovery and collection in software systems. We compare FBP with the currently prevalent service-oriented paradigm to assess characteristics of each paradigm in the context of ML deployment. We develop a data processing application, formulate a subsequent ML deployment task, and measure the impact of the task implementation within both programming paradigms. Our main conclusion is that FBP shows great potential for providing data-centric infrastructural benefits for deployment of ML. Additionally, we provide an insight into the current trend that prioritizes model development over data quality management.}
}

Endnote

%0 Conference Paper
%T Towards Better Data Discovery and Collection with Flow-Based Programming
%A Andrei Paleyes
%A Christian Cabrera
%A Neil D. Lawrence
%B Neurips Data-Centric AI Workshop (DCAI)
%D 2026	
%F publications/towards-better-data-discovery-and-collection-with-flow-based-programming
%U /publications/towards-better-data-discovery-and-collection-with-flow-based-programming.html
%X Despite huge successes reported by the field of machine learning, such as voice assistants or self-driving cars, businesses still observe very high failure rate when it comes to deployment of ML in production. We argue that part of the reason is infrastructure that was not designed for data-oriented activities. This paper explores the potential of flow-based programming (FBP) for simplifying data discovery and collection in software systems. We compare FBP with the currently prevalent service-oriented paradigm to assess characteristics of each paradigm in the context of ML deployment. We develop a data processing application, formulate a subsequent ML deployment task, and measure the impact of the task implementation within both programming paradigms. Our main conclusion is that FBP shows great potential for providing data-centric infrastructural benefits for deployment of ML. Additionally, we provide an insight into the current trend that prioritizes model development over data quality management.

RIS


TY  - CPAPER
TI  - Towards Better Data Discovery and Collection with Flow-Based Programming
AU  - Andrei Paleyes
AU  - Christian Cabrera
AU  - Neil D. Lawrence
BT  - Neurips Data-Centric AI Workshop (DCAI)
DA  - 2026/09/27	
ID  - publications/towards-better-data-discovery-and-collection-with-flow-based-programming
UR  - /publications/towards-better-data-discovery-and-collection-with-flow-based-programming.html
AB  - Despite huge successes reported by the field of machine learning, such as voice assistants or self-driving cars, businesses still observe very high failure rate when it comes to deployment of ML in production. We argue that part of the reason is infrastructure that was not designed for data-oriented activities. This paper explores the potential of flow-based programming (FBP) for simplifying data discovery and collection in software systems. We compare FBP with the currently prevalent service-oriented paradigm to assess characteristics of each paradigm in the context of ML deployment. We develop a data processing application, formulate a subsequent ML deployment task, and measure the impact of the task implementation within both programming paradigms. Our main conclusion is that FBP shows great potential for providing data-centric infrastructural benefits for deployment of ML. Additionally, we provide an insight into the current trend that prioritizes model development over data quality management.
ER  -

APA


Paleyes, A., Cabrera, C. & Lawrence, N.D.. (2026). Towards Better Data Discovery and Collection with Flow-Based Programming. Neurips Data-Centric AI Workshop (DCAI) Available from /publications/towards-better-data-discovery-and-collection-with-flow-based-programming.html.

Related People

Andrei Paleyes

Visiting Researcher, Cambridge University

View Profile

Christian Cabrera Jojoa

Assistant Research Professor, Cambridge University

View Profile

Neil D. Lawrence

The DeepMind Professor of Machine Learning, Cambridge University

View Profile

Related Projects

AutoAI

While excitement about the potential of artificial intelligence (AI) technologies continues to build, a gap is emerging between our aspirations for the benefits of AI and our ability to deploy these technologies to tackle real-world challenges.

Data-Oriented Architectures for AI-based Systems

Data Oriented Architecture (DOA) is a software architecture pattern that creates data-driven, loosely coupled, decentralised, and open systems. DOA achieves these goals by exposing systems' data as first class citizen to distributed, stateless, and asynchronous systems’ components. These design decisions enable DOA-based systems to achieve desirable properties such as data availability, reusability, and monitoring, as well as systems adaptability, scalability, and autonomy.

Evaluation of dataflow for ML deployment

When deploying machine learning algorithms in real-world systems, software developers face a new set of challenges. This project asks: "What are the underlying reasons for these challenges to arise?", and proposes that the problem lies with the modern trend of building software systems as microservices.