Benchmarking Big Data and Artificial Intelligence – Evaluation schemes for Big data and AI Performance of high Business impact

For many years companies have been running technical benchmarks to compare the performance of different technologies and systems. This benchmarking process provides very valuable information but in many cases, it is not correlated with the impact of the solutions at business level. This was precisely the starting point of the DataBench project, funded by the EC under the Big Data Value PPP. After three years of intensive work DataBench has created an evaluation framework that is already openly available for the community. The session will introduce the different elements of the framework, including the toolbox, the way it can be used and some concrete examples run with companies and projects that work heavily with data.

• The current landscape of Big Data and AI benchmarks
• The DataBench framework, which includes a complete set of metrics for the assessment of Big Data technologies
• The DataBench toolbox, a web-based tool that provides a unique environment to search, select and deploy big data benchmarking tools, giving the possibility to generate unified technical metrics and derive business KPIs
• A comprehensive set of use cases that we have run with companies in different industrial domains and projects of the Big Data PPP to illustrate the way you can get value out of using DataBench
• Pipelines and blueprints

The DataBench toolbox follows a platform model and provides a single place where benchmarking communities and users of such benchmarks can meet. It goes beyond any work done so far by translating major technical KPIs into business KPIs, allowing us to establish relationships between technical and business decisions. As such, the session targets a multiplicity of stakeholders: benchmark providers, companies interested in benchmarking big data, projects and members of the big data and AI communities, decision-makers of companies that are thinking about their data-driven transformation and policy makers looking for evidence-based decision-support tools.

On our side, we will bring a comprehensive set of speakers falling precisely under all those categories.

The workshop will be structured around two parts (each of them can be attended independently; like this we want to facilitate that those participants that are not available for the entire workshop, select the most interesting part for them). Attending the complete workshop is in any case recommended to take maximum advantage of the contents.

Detailed Agenda

10:00-10:05 Intro. Objectives of the session (Nuria de Lama (Atos)

10:05-10:15 DataBench General Overview (Richard Stevens (IDC, DataBench coordinator)

PART I. Big Data Benchmarking landscape and Big Data Pipelines

• 10:15-11:15 Session 1. The current landscape of Big Data benchmarks

10:15-10:25: DataBench Framework for Benchmarks, Arne J. Berre, SINTEF

10:25-10:40: Hobbit, Benchmarking Platforms and AI, Axel Ngonga, BDVA TF6 Benchmark Lead, University of Paderborn

10:40-10:55: BenchCouncil Big Data and AI Benchmarks, Wanling Gao, Chinese Academy of Sciences

10:55-11:10: MLPerf AI and ABench, Rekha Singhal, Senior Scientist and Head of the Computing Systems-Software Research area at TCS

11:10-11:15: Conclusion on Big Data and AI Benchmarks, Todor Ivanov, LeadConsult

• 11:15-12:15 Session 2. A Project perspective on Big Data and AI architectural pipelines and benchmarks

11:15-11:20: Introduction to Architectural pipelines, Arne J. Berre, SINTEF

11:20-11:30: I- BiDaaS – Leonidas Kallipolitis, AEGIS

11:30-11:40: TBFY – Brian Elvesæter, SINTEF

11:40-11:50: Track&KNow – Athanasios Koumparos, Vodafone

11:50-12:00: DataBio – Caj Södergård, VTT

12:00-12:10: DeepHealth – Jon Ander Gómez Adrián, Universitat Politecnica de Valencia

12:10-12:15: Conclusion on Pipelines and related benchmarks, Arne J. Berre, SINTEF

12:15-12:30 Short coffee break to relax and maybe grab a coffee

PART II. Big Data Business Framework and benchmarking tooling support

• 12:30-13:00 Session 3. The DataBench Business framework: a compelling offering to measure the impact of Big Data Technologies

12.30-12.40: The DataBench business framework, by Gabriella Cattaneo, Erica Spinoni and Chiara Francalanci

12: 40-12.50: The Whirlpool use case, Pierluigi Petrali (Whirlpool)

12.50-13.00:  The Fill use case, Harald Sehrschön (FILL)

• 13:00-13:30 Session 4. A practical journey on how to use the DataBench Toolbox

13:00-13:15 Demo of the DataBench Toolbox (Tomás Pariente, Atos)

13:15-13:25 AI Observatory (Marko Grobelnik, JSI)

13:25-13:30 Fostering adoption of DataBench results. Needs from the point of view of Digital Innovation Hubs (Daniel Sáez, ITI, EuHubs4Data coordinator)

• • 13:30 Concluding Remarks and closing of the session