Skip to content

Big Data Test Infrastructure#

Introduction#

The Big Data Test Infrastructure (BDTI) was created by the European Commission in 2019 and it is part of the Digital Europe Programme (DEP), which, with a planned overall budget of €7.5 billion, aims to accelerate the economic recovery and shape the digital transformation of Europe’s society and economy, increasing the easy availability, quality and usability of public sector information in compliance with the requirement of the Open Data Directive.

The DEP is one of the many activities that support the implementation of the European Data Strategy, adopted by the European Commission in 2020, which aims at creating a single market for data, ensuring that more data becomes available for use in the economy and society in Europe, keeping the companies and individuals who generate the data in control of them.

In response to the public consultation on the European Strategy for Data, the European Commission published the Data Governance Act in 2020, which is intended to foster a framework to facilitate data sharing across the EU and between sectors. During February 2022, the commission presented a new Data Act, that complement the Data Governance one, regulating who can access, use and under what conditions data generated in the EU in all economic sectors.

In this context, the concrete and immediate target of BDTI is making available some tools and an environment for public administrations to experiment on their data, in correspondence with the concept of data spaces, which contributes to the vision for having an easy way of sharing data.

BDTI provides a free of charge cloud-based analytics test environment for Public Administrations in the European Member States to experiment with open-source tools and to prototype solutions before deploying them in the production environment on their own premises. Any Public Administration at any level can request to use BDTI to perform analysis and experiments on their public sector information, for a period of six months, completely for free. The test environment provided by the BDTI consists of several open-source solutions, data sources and the required cloud infrastructure that includes the virtual machines, analytics clusters, storage facilities and networking facilities. BDTI offers the opportunity of starting to get insights from PAs public information, understanding how they can move towards data-driven decision making.

Drop us a line to learn more or request a BDTI pilot: EC-BDTI-PILOTS@ec.europa.eu

BDTI as a Service#

BDTI is a Platform-as-a-Service (PaaS), hosted in the cloud, that offers the necessary managed infrastructure and software frameworks for statistical analysis to data engineers, data scientist, and data analysts for a variety of use cases. The platform enables users to select from different components a deployment suited as a solution for their use case. Standard deployments are readily available, but BDTI allows combining components for a custom solution. Components range from scalable single and clustered virtual machines to databases, open-source frameworks and commercial tools for statistical analysis. All components that BDTI provides are hosted in the cloud. BDTI offers the provisioning of the components as a service and in addition to that, BDTI also manages security and user management controls. Furthermore, support and maintenance is done by BDTI throughout the run of a BDTI deployment. After receiving a provisioned deployment, users can manage their own applications and data in the platform within security and user management controls set by BDTI.

See below the BDTI service offering:

alt-text

BDTI is a platform that helps users to focus on the data science and less on the underlying infrastructure.

BDTI Architectural Component Categories#

The components are grouped under the following functional categories:

  • Databases
  • Data Lake
  • Development Environments
  • Advanced Processing
  • Visualization
  • Orchestration

Architectural Components#

The components of BDTI are hosted in the cloud and range from scalable single and clustered virtual machines to databases, open source frameworks and commercial tools for statistical analysis. Next to offering the provisioning of the components of BDTI as a service, BDTI also manages the security and user management controls, offers support to its users, and ensures the evaluative maintenance of the platform and its components. After receiving a provisioned deployment, the users are able to manage their own applications and data in the platform within security and user management controls set by BDTI.

Resources#

Please find below some interesting links providing more information on BDTI and DEP: