ExCALIBUR (Exascale Computing Algorithms & Infrastructures Benefiting UK Research) is a UK research programme that aims to deliver the next generation of high-performance simulation software for the highest-priority fields in UK research. It started in October 2019 and will run through until March 2025, redesigning high priority computer codes and algorithms to meet the demands of both advancing technology and UK research.
Project Activities
RCS is actively working on ExCALIBUR’s ExCALIData, a project which has a number of milestones and deliverables anticipated in the next three years.
Redesigning codes and workflows to take advantage of exascale simulation requires addressing I/O and workflow problems which arise throughout a sequence of activities that may begin and end with deep storage and will involve both simulation and analysis phases.
Many simulations begin by exploiting data which need to be extracted from deep storage. Such data are needed to provide ancillary data, initial conditions, and boundary conditions for simulations. In weather and climate use case some such data may need to be extracted from within large datasets, and even from within ‘files’ inside such datasets. There may be some processing required to do the extraction, possibly including decompression, subsetting, regridding (‘re-meshing’) and reformatting. Depending on volumes and available tools, such processing may be best handled adjacent (and/or within) the storage system itself or handled during the simulation workflow itself.
We address storage interfaces at the application level and tools for managing information about what is where across tiers and between institutions; we examine the possibilities for accelerating I/O using both network fabric and advanced burst buffers; we compare and contrast examples of generic and domain specific I/O-middleware.
This work recognises the difficulties that users have in managing the location of their data across tiered storage, and of configuring their workflows to take advantage of newly available acceleration options which otherwise require rather arcane knowledge. Recognising that there are different problems in the analysis and simulation phases, we target both the analysis and simulation phase. We specifically look at the impact of the various technologies on synthetic tests, and on real use-cases from both Weather and Climate, and Fusion (including techniques for AI/ML and other analysis on the huge datasets expected).
