The Research Computing Centre (RCC) and CMM are using Clowder, a customisable and scalable data management framework for structured and unstructured data developed by the US National Center for Supercomputing Applications (NCSA).
Clowder enables users to share, annotate, organise and analyse large collections of data sets.
Data from CMM’s instruments, such as microscopes, x-ray and mass spectrometry, will be ingested into the Clowder-based system, called Pitschi (Particle Imaging depoT using Storage CacHing Infrastructure), directly.
Project lead and RCC Systems Programmer Dr Hoang Nguyen said the development team evaluated a number of different platforms. “We decided to use Clowder as it can be customised to UQ storage and compute infrastructure, i.e. UQRDM, MeDiCI and UQ HPCs.
“Clowder also addresses the problem of data preservation in a world with growing amounts of digital data, much of which is not properly curated,” said Hoang.
The team has modified Clowder so it maps seamlessly with CMM’s instrument booking system, and with data storage on UQ Research Data Manager (UQRDM), and makes sharing and reusing data across campus and institutions easier.
They have also created a user-friendly image browser to navigate quickly through image and document data sets and enabled the harvesting of metadata during data ingestion.
CMM Data Informatics Manager Dr Rubbiya Ali, who worked closely with Hoang on the project, said: “We spent a lot of time working on pre-requirements, such as good network hardware, the protocols for high-speed communication between the repository and UQRDM, and then ingests of the data into the repository.
“With the help of RCC, we are now very close to achieving a complete data solution, Pitschi, for CMM instruments, which allows raw data to be streamed into the UQRDM and then into CVL@Weiner for researchers to do further 2D and 3D image processing.”
