SFU webinar 'Workflow hacks for large datasets in HPC'
Date: 20 May 2025 @ 17:00 - 18:00
RegisterOver the years, we have delivered webinars on tools that can significantly enhance research workflows involving large datasets. The recordings and materials for these webinars are freely available on our website.In this webinar, I will briefly highlight some of these valuable tools:- In-situ visualization enables interactive rendering of large in-memory arrays without the need to store them to disk.- Lossy 3D data compression can reduce the size of 3D datasets by up to 100X with no visible artifacts, making it ideal for storage and archival- Distributed storage of large datasets helps manage vast amounts of data across multiple locations.- DAR (Disk ARchiver) is a modern, high-performance alternative to TAR that offers indexing, differential archives, and faster extraction.I encourage you to integrate these tools into your workflow to improve data management and make better use of available computational resources.
Keywords: RDM, Research Data Management, Storage, HPC, Visualization, Git, Programming
Venue: Online
Activity log