Compression for instruments

Research topic and goals

Scientific instruments, simulation and data analytics applications are generating or using extremely large datasets that are difficult (if not impossible) to move, store and transform in full. Compression of scientific data, either lossless or lossy, is considered as one of the solutions to address this problem. Already several applications, including ECP applications, are relying on compression to reduce their data. The purpose of the collaboration is first to explore the compression needs at ANL and RCCS from instruments (in particular light sources like the APS and Spring-8), simulations and data analytics involving ML/DL. Second, the collaboration will design new compression algorithms or adapt existing ones responding to ANL and RCCS application needs. Third, the collaboration will explore advanced implementations in FPGA and GPU of the designed compression pipelines. We propose to:

Explore the compression needs at ANL and RCCS in terms of scientific instruments, simulations and data analytics, including ML/DL
Design of new compression algorithms for identified applications
Design and implementation of hardware accelerators for compression pipelines

Results for 2019/2020

One of the promising approaches is predictive delta compression. Predictive delta compression is a technique to store only difference between original data and predicted data or the difference between consecutive predicted frames. Therefore, accurate prediction to the original data, which is data to compress, is important for increasing the compression ratio. Because the image data from sensors are time-evolutional images, we need a technique to accuracy predict future image frames. Predictive neural network is a predictive coding based deep convolutional neural network which learns to predict future frames of a video sequence. PredNet is such an architecture which is trained to predict the future movement of objects. We use PredNet to predict the future frames from the given time-evolutional frames. In the evaluation, we observe that we can compress SPring-8 data by a factor of 600 compared to original size using a combination of predictive delta compression and the SZ compressor developed at Argonne with a PSNR of 56dB.

In addition, Argonne collaborated with Clemson to develop a control loop allowing the automatic setting of lossy compressor error bound from user defined analysis metrics. The first implementation of the control loop targeted the fix ratio compression objective. The control loop was used to search for the compressor error bound providing the best accuracy for the targeted compression ratio. A paper has been accepted at IPDPS on this topic.

Results for 2020/2021

The research on predictive delta compression continued at Riken and produced a paper accepted at IEEE CCGRID 2021. In parallel, the team at Argonne optimized the implementation of Huffman variable lenght coding on GPUs. This effort produced two publications at ACM PACT 2020 and IEEE IPDPS 2021.

Results for 2021/2022

The main result is the validation from the EXAFEL team of RiobinSZ as a valid lossy compressor for crystallography. The validation experiment used RoibinSZ on Se-SAD SFX Dataset (Selenium Single-wavelength Anomalous Diffraction. Serial Femtosecond Crystallography), 5043 images (1480x1552x4bytes). Many metrics were used to perform the assesment: Number of hits: Number indexed, Rsplit, CC1/2, CCano, Rwork, Rfree, Map-model CC. Roibin SZ electron density successfully reconstructs the protein structure. The dataset compressed with RiobinSZ offered a better result overall compared to the non-compressed dataset. The current explanation is that RiobinSZ not only compress the data but also performs some denoising that results in improving the analysis quality. A paper is submited to IEEE Bigdata on using SZ3 for the Argonne Photon Source data.

The second main results concerns the compression using SZ3 of X-ray ptychographic data acquired at the Advanced Photon Source (APS). A Dectris Eiger detector (514x1030 pixels) was used to acquire diffraction patterns as X-ray beam scanned across the sample, and the 2D diffraction images were saved along the time dimension to form a 3D matrix array (19500 x 514 x 1030 for chip pillar and 16800 x 514 x 1030 for flat chip). When the decompression data is near lossless (i.e., error bound less than 0:5), the compression ratio gain of the proposed compression pipeline is 18% on chip pillar and 12% on flat chip compared with the second-best one.

The third main result is the phychography of a Catalyst Particle Compressed with SZ2.1 (not RiobinSZ). The dataset is a single scan (diffraction patterns): 1856x1030x514. In practice we compressed 1856 images of 514x1030 pixels. For the spatial compression, the dataset is treated as a 3D dataset, so the SZ2.1 predictor adopts a 3D Lorenzo + 3D Linear regression; For the temporal compression, the compressor predicts each data point only based on the temporal dimension. We tested absolute error bound from 2 to 64. Absolute error bound of 2 translates to (+/-) 2photon count error on the detector. The test consisted in a ptychographic reconstruction on (sz) decompressed diffraction patterns of a Catalyst Particle. The reconstruction used the Conjugate Gradient iterative method that took 300 steps. For spatial compression with an error bound of 4, the compression ratio reached 97 with a SSIM >0.96. For temporal compression with an error bound of 8, the compression ratio reaches: 245 for a SSIM >0.94. The SSIM results validates the use of SZ2.1 for phychrography at the APS.

The research at Argonne also optimized the compression method for crystallography data resulting in ROIBIN-SZ3. This optimized compression strategy resulted in improvements to compression ratios as well as compression and decompression bandwidth subject to the user’s quality constraints with publication forthcoming. Further improvements may be possible by lowering certain aspects of the ROIBIN-SZ3 process to the GPU pending further validation of correctness and effectiveness of the CPU implementation.

Riken released the compression code developed for SPRING-8: https://github.com/kento/TEZip

Results for 2023/2024

Robert Underwood (Argonne), Kento Sato (Riken), and Amarjit Singh(Riken) supervised a summer student at Riken, Isita, to integrate LibPressio with TEZip. This integration allows for more exhaustive comparisons between TEZip and other compressor such as SZ3 and ROIBIN-SZ. There were several technical challenges to overcome for this integration: 1) LibPressio and the other compressors are implemented in C++, but TEZip is written in python, 2) to make the comparisons possible and efficient, extensions were need to use shared memory to avoid the use of the file system for these large files, 3) TEZip makes radically different assumptions about input formats that needed to be overcome. These preliminary results show that TEZip may have an advantage at extremely high compression ratios, but at the cost of low bandwidth while compressors developed at Argonne have only high compression ratios but at much higher speed. A poster on these efforts was presented by Isita to SC23. Robert Underwood will visit Riken in April 2024 to help complete and submit these results for publication.

At Argonne, forward progress was also made on RoIBIN-SZ. RoIBIN-SZ was successfully deployed and used at scale in the S3DM data reduction pipeline cluster at SLAC National Accelerator Center for their SFX workflow demonstrating the viability of this approach in production workflows. Additionally, there were efforts to port portions of the RoIBIN-SZ pipeline to the GPU – these efforts show a end-to-end improvement over the CPU with only a potion of the current pipeline on the GPU indicating that even with transfers to/from the GPU it is possible to accelerate this workflow on the GPU. In the next year, we hope to complete porting of ROIBIN-SZ to the GPU and evaluate its improved performance which should remove the needs to transfer to/from the GPU dramatically improving performance. An overview of RoIBIN-SZ was included as invited paper to Synchrotron Radiation News.

Visits and meetings

2020:

Franck Cappello visited Riken-CCS and Riken Spring-8 facility in August 2019 for 1 week
Bogdan Nicolae visited Riken-CCS and Riken Spring-8 facility in August 2019 for 1 week
Ali Murat Gok visited Riken-CCS and Riken Spring-8 facility in August 2019 for 1 week
Kazumoto Yoshii visited Riken-CCS and Riken Spring-8 facility in August 2019 for 1 week

2021: There was no visit due to the Covid situaiton.

2022: There was no visit due to the COVID situation.

2023: There was no visit in 2023.

2024:

Robert Underwood will visit Riken in April advance of the JLESC workshop for 1 week.

Impact and publications

The first presentation of this collaboration was made during the DOE-MEXT meeting at Chicago in May 2019. Franck Cappello presentation of lossy compression for photon source at the International Forum on Detectors for Photon Science March-April 2020 [cancelled because of COVID-19].

Papers

A paper at Synchrotron Radiation News providing an overview RoIBIN-SZ (“ROIBIN-SZ: Fast and Science-Preserving Compression for Serial Crystallography” 2023)
A poster at SC23 discussing preliminary evaluation of TEZip against other leading lossy compressors (“Integrating TEZip into LibPressio: A Acase Study of Integrating a Dynamic Application into a Static C Envionment” 2023)
A paper at IPDPS on fix ratio compression using control loop (Underwood et al. 2020)
A poster presented the results of the delta compression developed at Riken combined with the SZ compressor developed at Argonne. (Roy et al. 2019)
A paper at ACM PACT 2020 (Tian et al. 2020)
A paper at IEEE CCGRID 2021 (Roy et al. 2020)
A paper at IEEE IPDPS 2021 (Tian et al. 2021)
A paper at SC/DRBSD 2021 workshop (Underwood et al. 2021)

“ROIBIN-SZ: Fast and Science-Preserving Compression for Serial Crystallography.” 2023. Synchrotron Radiation News 36 (4): 17–22. https://doi.org/10.1080/08940886.2023.2245722.

@article{UnderwoodEtAl2023,
  author = {},
  title = {ROIBIN-SZ: Fast and Science-Preserving Compression for Serial Crystallography},
  journal = {Synchrotron Radiation News},
  volume = {36},
  number = {4},
  pages = {17-22},
  year = {2023},
  publisher = {Taylor & Francis},
  doi = {10.1080/08940886.2023.2245722},
  url = {https://doi.org/10.1080/08940886.2023.2245722},
  eprint = {https://doi.org/10.1080/08940886.2023.2245722}
}

“Integrating TEZip into LibPressio: A Acase Study of Integrating a Dynamic Application into a Static C Envionment.” 2023.

@misc{TalukdarEtAl2023,
  author = {},
  title = {Integrating TEZip into LibPressio: A Acase Study of Integrating a Dynamic Application into a Static C Envionment},
  year = {2023}
}

Underwood, Robert, Victoriana Malvoso, Jon C. Calhoun, Sheng Di, and Franck Cappello. 2021. “Productive and Performant Generic Lossy Data Compression with LibPressio.” In 2021 7th International Workshop on Data Analysis and Reduction for Big Scientific Data (DRBSD-7), 1–10. https://doi.org/10.1109/DRBSD754563.2021.00005.

@inproceedings{libpressio,
  author = {Underwood, Robert and Malvoso, Victoriana and Calhoun, Jon C. and Di, Sheng and Cappello, Franck},
  booktitle = {2021 7th International Workshop on Data Analysis and Reduction for Big Scientific Data (DRBSD-7)},
  title = {Productive and Performant Generic Lossy Data Compression with LibPressio},
  year = {2021},
  volume = {},
  number = {},
  pages = {1-10},
  doi = {10.1109/DRBSD754563.2021.00005}
}

Tian, Jiannan, Cody Rivera, Sheng Di, Jieyang Chen, Liang Xin, Dingwen Tao, and Franck Cappello. 2021. “Revisiting Huffman Coding: Toward Extreme Performance on Modern GPU Architectures.” In IPDPS’21: 35th IEEE International Parallel and Distributed Processing Symposium.

@inproceedings{Tian21,
  title = {{Revisiting Huffman Coding: Toward Extreme Performance on Modern GPU Architectures}},
  author = {Tian, Jiannan and Rivera, Cody and Di, Sheng and Chen, Jieyang and Xin, Liang and Tao, Dingwen and Cappello, Franck},
  booktitle = {{IPDPS'21: 35th IEEE International parallel and distributed processing symposium}},
  year = {2021}
}

Tian, Jiannan, Sheng Di, Kai Zhao, Rivera Cody, Megan Hickman, Robert Underwood, Sian Jin, et al. 2020. “CuSZ: An Efficient GPU Based Error-Bounded Lossy Compression, Framework for Scientific Data.” In PACT’20: ACM International Conference on Parallel Architectures and Compilation Techniques.

@inproceedings{Tian20,
  title = {{cuSZ: An Efficient GPU Based Error-Bounded Lossy Compression, Framework for Scientific Data}},
  author = {Tian, Jiannan and Di, Sheng and Zhao, Kai and Cody, Rivera and Hickman, Megan and Underwood, Robert and Jin, Sian and Liang, Xin and Calhoun, Jon and Tao, Dingwen and Cappello, Franck},
  booktitle = {{PACT'20: ACM International Conference on Parallel Architectures and Compilation Techniques}},
  year = {2020}
}

Roy, Rupak, Kento Sato, Subhadeep Bhattacharya, Xingang Fang, Yasumasa Joti, Takaki Hatsui, Toshiyuki Hiraki, Jian Guo, and Weikuan Yu. 2020. “Compression of Time Evolutionary Image Data through Predictive Deep Neural Networks.” In CCGRID’21: IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing.

@inproceedings{Rupak21,
  title = {{Compression of Time Evolutionary Image Data through Predictive Deep Neural Networks}},
  author = {Roy, Rupak and Sato, Kento and Bhattacharya, Subhadeep and Fang, Xingang and Joti, Yasumasa and Hatsui, Takaki and Hiraki, Toshiyuki and Guo, Jian and Yu, Weikuan},
  booktitle = {{CCGRID'21: IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing}},
  year = {2020}
}

Underwood, Robert, Sheng Di, Jon Calhoun, and Franck Cappello. 2020. “FRaZ: A Generic High-Fidelity Fixed-Ratio Lossy Compression Framework for Scientific Floating-Point Data.” In IPDPS20: 34th IEEE International Parallel and Distributed Processing Symposium.

@inproceedings{Underwood20,
  title = {{FRaZ: A Generic High-Fidelity Fixed-Ratio Lossy Compression Framework for Scientific Floating-point Data}},
  author = {Underwood, Robert and Di, Sheng and Calhoun, Jon and Cappello, Franck},
  booktitle = {{IPDPS20: 34th IEEE International parallel and distributed processing symposium}},
  year = {2020}
}

Roy, Rupak, Kento Sato, Jian Guo, Jens Domke, Weikuan Yu, Takaki Hatsui, and Yasumasa Joti. 2019. “Poster: Improving Data Compression with Deep Predictive Neural Network for Time Evolutional Data.” In SC’2019, the IEEE/ACM Conference on High Performance Computing Networking, Storage and Analysis. ACM Press.

@inproceedings{Rupak20,
  author = {Roy, Rupak and Sato, Kento and Guo, Jian and Domke, Jens and Yu, Weikuan and Hatsui, Takaki and Joti, Yasumasa},
  booktitle = {{SC'2019}, the IEEE/ACM Conference on High Performance Computing Networking, Storage and Analysis},
  title = {Poster: Improving Data Compression with Deep Predictive Neural Network for Time Evolutional Data},
  publisher = {ACM Press},
  year = {2019}
}

Funding

Robert Underwood from Clemson (Jon Calhoun’s group) received a DOE funding for his 6 months stay at Argonne.

Impact on other projects

The compression scheme (ROI-SZ) developed by Argonne is in extensive testing at LCLS for integration in the data reduction pipeline of the LCLS2. It will be also tested in Germany. This project has a direct impact on the APS (Argonne Photon Source) and Spring-8 instruments. This project has a broad impact on other photo sources. Results from this project are impacting the collaborations with the LCLS (Linac Coherent Light Source) instruments.

The implementation of Huffman variable length coding in SZ (part of ROI-S) is the first high performance implementation of Huffman coding on GPU. We will make it stand alone and available for the community, independently of SZ.

Future plans

Exchange results of experiments and testing of the compression schemes.

References

Underwood, Robert, Jon Calhoun, Sheng Di, and Franck Cappello. 2020. “FRaZ: A Generic High-Fidelity Fixed-Ratio Lossy Compression Framework For Scientific Data.” In 2020 IEEE International Parallel and Distributed Processing Symposium, IPDPS 2020, New Orleans, USA, May 18-22, 2020. IEEE.

@inproceedings{UnderwoodEtAl2020,
  author = {Underwood, Robert and Calhoun, Jon and Di, Sheng and Cappello, Franck},
  title = {FRaZ: A Generic High-Fidelity Fixed-Ratio Lossy Compression Framework for
        Scientific Data},
  booktitle = {2020 {IEEE} International Parallel and Distributed Processing Symposium,
                 {IPDPS} 2020, New Orleans, USA, May 18-22, 2020},
  pages = {},
  publisher = {{IEEE}},
  year = {2020},
  url = {}
}

Reza, Tasmia, Jon Calhoun, Kristopher Keipert, Sheng Di, and Franck Cappello. 2019. “ Analyzing the Performance and Accuracy of Lossy Checkpointing on Sub-Iteration of NWChem.” In 2019 IEEE/ACM 5th International Workshop on Data Analysis and Reduction for Big Scientific Data (DRBSD).

@inproceedings{RezaEtAl2019,
  author = {Reza, Tasmia and Calhoun, Jon and Keipert, Kristopher and Di, Sheng and Cappello, Franck},
  title = { Analyzing the Performance and Accuracy of Lossy Checkpointing on Sub-Iteration 
        of NWChem},
  booktitle = {2019 IEEE/ACM 5th International Workshop on Data Analysis and Reduction for Big 
        Scientific Data (DRBSD)},
  year = {2019},
  volume = {},
  number = {},
  pages = {},
  doi = {},
  issn = {},
  month = nov
}

Sasaki, N., K. Sato, T. Endo, and S. Matsuoka. 2015. “Exploration of Lossy Compression for Application-Level Checkpoint/Restart.” In Parallel and Distributed Processing Symposium (IPDPS), 2015 IEEE International, 914–22.

@inproceedings{SasakiETAl2015,
  author = {Sasaki, N. and Sato, K. and Endo, T. and Matsuoka, S.},
  booktitle = {Parallel and Distributed Processing Symposium (IPDPS), 2015 IEEE International},
  title = {Exploration of Lossy Compression for Application-Level Checkpoint/Restart},
  year = {2015},
  pages = {914-922},
  month = may
}

Ni, Xiang, Tanzima Islam, Kathryn Mohror, Adam Moody, and Laxmikant V Kale. 2014. “Lossy Compression for Checkpointing: Fallible or Feasible?” In International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

@inproceedings{NiETAl2014,
  title = {Lossy compression for checkpointing: Fallible or feasible?},
  author = {Ni, Xiang and Islam, Tanzima and Mohror, Kathryn and Moody, Adam and Kale, Laxmikant V},
  booktitle = {International Conference for High Performance Computing, Networking, Storage and Analysis (SC)},
  year = {2014}
}

Ibtesham, Dewan, Dorian Arnold, Kurt B. Ferreira, and Patrick G. Bridges. 2012. “On the Viability of Checkpoint Compression for Extreme Scale Fault Tolerance.” In Proceedings of the 2011 International Conference on Parallel Processing - Volume 2, 302–11. Euro-Par’11. Berlin, Heidelberg: Springer-Verlag. https://doi.org/10.1007/978-3-642-29740-3_34.

@inproceedings{IbteshamETAl2012,
  author = {Ibtesham, Dewan and Arnold, Dorian and Ferreira, Kurt B. and Bridges, Patrick G.},
  title = {On the Viability of Checkpoint Compression for Extreme Scale Fault Tolerance},
  booktitle = {Proceedings of the 2011 International Conference on Parallel Processing - Volume 2},
  series = {Euro-Par'11},
  year = {2012},
  isbn = {978-3-642-29739-7},
  location = {Bordeaux, France},
  pages = {302--311},
  numpages = {10},
  url = {http://dx.doi.org/10.1007/978-3-642-29740-3_34},
  doi = {10.1007/978-3-642-29740-3_34},
  acmid = {2238474},
  publisher = {Springer-Verlag},
  address = {Berlin, Heidelberg},
  keywords = {checkpoint data compression, checkpoint/restart, extreme scale fault-tolerance}
}

Lindstrom, P., and M. Isenburg. 2006. “Fast and Efficient Compression of Floating-Point Data.” IEEE Transactions on Visualization and Computer Graphics 12 (5): 1245–50.

@article{LindstromETAl2006,
  author = {Lindstrom, P. and Isenburg, M.},
  journal = {IEEE Transactions on Visualization and Computer Graphics},
  title = {Fast and Efficient Compression of Floating-Point Data},
  year = {2006},
  volume = {12},
  number = {5},
  pages = {1245-1250},
  month = sep
}