When decoding data out of a dataset encoded with the H5Z_NBIT decoding, the HDF5 1.8.16 library will fail to ensure that the precision is within the bounds of the size leading to arbitrary code execution.
HDF5 is a file format that is maintained by a non-profit organization, The HDF Group. HDF5 is designed to be used for storage and organization of large amounts of scientific data and is used to exchange data structures between applications in industries such as the GIS industry via libraries such as GDAL, OGR, or as part of software like ArcGIS.
The vulnerability exists when the library is decoding data out of a dataset encoded with the H5Z_NBIT decoding. When calculating the precision that a BCD number is encoded as, the library will fail to ensure that the precision is within the bounds of the size. Due to this, the library will calculate an index outside the bounds of the space allocated for the BCD number. Whilst decoding this data, the library will then write outside the bounds of the buffer leading to a heap-based buffer overflow. This can lead to code execution under the context of the application using the library.