“OSError: Unable to open file (bad superblock version number)" in python h5py file read

+2 votes

Short Description

I have an issue where an hdf5 file has been written on a Windows machine and is unable to be opened on a Linux machine. The error message is "OSError: Unable to open file (bad superblock version number)". (As such, this issue may not be related to h5py at all, but rather a general linux/windows compatibility issue in python file open).

Long Description

A python virtual environment with the following packages was used on both Windows and Linux:

  • Flask-0.12.2
  • Flask-RESTful-0.3.6
  • Jinja2-2.10
  • MarkupSafe-1.0
  • Werkzeug-0.14.1
  • aniso8601-3.0.0
  • click-6.7
  • h5py-2.7.1
  • h5py-cache-1.0
  • itsdangerous-0.24
  • lockfile-0.12.2
  • numpy-1.14.0
  • pytz-2018.3
  • six-1.11.0

On Windows, the file could be opened and read without issues, but on Linux it couldn't, throwing an OSError. Simply starting a new python session and typing the following is enough:

import h5py

f1 = h5py.File("myfile.hdf5", "r")

Full error:

Traceback (most recent call last):

File "stdin", line 1, in module

File "/usr/local/lib/python3.6/site-packages/h5py/_hl/files.py", line 312, in _ _ init_ _

fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)

File "/usr/local/lib/python3.6/site-packages/h5py/_hl/files.py", line 142, in make_fid

fid = h5f.open(name, flags, fapl=fapl)

File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper

File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper

File "h5py/h5f.pyx", line 78, in h5py.h5f.open

OSError: Unable to open file (bad superblock version number)

The HDF5 file is written on a Windows machine with Java HDF5 Library from a program that I cannot modify, with hdf5 java 1.10.0 in SWMR (single-write, multiple-reader) mode.

It is possible that the program doesn't close the file properly before sending it further to my program, a light-weight Linux application.

In http://web.mit.edu/fwtools_v3.1.0/www/H5.format.html, the "Version Number of the Super Block" is described as follows...

This value is used to determine the format of the information in the super block. When the format of the information in the super block is changed, the version number is incremented to the next integer and can be used to determine how the information in the super block is formatted.

Values of 0 and 1 are defined for this field.

This field is present in version 0+ of the superblock.

...which doesn't help me understand what the bad superblock version number error could be about.

Here's a sample file I'm trying to open: https://drive.google.com/open?id=10hpbWj4HBwIMq0X6Rq7yVzJATOiYHJcc

Why make a stackoverflow question out of this?

This issue might affect everyone who, on a Linux machine, want to read hdf5 files generated on a Windows machine and not correctly closed/formatted/etc. I'd like to know the reason(s) why this happens and how to work around this on my end, on Linux. If the only solution is "It needs to be fixed by the Windows program generating the HDF5 file, as this cannot be fixed afterwards", then that is an acceptable answer as well. Is that the case here?

Actions Taken

  • Upgrading to h5py to 2.8.0rc1 doesn't solve the issue
  • Other HDF5 files can be opened as expected

Related Topics

I've looked at the following topics and sites for possible reasons, but have come up empty handed:

1) https://support.hdfgroup.org/HDF5/faq/bkfwd-compat.html

2) h5py OSError: Unable to open file (File signature not found)

3) HDF5 file created with h5py can't be opened by h5py

4) https://github.com/h5py/h5py/issues/757

5) http://web.mit.edu/fwtools_v3.1.0/www/H5.format.html

Edit 1:

Thanks to @Tom de Geus, I tried HDF View on Linux and Windows and discovered that the sample file cannot be opened on Linux HDF View, but it can be opened with Windows HDF View. This suggests that the issue is in the file and HDF, not h5py.

Sep 26, 2018 in Python by bug_seeker
• 15,350 points
1,170 views

Have you tried reading with HDF View? Or any API? 

1 answer to this question.

0 votes

File could indeed be opened on Linux with the correct HDF version, 1.10.0. On Windows, it turns out I was running HDF 1.10.1 (installed through h5py), but on Linux the default installation of h5py gave me 1.8.18. After building HDF from sources with cmake, I was able to open the file on Linux as well, with version 1.10.0 or above.

Guide for installing HDF 1.10.0 with cmake:https://support.hdfgroup.org/HDF5/release/cmakebuild.html

The error message "OSError: Unable to open file (bad superblock version number)" was confusing but seems to indicate that the old version of HDF (1.8.18) wasn't configured to recognize the new superblock version number written by HDF version 1.10.0.

answered Sep 26, 2018 by Priyaj
• 56,900 points

Related Questions In Python

0 votes
1 answer

Need help to open excel file and read in Python

You can use pandas module to do ...READ MORE

answered Jul 22 in Python by Tina
29 views
0 votes
1 answer

When to use file vs open in Python?

File() has been removed since Python 3.0 ...READ MORE

answered Oct 30, 2018 in Python by SDeb
• 13,160 points
29 views
0 votes
1 answer

How to create and read from a temporary file in Python?

Hi, there is a very simple solution ...READ MORE

answered Jan 29 in Python by Nymeria
• 3,520 points
79 views
0 votes
1 answer

How to read numbers from file in Python?

Assuming you don't have extraneous whitespace: with open('file') ...READ MORE

answered Apr 16 in Python by SDeb
• 13,160 points
94 views
0 votes
1 answer
0 votes
1 answer

Getting friendly device names in python

Regarding Linux, if all you need is ...READ MORE

answered Mar 26 in Python by SDeb
• 13,160 points
25 views
0 votes
1 answer

How to read Excel File in Python

With pandas it is possible to get ...READ MORE

answered Oct 22, 2018 in Python by Priyaj
• 56,900 points
165 views
+2 votes
3 answers

How can I play an audio file in the background using Python?

down voteacceptedFor windows: you could use  winsound.SND_ASYNC to play them ...READ MORE

answered Apr 3, 2018 in Python by charlie_brown
• 7,720 points
1,908 views