“OSError: Unable to open file (bad superblock version number)" in python h5py file read

+2 votes

Short Description

I have an issue where an hdf5 file has been written on a Windows machine and is unable to be opened on a Linux machine. The error message is "OSError: Unable to open file (bad superblock version number)". (As such, this issue may not be related to h5py at all, but rather a general linux/windows compatibility issue in python file open).

Long Description

A python virtual environment with the following packages was used on both Windows and Linux:

  • Flask-0.12.2
  • Flask-RESTful-0.3.6
  • Jinja2-2.10
  • MarkupSafe-1.0
  • Werkzeug-0.14.1
  • aniso8601-3.0.0
  • click-6.7
  • h5py-2.7.1
  • h5py-cache-1.0
  • itsdangerous-0.24
  • lockfile-0.12.2
  • numpy-1.14.0
  • pytz-2018.3
  • six-1.11.0

On Windows, the file could be opened and read without issues, but on Linux it couldn't, throwing an OSError. Simply starting a new python session and typing the following is enough:

import h5py

f1 = h5py.File("myfile.hdf5", "r")

Full error:

Traceback (most recent call last):

File "stdin", line 1, in module

File "/usr/local/lib/python3.6/site-packages/h5py/_hl/files.py", line 312, in _ _ init_ _

fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)

File "/usr/local/lib/python3.6/site-packages/h5py/_hl/files.py", line 142, in make_fid

fid = h5f.open(name, flags, fapl=fapl)

File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper

File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper

File "h5py/h5f.pyx", line 78, in h5py.h5f.open

OSError: Unable to open file (bad superblock version number)

The HDF5 file is written on a Windows machine with Java HDF5 Library from a program that I cannot modify, with hdf5 java 1.10.0 in SWMR (single-write, multiple-reader) mode.

It is possible that the program doesn't close the file properly before sending it further to my program, a light-weight Linux application.

In http://web.mit.edu/fwtools_v3.1.0/www/H5.format.html, the "Version Number of the Super Block" is described as follows...

This value is used to determine the format of the information in the super block. When the format of the information in the super block is changed, the version number is incremented to the next integer and can be used to determine how the information in the super block is formatted.

Values of 0 and 1 are defined for this field.

This field is present in version 0+ of the superblock.

...which doesn't help me understand what the bad superblock version number error could be about.

Here's a sample file I'm trying to open: https://drive.google.com/open?id=10hpbWj4HBwIMq0X6Rq7yVzJATOiYHJcc

Why make a stackoverflow question out of this?

This issue might affect everyone who, on a Linux machine, want to read hdf5 files generated on a Windows machine and not correctly closed/formatted/etc. I'd like to know the reason(s) why this happens and how to work around this on my end, on Linux. If the only solution is "It needs to be fixed by the Windows program generating the HDF5 file, as this cannot be fixed afterwards", then that is an acceptable answer as well. Is that the case here?

Actions Taken

  • Upgrading to h5py to 2.8.0rc1 doesn't solve the issue
  • Other HDF5 files can be opened as expected

Related Topics

I've looked at the following topics and sites for possible reasons, but have come up empty handed:

1) https://support.hdfgroup.org/HDF5/faq/bkfwd-compat.html

2) h5py OSError: Unable to open file (File signature not found)

3) HDF5 file created with h5py can't be opened by h5py

4) https://github.com/h5py/h5py/issues/757

5) http://web.mit.edu/fwtools_v3.1.0/www/H5.format.html

Edit 1:

Thanks to @Tom de Geus, I tried HDF View on Linux and Windows and discovered that the sample file cannot be opened on Linux HDF View, but it can be opened with Windows HDF View. This suggests that the issue is in the file and HDF, not h5py.

Sep 26, 2018 in Python by bug_seeker
• 14,960 points
536 views

Have you tried reading with HDF View? Or any API? 

1 answer to this question.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
0 votes

File could indeed be opened on Linux with the correct HDF version, 1.10.0. On Windows, it turns out I was running HDF 1.10.1 (installed through h5py), but on Linux the default installation of h5py gave me 1.8.18. After building HDF from sources with cmake, I was able to open the file on Linux as well, with version 1.10.0 or above.

Guide for installing HDF 1.10.0 with cmake:https://support.hdfgroup.org/HDF5/release/cmakebuild.html

The error message "OSError: Unable to open file (bad superblock version number)" was confusing but seems to indicate that the old version of HDF (1.8.18) wasn't configured to recognize the new superblock version number written by HDF version 1.10.0.

answered Sep 26, 2018 by Priyaj
• 56,100 points

Related Questions In Python

0 votes
1 answer

When to use file vs open in Python?

File() has been removed since Python 3.0 ...READ MORE

answered Oct 30, 2018 in Python by SDeb
• 9,380 points
16 views
0 votes
1 answer

How to create and read from a temporary file in Python?

Hi, there is a very simple solution ...READ MORE

answered Jan 29 in Python by Nymeria
• 3,500 points
12 views
0 votes
1 answer

How to read numbers from file in Python?

Assuming you don't have extraneous whitespace: with open('file') ...READ MORE

answered Apr 16 in Python by SDeb
• 9,380 points
52 views
+3 votes
5 answers

How to read multiple data files in python

Firstly we will import pandas to read ...READ MORE

answered Apr 6, 2018 in Python by DeepCoder786
• 1,700 points
494 views
0 votes
1 answer
0 votes
1 answer

Getting friendly device names in python

Regarding Linux, if all you need is ...READ MORE

answered Mar 26 in Python by SDeb
• 9,380 points
6 views
0 votes
1 answer

How to read Excel File in Python

With pandas it is possible to get ...READ MORE

answered Oct 22, 2018 in Python by Priyaj
• 56,100 points
55 views
+2 votes
3 answers

How can I play an audio file in the background using Python?

down voteacceptedFor windows: you could use  winsound.SND_ASYNC to play them ...READ MORE

answered Apr 3, 2018 in Python by charlie_brown
• 7,710 points
937 views

© 2018 Brain4ce Education Solutions Pvt. Ltd. All rights Reserved.
"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.