How to use Pandas HDF5 as a Database in Python?

0 votes

Pandas is not a database, right?

Is there a way to integrate the analysis power of pandas into a flat HDF5 file database? I know that unfortunately HDF5 is not designed to deal natively with concurrency.

Also been looking for inspiration into parallel HDF5, flat file database managers or multiprocessing but I still lack an idea. Help?

Nov 30, 2018 in Python by Anirudh
• 2,070 points

edited Dec 10, 2018 by Anirudh 98 views

1 answer to this question.

0 votes

HDF5 works fine for concurrent read only access.

For concurrent write access you either have to use parallel HDF5 or have a worker process that takes care of writing to an HDF5 store.

I recommend to use a hybrid approach and expose it via a RESTful API. You can store meta-information in a SQL/NoSQL database and keep the raw data (time series data) in one or multiple HDF5 files.

There is one public REST API to access the data and the user doesn't have to care what happens behind the curtains.

answered Nov 30, 2018 by Nymeria
• 3,520 points

edited Dec 10, 2018 by Nymeria

Related Questions In Python

0 votes
2 answers

How do I connect to a MySQL Database in Python?

connect mysql database with python import MySQLdb db = ...READ MORE

answered Mar 28 in Python by rajesh
• 1,210 points
137 views
0 votes
1 answer

How to correctly return an a dictionary as an output in zappier code using python?

David here, from the Zapier Platform team. ...READ MORE

answered Dec 3, 2018 in Python by charlie_brown
• 7,720 points
111 views
0 votes
3 answers
0 votes
1 answer

How to add a new line in Python?

You can use '\n' for a next ...READ MORE

answered May 2, 2018 in Python by aayushi
• 750 points
87 views
+1 vote
2 answers

how can i count the items in a list?

Syntax :            list. count(value) Code: colors = ['red', 'green', ...READ MORE

answered Jul 6 in Python by Neha
• 330 points

edited Jul 8 by Kalgi 216 views
+4 votes
6 answers
0 votes
1 answer

How to convert a Pandas GroupBy object to DataFrame in Python

g1 here is a DataFrame. It has a hierarchical index, ...READ MORE

answered Nov 12, 2018 in Python by Nymeria
• 3,520 points
5,437 views
0 votes
1 answer

How do I use urllib to see if a website is 404 or 200 in Python?

For Python 3, try doing this: import urllib.request, ...READ MORE

answered Nov 29, 2018 in Python by Nymeria
• 3,520 points

edited Dec 11, 2018 by Nymeria 582 views