How to use Pandas HDF5 as a Database in Python?

0 votes

Pandas is not a database, right?

Is there a way to integrate the analysis power of pandas into a flat HDF5 file database? I know that unfortunately HDF5 is not designed to deal natively with concurrency.

Also been looking for inspiration into parallel HDF5, flat file database managers or multiprocessing but I still lack an idea. Help?

Nov 30, 2018 in Python by Anirudh
• 2,080 points

edited Dec 10, 2018 by Anirudh 198 views

1 answer to this question.

0 votes

HDF5 works fine for concurrent read only access.

For concurrent write access you either have to use parallel HDF5 or have a worker process that takes care of writing to an HDF5 store.

I recommend to use a hybrid approach and expose it via a RESTful API. You can store meta-information in a SQL/NoSQL database and keep the raw data (time series data) in one or multiple HDF5 files.

There is one public REST API to access the data and the user doesn't have to care what happens behind the curtains.

answered Nov 30, 2018 by Nymeria
• 3,540 points

edited Dec 10, 2018 by Nymeria

Related Questions In Python

0 votes
2 answers

How do I connect to a MySQL Database in Python?

connect mysql database with python import MySQLdb db = ...READ MORE

answered Mar 28, 2019 in Python by rajesh
• 1,230 points
0 votes
1 answer

How to correctly return an a dictionary as an output in zappier code using python?

David here, from the Zapier Platform team. ...READ MORE

answered Dec 3, 2018 in Python by charlie_brown
• 7,760 points
0 votes
3 answers
0 votes
1 answer

How to add a new line in Python?

You can use '\n' for a next ...READ MORE

answered May 2, 2018 in Python by aayushi
• 750 points
+1 vote
2 answers

how can i count the items in a list?

Syntax :            list. count(value) Code: colors = ['red', 'green', ...READ MORE

answered Jul 6, 2019 in Python by Neha
• 330 points

edited Jul 8, 2019 by Kalgi 929 views
0 votes
0 answers
+4 votes
6 answers
0 votes
1 answer

How to convert a Pandas GroupBy object to DataFrame in Python

g1 here is a DataFrame. It has a hierarchical index, ...READ MORE

answered Nov 12, 2018 in Python by Nymeria
• 3,540 points
0 votes
1 answer

How do I use urllib to see if a website is 404 or 200 in Python?

For Python 3, try doing this: import urllib.request, ...READ MORE

answered Nov 29, 2018 in Python by Nymeria
• 3,540 points

edited Dec 11, 2018 by Nymeria 3,855 views