How to use Pandas HDF5 as a Database in Python

0 votes

Pandas is not a database, right?

Is there a way to integrate the analysis power of pandas into a flat HDF5 file database? I know that unfortunately HDF5 is not designed to deal natively with concurrency.

Also been looking for inspiration into parallel HDF5, flat file database managers or multiprocessing but I still lack an idea. Help?

Nov 30, 2018 in Python by Anirudh
• 2,080 points

edited Dec 10, 2018 by Anirudh 405 views

1 answer to this question.

0 votes

HDF5 works fine for concurrent read only access.

For concurrent write access you either have to use parallel HDF5 or have a worker process that takes care of writing to an HDF5 store.

I recommend to use a hybrid approach and expose it via a RESTful API. You can store meta-information in a SQL/NoSQL database and keep the raw data (time series data) in one or multiple HDF5 files.

There is one public REST API to access the data and the user doesn't have to care what happens behind the curtains.

answered Nov 30, 2018 by Nymeria
• 3,520 points

edited Dec 10, 2018 by Nymeria

Related Questions In Python

0 votes
2 answers

How do I connect to a MySQL Database in Python?

connect mysql database with python import MySQLdb db = ...READ MORE

answered Mar 28, 2019 in Python by rajesh
• 1,230 points
819 views
0 votes
1 answer

How to correctly return an a dictionary as an output in zappier code using python?

David here, from the Zapier Platform team. ...READ MORE

answered Dec 3, 2018 in Python by charlie_brown
• 7,780 points
589 views
0 votes
4 answers

What is a Tuple in Python and how to use it?

Tuples  are a  Unchanging sequence of values, ...READ MORE

answered Jun 21, 2020 in Python by sahil
• 540 points
637 views
0 votes
1 answer

How to create a database engine using Sqlalchemy in Python?

Hi@akhtar, You can use the Sqlalchemy module in ...READ MORE

answered Jun 26, 2020 in Python by MD
• 95,240 points
405 views
0 votes
2 answers
+1 vote
2 answers

how can i count the items in a list?

Syntax :            list. count(value) Code: colors = ['red', 'green', ...READ MORE

answered Jul 6, 2019 in Python by Neha
• 330 points

edited Jul 8, 2019 by Kalgi 2,254 views
0 votes
0 answers
+5 votes
6 answers

Lowercase in Python

You can simply the built-in function in ...READ MORE

answered Apr 11, 2018 in Python by hemant
• 5,810 points
1,687 views
0 votes
1 answer

How to convert a Pandas GroupBy object to DataFrame in Python

g1 here is a DataFrame. It has a hierarchical index, ...READ MORE

answered Nov 12, 2018 in Python by Nymeria
• 3,520 points
29,149 views
0 votes
1 answer

How do I use urllib to see if a website is 404 or 200 in Python?

For Python 3, try doing this: import urllib.request, ...READ MORE

answered Nov 29, 2018 in Python by Nymeria
• 3,520 points

edited Dec 11, 2018 by Nymeria 8,797 views