How to use Pandas HDF5 as a Database in Python?

0 votes

Pandas is not a database, right?

Is there a way to integrate the analysis power of pandas into a flat HDF5 file database? I know that unfortunately HDF5 is not designed to deal natively with concurrency.

Also been looking for inspiration into parallel HDF5, flat file database managers or multiprocessing but I still lack an idea. Help?

Nov 30, 2018 in Python by Anirudh
• 2,050 points

edited Dec 10, 2018 by Anirudh 48 views

1 answer to this question.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
0 votes

HDF5 works fine for concurrent read only access.

For concurrent write access you either have to use parallel HDF5 or have a worker process that takes care of writing to an HDF5 store.

I recommend to use a hybrid approach and expose it via a RESTful API. You can store meta-information in a SQL/NoSQL database and keep the raw data (time series data) in one or multiple HDF5 files.

There is one public REST API to access the data and the user doesn't have to care what happens behind the curtains.

answered Nov 30, 2018 by Nymeria
• 3,500 points

edited Dec 10, 2018 by Nymeria

Related Questions In Python

0 votes
2 answers

How do I connect to a MySQL Database in Python?

connect mysql database with python import MySQLdb db = ...READ MORE

answered Mar 28 in Python by rajesh
• 1,210 points
60 views
0 votes
1 answer

How to correctly return an a dictionary as an output in zappier code using python?

David here, from the Zapier Platform team. ...READ MORE

answered Dec 3, 2018 in Python by charlie_brown
• 7,710 points
58 views
0 votes
2 answers

What is a Tuple in Python and how to use it?

  Tuple is a collection of Python objects.it is ...READ MORE

answered Apr 12 in Python by rajesh
• 1,210 points
44 views
0 votes
1 answer

How to add a new line in Python?

You can use '\n' for a next ...READ MORE

answered May 2, 2018 in Python by aayushi
• 750 points
22 views
0 votes
1 answer

how can i count the items in a list?

suppose you have a list a = [0,1,2,3,4,5,6,7,8,9,10] now ...READ MORE

answered May 2 in Python by Mohammad
• 1,400 points
20 views
+4 votes
6 answers
0 votes
1 answer

How to convert a Pandas GroupBy object to DataFrame in Python

g1 here is a DataFrame. It has a hierarchical index, ...READ MORE

answered Nov 12, 2018 in Python by Nymeria
• 3,500 points
2,385 views
0 votes
1 answer

How do I use urllib to see if a website is 404 or 200 in Python?

For Python 3, try doing this: import urllib.request, ...READ MORE

answered Nov 29, 2018 in Python by Nymeria
• 3,500 points

edited Dec 11, 2018 by Nymeria 120 views

© 2018 Brain4ce Education Solutions Pvt. Ltd. All rights Reserved.
"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.