Using SQL Stored Procedure vs Databricks in Azure Data Factory

0 votes

ADF pipeline must be used to write up to 500k records per day to Azure SQL DB. I used some straightforward calculations as part of the data transformation process that a SQL Stored Procedure activity can carry out. Additionally, I've noticed that Databricks Notebooks are frequently used, particularly because of their future scalability. However, there are overhead activities such as managing authentication, moving files to a new location after transformation, etc., so I want to avoid over-engineering unless absolutely necessary. For about 50k records, the SQL Stored Proc I tested performed admirably (not yet tested with higher volumes).

However, I'd still like to hear a general opinion about which of the two options is better, especially from knowledgeable Azure or data engineers. 

Thanks

Feb 14, 2023 in Azure by Damonlang
• 1,230 points
948 views

No answer to this question. Be the first to respond.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.

Related Questions In Azure

0 votes
1 answer

Unzip gzip files in Azure Data factory

How to unzip a GZIP file? here's ...READ MORE

answered Mar 24, 2022 in Azure by Edureka
• 12,690 points
2,261 views
0 votes
0 answers
0 votes
1 answer

Sign in to Azure Account from VS Code

Open VS Code. Select Extensions on the left-side corner. In the ...READ MORE

answered Mar 30, 2022 in Azure by Edureka
• 13,620 points

edited Jun 21, 2023 by Khan Sarfaraz 6,376 views
0 votes
0 answers

Best way to keep Azure Postgres in sync with Azure SQL

For the use of pretty much everything, ...READ MORE

Feb 14, 2023 in Azure by Damonlang
• 1,230 points
488 views
0 votes
1 answer

How to create table in Azure SQL Database?

Creating tables in Azure SQL Database: In Object Explorer, ...READ MORE

answered Jan 22, 2020 in Azure by Sirajul
• 59,230 points
37,838 views
0 votes
1 answer
0 votes
1 answer

Does Azure have a FREE tier?

When you join up for a free ...READ MORE

answered Apr 11, 2022 in Azure by Edureka
• 12,690 points

edited Jun 27, 2023 by Khan Sarfaraz 841 views
0 votes
1 answer

Power BI and Azure Key vault

Yes, Azure Key Vault is compatible with ...READ MORE

answered Apr 11, 2022 in Azure by Edureka
• 12,690 points
1,971 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP