what technologies to integrate with selenium webdriver for making it works at large scale

+1 vote
Hi everyone,

I try to collect some posts from Facebook for research purposes. The amount of data collected is growing by time and I would like to ask if there are some technologies to integrate with selenium webdriver (using java) for making it works on at large scale when scraping data from Facebook.

Thanks in advance.
Oct 20, 2019 in Selenium by Amina
• 130 points

1 answer to this question.

0 votes

Hi Amina, you can use following technologies to work on large scale web scraping project along with Selenium Webdriver:

  • JUnit:  Open source Unit Testing Framework for JAVA. Useful for Java Developers to write and run repeatable tests and important in the development of test-driven development

  • Apache Ant: Java library and command-line tool whose mission is to drive processes described in build files as targets and extension points dependent upon each other. The main known usage of Ant is the build of Java applications.

  • Jenkins: An open source automation server which enables developers around the world to reliably build, test, and deploy their software.

  • Eclipse: A developer workspace server and cloud IDE developed using Java. Eclipse can be used as an IDE for any programming language for which a plug-in is available.

  • Git/Bitbucket: Free and open source distributed version control systems  designed to handle everything from small to very large projects with speed and efficiency. 

  • QuerySurge: The smart Data Testing solution that automates the data validation & testing of Big Data, Data Warehouses, and Business Intelligence Reports.

answered Oct 21, 2019 by Abha
• 28,140 points

Related Questions In Selenium

0 votes
1 answer

Is it possible for a website to detect that we are using Selenium with ChromeDriver

Selenium tests for pre-defined javascript variables which ...READ MORE

answered Apr 28, 2018 in Selenium by Meci Matt
• 9,460 points
0 votes
2 answers
0 votes
1 answer
0 votes
1 answer
+4 votes
9 answers

***IMPORTANT*** AngularJS Interview Questions.

Yes, I agree with Omkar AngularJs is ...READ MORE

answered Mar 17, 2019 in Career Counselling by Sharad
• 180 points
0 votes
1 answer

Cannot refresh AWS Web console during EC2 reboot

This never happens but you can see ...READ MORE

answered Oct 17, 2018 in AWS by Priyaj
• 58,090 points
0 votes
1 answer

Sending Web requests from my Web server to my IoT device

You need to start an HTTP server ...READ MORE

answered Jan 8, 2019 in IoT (Internet of Things) by nirvana
• 3,130 points
0 votes
1 answer

How to run Nutch in Hadoop installed in pseudo-distributed mode

Make sure you have built Nutch from ...READ MORE

answered Jan 24, 2019 in Big Data Hadoop by Frankie
• 9,830 points
0 votes
1 answer
0 votes
1 answer

What is Apache POI in Selenium and what it is used for?

Hi Raj, Apache POI is the most commonly ...READ MORE

answered May 8, 2019 in Selenium by Abha
• 28,140 points
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP