what technologies to integrate with selenium webdriver for making it works at large scale?

+1 vote
Hi everyone,

I try to collect some posts from Facebook for research purposes. The amount of data collected is growing by time and I would like to ask if there are some technologies to integrate with selenium webdriver (using java) for making it works on at large scale when scraping data from Facebook.

Thanks in advance.
Oct 19 in Selenium by Amina
• 130 points
89 views

1 answer to this question.

0 votes

Hi Amina, you can use following technologies to work on large scale web scraping project along with Selenium Webdriver:

  • JUnit:  Open source Unit Testing Framework for JAVA. Useful for Java Developers to write and run repeatable tests and important in the development of test-driven development

  • Apache Ant: Java library and command-line tool whose mission is to drive processes described in build files as targets and extension points dependent upon each other. The main known usage of Ant is the build of Java applications.

  • Jenkins: An open source automation server which enables developers around the world to reliably build, test, and deploy their software.

  • Eclipse: A developer workspace server and cloud IDE developed using Java. Eclipse can be used as an IDE for any programming language for which a plug-in is available.

  • Git/Bitbucket: Free and open source distributed version control systems  designed to handle everything from small to very large projects with speed and efficiency. 

  • QuerySurge: The smart Data Testing solution that automates the data validation & testing of Big Data, Data Warehouses, and Business Intelligence Reports.

answered Oct 21 by Abha
• 27,180 points

Related Questions In Selenium

0 votes
1 answer

Is it possible for a website to detect that we are using Selenium with ChromeDriver

Selenium tests for pre-defined javascript variables which ...READ MORE

answered Apr 27, 2018 in Selenium by Meci Matt
• 9,420 points
2,374 views
0 votes
1 answer
+4 votes
9 answers

***IMPORTANT*** AngularJS Interview Questions.

Yes, I agree with Omkar AngularJs is ...READ MORE

answered Mar 17 in Career Counselling by Sharad
• 180 points
556 views
0 votes
1 answer

Cannot refresh AWS Web console during EC2 reboot

This never happens but you can see ...READ MORE

answered Oct 17, 2018 in AWS by Priyaj
• 56,940 points
47 views
0 votes
1 answer

Sending Web requests from my Web server to my IoT device

You need to start an HTTP server ...READ MORE

answered Jan 8 in IoT (Internet of Things) by nirvana
• 3,060 points
81 views
0 votes
1 answer

How to run Nutch in Hadoop installed in pseudo-distributed mode

Make sure you have built Nutch from ...READ MORE

answered Jan 24 in Big Data Hadoop by Frankie
• 9,810 points
48 views
0 votes
1 answer

What is Apache POI in Selenium and what it is used for?

Hi Raj, Apache POI is the most commonly ...READ MORE

answered May 8 in Selenium by Abha
• 27,180 points
398 views