what technologies to integrate with selenium webdriver for making it works at large scale

Question

Hi everyone,

I try to collect some posts from Facebook for research purposes. The amount of data collected is growing by time and I would like to ask if there are some technologies to integrate with selenium webdriver (using java) for making it works on at large scale when scraping data from Facebook.

Thanks in advance.

Answer 1

Hi Amina, you can use following technologies to work on large scale web scraping project along with Selenium Webdriver:

• JUnit:  Open source Unit Testing Framework for JAVA. Useful for Java Developers to write and run repeatable tests and important in the development of test-driven development

• Apache Ant: Java library and command-line tool whose mission is to drive processes described in build files as targets and extension points dependent upon each other. The main known usage of Ant is the build of Java applications.

• Jenkins: An open source automation server which enables developers around the world to reliably build, test, and deploy their software.

• Eclipse: A developer workspace server and cloud IDE developed using Java. Eclipse can be used as an IDE for any programming language for which a plug-in is available.

• Git/Bitbucket: Free and open source distributed version control systems  designed to handle everything from small to very large projects with speed and efficiency. 

• QuerySurge: The smart Data Testing solution that automates the data validation & testing of Big Data, Data Warehouses, and Business Intelligence Reports.