What are some good web crawlers in Python

0 votes
I am new to Python. Now I want to learn web crawling in Python. What are some good frameworks used for this purpose in Python?
Jul 22, 2019 in Python by Neel
• 3,020 points

1 answer to this question.

0 votes
Following are the libraries that can help you perform web crawling -

1. Scrapy

2. Urllib2

3. Beautifulsoup

4. Mechanize

5. Twil
answered Jul 22, 2019 by Arvind
• 3,040 points
Which one is the best?

Personally I like beautifulsoup. It is very convenient to use. Following are the steps involved in implementing it for web crawling -

  1. Send a HTTP request to the URL of the webpage you want to access. The server responds to the request by returning the HTML content of the webpage. For this task, we will use a third-party HTTP library for python requests.
  2. Once we have accessed the HTML content, we are left with the task of parsing the data. Since most of the HTML data is nested, we cannot extract data simply through string processing. One needs a parser which can create a nested/tree structure of the HTML data.
    There are many HTML parser libraries available but the most advanced one is html5lib.
  3. Now, all we need to do is navigating and searching the parse tree that we created, i.e. tree traversal. For this task, we will be using another third-party python library, beautifulsoup. It is a Python library for pulling data out of HTML and XML files. 
to know more you refer to the following link - https://bit.ly/2Gqjbis

Related Questions In Python

0 votes
0 answers

What are some sources to learn design techniques in python?

The idea is to build creative error ...READ MORE

Jun 6, 2019 in Python by Waseem
• 4,540 points
0 votes
1 answer
0 votes
1 answer

what are "and" and "or" operators in Python?

AND - True if both the operands ...READ MORE

answered Apr 18, 2018 in Python by Johnathon
• 9,090 points
0 votes
2 answers

What are the types of dictionary in python?

There are 4 types of dictionary Empty Integer Mixed Dictionary with ...READ MORE

answered Feb 14, 2019 in Python by Shashank
• 1,370 points
0 votes
2 answers
+1 vote
2 answers

how can i count the items in a list?

Syntax :            list. count(value) Code: colors = ['red', 'green', ...READ MORE

answered Jul 7, 2019 in Python by Neha
• 330 points

edited Jul 8, 2019 by Kalgi 4,205 views
0 votes
1 answer
0 votes
1 answer

What are first class objects in Python?

First-class objects in a language are handled uniformly ...READ MORE

answered Aug 2, 2019 in Python by Arvind
• 3,040 points
0 votes
1 answer

What is the difference between str() and repr() functions in Python?

str() is mostly used to create output ...READ MORE

answered Jul 8, 2019 in Python by Arvind
• 3,040 points
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP