web scraping using python

0 votes
How can I scrape details like name email job and link of profile from websites like GitHub etc? Thank you!
May 6, 2020 in Python by Pythonist
• 120 points
1,022 views

1 answer to this question.

0 votes

Hey, @Pythonist,

You can try this,

##################################### Method 1
import mechanize
import cookielib
from BeautifulSoup import BeautifulSoup
import html2text

# Browser
br = mechanize.Browser()

# Cookie Jar
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)

# Browser options
br.set_handle_equiv(True)
br.set_handle_gzip(True)
br.set_handle_redirect(True)
br.set_handle_referer(True)
br.set_handle_robots(False)
br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)

br.addheaders = [('User-agent', 'Chrome')]

# The site we will navigate into, handling it's session
br.open('https://github.com/login')

# View available forms
for f in br.forms():
    print f

# Select the second (index one) form (the first form is a search query box)
br.select_form(nr=1)

# User credentials
br.form['login'] = 'mylogin'
br.form['password'] = 'mypass'

# Login
br.submit()

print(br.open('https://github.com/settings/emails').read())
answered May 6, 2020 by Gitika
• 65,910 points

Related Questions In Python

0 votes
1 answer

How to perform web scraping with python?

Hey, there are various libraries used in ...READ MORE

answered Apr 20, 2018 in Python by aayushi
• 750 points
1,562 views
0 votes
1 answer

How to web scrape using python without using a browser?

Yes, you can use the headless mode. ...READ MORE

answered Apr 2, 2019 in Python by Yogi

edited Oct 7, 2021 by Sarfaraz 12,381 views
0 votes
1 answer
0 votes
1 answer
+1 vote
1 answer

Web Scraping with Python:-> ModuleNotFoundError: no module named 'selenium'

Hello @shamsher, First try to check what version ...READ MORE

answered Aug 10, 2020 in Python by Niroj
• 82,880 points
3,766 views
+2 votes
2 answers

How to make a laplacian pyramid using OpenCV python?

down voteacceptTheeThe problem is that you're iterating ...READ MORE

answered Apr 3, 2018 in Python by charlie_brown
• 7,720 points
4,452 views
+2 votes
3 answers

How can I play an audio file in the background using Python?

down voteacceptedFor windows: you could use  winsound.SND_ASYNC to play them ...READ MORE

answered Apr 4, 2018 in Python by charlie_brown
• 7,720 points
12,858 views
0 votes
1 answer

I want to download a file from the website by web scraping. Can anyone explain how to do this in jupyter lab (python) with an example?

Hey, Web scraping is a technique to automatically ...READ MORE

answered Apr 7, 2020 in Python by Gitika
• 65,910 points
2,066 views
0 votes
3 answers

How to get the return value from a thread using python?

FWIW, the multiprocessing module has a nice interface for ...READ MORE

answered Dec 15, 2020 in Python by Roshni
• 10,520 points
105,026 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP