web scraping using python

0 votes
How can I scrape details like name email job and link of profile from websites like GitHub etc? Thank you!
May 6 in Python by Pythonist
• 120 points
159 views

1 answer to this question.

0 votes

Hey, @Pythonist,

You can try this,

##################################### Method 1
import mechanize
import cookielib
from BeautifulSoup import BeautifulSoup
import html2text

# Browser
br = mechanize.Browser()

# Cookie Jar
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)

# Browser options
br.set_handle_equiv(True)
br.set_handle_gzip(True)
br.set_handle_redirect(True)
br.set_handle_referer(True)
br.set_handle_robots(False)
br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)

br.addheaders = [('User-agent', 'Chrome')]

# The site we will navigate into, handling it's session
br.open('https://github.com/login')

# View available forms
for f in br.forms():
    print f

# Select the second (index one) form (the first form is a search query box)
br.select_form(nr=1)

# User credentials
br.form['login'] = 'mylogin'
br.form['password'] = 'mypass'

# Login
br.submit()

print(br.open('https://github.com/settings/emails').read())
answered May 6 by Gitika
• 41,720 points

Related Questions In Python

0 votes
1 answer

How to perform web scraping with python?

Hey, there are various libraries used in ...READ MORE

answered Apr 20, 2018 in Python by aayushi
• 750 points
738 views
0 votes
1 answer
0 votes
1 answer
0 votes
1 answer
+1 vote
1 answer

Web Scraping with Python:-> ModuleNotFoundError: no module named 'selenium'

Hello @shamsher, First try to check what version ...READ MORE

answered Aug 9 in Python by Niroj
• 67,020 points
369 views
+2 votes
2 answers

How to make a laplacian pyramid using OpenCV python?

down voteacceptTheeThe problem is that you're iterating ...READ MORE

answered Apr 3, 2018 in Python by charlie_brown
• 7,780 points
2,829 views
+2 votes
3 answers

How can I play an audio file in the background using Python?

down voteacceptedFor windows: you could use  winsound.SND_ASYNC to play them ...READ MORE

answered Apr 3, 2018 in Python by charlie_brown
• 7,780 points
6,452 views
0 votes
1 answer
+1 vote
1 answer

Want to know how to write paragraph on image using pillow python?

Hi, @Shantanu, You can make use of the ...READ MORE

answered Apr 14 in Python by Gitika
• 41,720 points
492 views