How to perform web scraping with Selenium

0 votes

I'm trying to scrape a website for the list of company names, code, industry, sector, market cap, etc in the table with Selenium. I'm new to it and I have tried writing the below code:

path_to_chromedriver = r'C:\Documents\chromedriver'
browser = webdriver.Chrome(executable_path=path_to_chromedriver)

url = r'https://www.edureka.co/'
browser.get(url)

time.sleep(15)
output = browser.page_source
print(output) 

However, I'm able to get the below tags, but not the data in it.

            <div class="table-wrapper results-display">
                <table>
                    <thead>
                        <tr></tr>
                    </thead>
                    <tbody></tbody>
                </table>
            </div>
            <div class="pager results-display"></div>

I have also tried BS4 to scrape it but failed at it. Any help on this is much appreciated.

Mar 26, 2019 in Selenium by Surya
• 970 points
1,639 views

1 answer to this question.

0 votes

Hey!

The results are in the iframe so, switch to it and then get the .page_source

iframe = driver.find_element_by_css_selector("#mainContent iframe")
driver.switch_to.frame(iframe)

I also added a wait for the table to be loaded:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10)

#locate and switch to the iframe
iframe = driver.find_element_by_css_selector("#mainContent iframe")
driver.switch_to.frame(iframe)

#wait for the table to load
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,'companyName')))

print(driver.page_source)

It worked for me. Hope this helps

answered Mar 26, 2019 by Vaishnavi
• 1,180 points

Related Questions In Selenium

0 votes
1 answer

How can I perform mouse hover to a web element with ruby and selenium webdriver?

Hi Siddhart, checkout the following script to ...READ MORE

answered Aug 26, 2019 in Selenium by Abha
• 28,140 points
2,594 views
0 votes
1 answer

How to resolve Pycharm Referenced Error with Import Selenium Web driver?

Below will help: Pycharm > Preferences > Project ...READ MORE

answered Jun 26, 2018 in Selenium by Samarpit
• 5,910 points
9,548 views
0 votes
1 answer

How to check if an image is displayed on web page while working with Selenium WebDriver?

I've come across a similar situation before, where the image ...READ MORE

answered May 10, 2019 in Selenium by Surya
• 970 points
7,902 views
0 votes
3 answers

How to print text from a list of all web elements with same class name in Selenium?

Try using List <WebElement> to access all similar elements ...READ MORE

answered Dec 16, 2020 in Selenium by Roshni
• 10,520 points
87,316 views
0 votes
1 answer

Installing Selenium Webdriver with Python package

Hey Hemant, for installing Selenium Webdriver with ...READ MORE

answered May 8, 2019 in Selenium by Anvi
• 14,150 points
15,820 views
0 votes
2 answers

Finding WebDriver element with Class Name in java

The better way to handle this element ...READ MORE

answered Apr 10, 2018 in Selenium by nsv999
• 5,500 points
13,633 views
0 votes
2 answers

Problem while using InternetExplorerDriver in Selenium WebDriver

enable trusted connection  in internet explorer by ...READ MORE

answered Aug 31, 2020 in Selenium by Sri
• 3,190 points
9,120 views
0 votes
1 answer

Geo-location microphone camera pop up

To Allow or Block the notification, access using Selenium and you have to ...READ MORE

answered May 11, 2018 in Selenium by Samarpit
• 5,910 points
7,207 views
0 votes
1 answer

How to auto-refresh the ChromeDriver using Selenium Webdriver?

You can use this command. Also, refresh ...READ MORE

answered May 10, 2019 in Selenium by Vaishnavi
• 1,180 points

edited May 10, 2019 by Omkar 13,700 views
0 votes
1 answer
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP