How to perform web scraping with Selenium?

0 votes

I'm trying to scrape a website for the list of company names, code, industry, sector, market cap, etc in the table with Selenium. I'm new to it and I have tried writing the below code:

path_to_chromedriver = r'C:\Documents\chromedriver'
browser = webdriver.Chrome(executable_path=path_to_chromedriver)

url = r'http://sgx.com/wps/portal/sgxweb/home/company_disclosure/stockfacts'
browser.get(url)

time.sleep(15)
output = browser.page_source
print(output) 

However, I'm able to get the below tags, but not the data in it.

            <div class="table-wrapper results-display">
                <table>
                    <thead>
                        <tr></tr>
                    </thead>
                    <tbody></tbody>
                </table>
            </div>
            <div class="pager results-display"></div>

I have also tried BS4 to scrape it but failed at it. Any help on this is much appreciated.

Mar 25 in Selenium by Surya
• 820 points
42 views

1 answer to this question.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
0 votes

Hey!

The results are in the iframe so, switch to it and then get the .page_source

iframe = driver.find_element_by_css_selector("#mainContent iframe")
driver.switch_to.frame(iframe)

I also added a wait for the table to be loaded:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10)

#locate and switch to the iframe
iframe = driver.find_element_by_css_selector("#mainContent iframe")
driver.switch_to.frame(iframe)

#wait for the table to load
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,'companyName')))

print(driver.page_source)

It worked for me. Hope this helps

answered Mar 25 by Vaishnavi
• 720 points

Related Questions In Selenium

0 votes
1 answer

How to resolve Pycharm Referenced Error with Import Selenium Web driver?

Below will help: Pycharm > Preferences > Project ...READ MORE

answered Jun 26, 2018 in Selenium by Samarpit
• 5,130 points
1,054 views
0 votes
1 answer
0 votes
1 answer
0 votes
1 answer

Need to perform parallel execution (multiple browser sessions) with Selenium & TestNG by using @BeforeSuite

@Beforesuite annotated method runs before the testNG.XML ...READ MORE

answered Mar 30, 2018 in Selenium by nsv999
• 5,030 points
901 views
0 votes
1 answer

Installing Selenium Webdriver with Python package

Hey Hemant, for installing Selenium Webdriver with ...READ MORE

answered May 8 in Selenium by Anvi
• 5,960 points
15 views
0 votes
1 answer

Finding WebDriver element with Class Name in java

The better way to handle this element ...READ MORE

answered Apr 10, 2018 in Selenium by nsv999
• 5,030 points
75 views
0 votes
1 answer
0 votes
1 answer

Geo-location microphone camera pop up

To Allow or Block the notification, access using Selenium and you have to ...READ MORE

answered May 11, 2018 in Selenium by Samarpit
• 5,130 points
448 views
0 votes
1 answer

How to auto-refresh the ChromeDriver using Selenium Webdriver?

You can use this command. Also, refresh ...READ MORE

answered May 10 in Selenium by Vaishnavi
• 720 points

edited May 10 by Omkar 32 views
0 votes
1 answer

© 2018 Brain4ce Education Solutions Pvt. Ltd. All rights Reserved.
"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.