How to perform web scraping with Selenium

Question

I'm trying to scrape a website for the list of company names, code, industry, sector, market cap, etc in the table with Selenium. I'm new to it and I have tried writing the below code:

path_to_chromedriver = r'C:\Documents\chromedriver'
browser = webdriver.Chrome(executable_path=path_to_chromedriver)

url = r'http://sgx.com/wps/portal/sgxweb/home/company_disclosure/stockfacts'
browser.get(url)

time.sleep(15)
output = browser.page_source
print(output) 

However, I'm able to get the below tags, but not the data in it.

            <div class="table-wrapper results-display">
                <table>
                    <thead>
                        <tr></tr>
                    </thead>
                    <tbody></tbody>
                </table>
            </div>
            <div class="pager results-display"></div>

I have also tried BS4 to scrape it but failed at it. Any help on this is much appreciated.

Answer 1

Hey!

The results are in the iframe so, switch to it and then get the .page_source

iframe = driver.find_element_by_css_selector("#mainContent iframe")
driver.switch_to.frame(iframe)

I also added a wait for the table to be loaded:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10)

#locate and switch to the iframe
iframe = driver.find_element_by_css_selector("#mainContent iframe")
driver.switch_to.frame(iframe)

#wait for the table to load
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,'companyName')))

print(driver.page_source)

It worked for me. Hope this helps