How to get page source as it is in browser using selenium

+1 vote

I know that driver.getPageSource() gets you the web page as Html but the issue I am having is that the site I am testing puts data in brackets like {{{DepartDayShort}}}, maybe using some js framework to get the data? I can view the data when I inspect the element in browser but viewing the page source in browser also show only the brackets and not the data.

Below is the website I am testing to be exact:

driver.get("https://www.united.com/ual/en/US/flight-search/book-a-flight/results/rev?f=Miami,+FL,+US+(MIA+-+All+Airports)&t=IAH&d=2020-05-02&r=2020-05-16&sc=7,7&px=1&taxng=1&newHP=True&idx=1");
String pageSource = driver.getPageSource();
Apr 17, 2020 in Selenium by laiman
• 330 points
8,785 views

Have you tried using 

String pageSource=driver.findElement(By.tagName("body")).getText()

Hi, I tried that but it only gives me stripped-down information like page title and headers. It does not contain much of the pages info.

2 answers to this question.

+1 vote
Best answer

I needed to wait for an element to load first before getting the page source:

WebDriverWait wait = new WebDriverWait(driver, SECONDS);
wait.until(ExpectedConditions.elementToBeClickable(By.className("ClassName")));
String pageSource = driver.getPageSource();
answered Apr 18, 2020 by laiman
• 330 points

selected Apr 19, 2020 by Sirajul
+1 vote

I too had faced a problem while viewing the source code in the browser. There were many extra characters that got appended apparently. By using JavaScript code to get the innerHTML property, it finally worked for me. You could probably try executing the following code:

String javascript = "return arguments[0].innerHTML";
String pageSource=(String)(JavascriptExecutor)driver)
    .executeScript(javascript, driver.findElement(By.tagName("html")));
pageSource = "<html>"+pageSource +"</html>";
System.out.println(pageSource);
//FileUtils.write(new File("e:\\test.html"), pageSource,);
answered Apr 17, 2020 by Liana

Hi, this is an interesting approach. However, it was still not getting the data I was looking for. It did head me in the right direction as I needed to use  WebDriverWait wait and ExpectedConditions.elementToBeClickable to get the data.

I had this in a different situation where the page is different from the save page data, and it worked perfect.

String javascript = "return arguments[0].innerHTML";
String pageSource =(String)((JavascriptExecutor)driver).executeScript(javascript, driver.findElement(By.tagName("html")));
pageSource = "<html>"+pageSource +"</html>";

Hey @Laiman, Thanks a lot for your contribution. Could you please update the original answer?

Related Questions In Selenium

0 votes
1 answer
0 votes
2 answers
+1 vote
1 answer

How to get the HTML source of a webpage using Selenium in Java?

There is a method called getPageSource() in ...READ MORE

answered Jan 10, 2019 in Selenium by Sneha
17,717 views
0 votes
1 answer

How to extract text from a web page using selenium and save it as a text file?

Hello Isha, you can checkout this code ...READ MORE

answered May 7, 2019 in Selenium by Anvi
• 14,150 points
26,963 views
0 votes
1 answer
0 votes
2 answers

Finding WebDriver element with Class Name in java

The better way to handle this element ...READ MORE

answered Apr 10, 2018 in Selenium by nsv999
• 5,520 points
5,080 views
0 votes
2 answers

Problem while using InternetExplorerDriver in Selenium WebDriver

enable trusted connection  in internet explorer by ...READ MORE

answered Aug 31, 2020 in Selenium by Sri
• 3,010 points
5,816 views
0 votes
1 answer

Geo-location microphone camera pop up

To Allow or Block the notification, access using Selenium and you have to ...READ MORE

answered May 11, 2018 in Selenium by Samarpit
• 5,890 points
3,580 views
0 votes
2 answers

How to use such xpath to find web elements

xpath are two types. 1) Absolute XPath:    /html/b ...READ MORE

answered Sep 2, 2020 in Selenium by Sri
• 3,010 points
4,323 views