Can we get the HTTP Response Code in Selenium with Java?

+4 votes
I have these Selenium WebDriver tests and I want to check if the HTTP Request returns any HTTP 403 Forbidden. Can I get these HTTP response status code with Selenium WebDriver? Maybe display it in the logger or save to a file?
Apr 20, 2018 in Selenium by kappa3010
• 2,020 points
13,105 views

2 answers to this question.

+3 votes

It is indeed possible to get http request's response code when using Selenium with either Chrome or Firefox. But the catch is, you have to start them in logging mode. 
You have to tell chromedriver to do "Network.enable". This can be achieved by enabling Performance logging. Let me give you an example.

LoggingPreferences logPrefs = new LoggingPreferences();
logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
cap.setCapability(CapabilityType.LOGGING_PREFS, logPrefs);

Once the request is posted, you have to simply iterate the Perfomance logs and look for "Network.responseReceived" for requested url:

LogEntries logs = driver.manage().logs().get("performance");

Below is my complete code:

public class TestResponseCode
{
    public static void main(String[] args)
    {
        // simple page (without many resources so that the output is
        // easy to understand
        String url = "http://www.york.ac.uk/teaching/cws/wws/webpage1.html";

        DownloadPage(url);
    }

    private static void DownloadPage(String url)
    {
        ChromeDriver driver = null;

        try
        {
            ChromeOptions options = new ChromeOptions();
            // add whatever extensions you need
            // for example I needed one of adding proxy, and one for blocking
            // images
            // options.addExtensions(new File(file, "proxy.zip"));
            // options.addExtensions(new File("extensions",
            // "Block-image_v1.1.crx"));

            DesiredCapabilities cap = DesiredCapabilities.chrome();
            cap.setCapability(ChromeOptions.CAPABILITY, options);

            // set performance logger
            // this sends Network.enable to chromedriver
            LoggingPreferences logPrefs = new LoggingPreferences();
            logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
            cap.setCapability(CapabilityType.LOGGING_PREFS, logPrefs);

            driver = new ChromeDriver(cap);

            // navigate to the page
            System.out.println("Navigate to " + url);
            driver.navigate().to(url);

            // and capture the last recorded url (it may be a redirect, or the
            // original url)
            String currentURL = driver.getCurrentUrl();

            // then ask for all the performance logs from this request
            // one of them will contain the Network.responseReceived method
            // and we shall find the "last recorded url" response
            LogEntries logs = driver.manage().logs().get("performance");

            int status = -1;

            System.out.println("\nList of log entries:\n");

            for (Iterator<LogEntry> it = logs.iterator(); it.hasNext();)
            {
                LogEntry entry = it.next();

                try
                {
                    JSONObject json = new JSONObject(entry.getMessage());

                    System.out.println(json.toString());

                    JSONObject message = json.getJSONObject("message");
                    String method = message.getString("method");

                    if (method != null
                            && "Network.responseReceived".equals(method))
                    {
                        JSONObject params = message.getJSONObject("params");

                        JSONObject response = params.getJSONObject("response");
                        String messageUrl = response.getString("url");

                        if (currentURL.equals(messageUrl))
                        {
                            status = response.getInt("status");

                            System.out.println(
                                    "---------- bingo !!!!!!!!!!!!!! returned response for "
                                            + messageUrl + ": " + status);

                            System.out.println(
                                    "---------- bingo !!!!!!!!!!!!!! headers: "
                                            + response.get("headers"));
                        }
                    }
                } catch (JSONException e)
                {
                    // TODO Auto-generated catch block
                    e.printStackTrace();
                }
            }

            System.out.println("\nstatus code: " + status);
        } finally
        {
            if (driver != null)
            {
                driver.quit();
            }
        }
    }
}

And the log will look something like this:

Navigate to http://www.york.ac.uk/teaching/cws/wws/webpage1.html

List of log entries:

    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Page.frameAttached","params":{"parentFrameId":"172.1","frameId":"172.2"}}}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Page.frameStartedLoading","params":{"frameId":"172.2"}}}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Page.frameNavigated","params":{"frame":{"securityOrigin":"://","loaderId":"172.1","name":"chromedriver dummy frame","id":"172.2","mimeType":"text/html","parentId":"172.1","url":"about:blank"}}}}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Page.frameStoppedLoading","params":{"frameId":"172.2"}}}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Page.frameStartedLoading","params":{"frameId":"3928.1"}}}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Network.requestWillBeSent","params":{"request":{"headers":{"Upgrade-Insecure-Requests":"1","User-Agent":"Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36"},"initialPriority":"VeryHigh","method":"GET","mixedContentType":"none","url":"http://www.york.ac.uk/teaching/cws/wws/webpage1.html"},"frameId":"3928.1","requestId":"3928.1","documentURL":"http://www.york.ac.uk/teaching/cws/wws/webpage1.html","initiator":{"type":"other"},"loaderId":"3928.1","wallTime":1.47619492749007E9,"type":"Document","timestamp":20226.652971}}}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Network.responseReceived","params":{"frameId":"3928.1","requestId":"3928.1","response":{"headers":{"Accept-Ranges":"bytes","Keep-Alive":"timeout=4, max=100","Cache-Control":"max-age=300","Server":"Apache/2.2.22 (Ubuntu)","Connection":"Keep-Alive","Content-Encoding":"gzip","Vary":"Accept-Encoding","Expires":"Tue, 11 Oct 2016 14:13:47 GMT","Content-Length":"1957","Date":"Tue, 11 Oct 2016 14:08:47 GMT","Content-Type":"text/html"},"connectionReused":false,"timing":{"pushEnd":0,"workerStart":-1,"proxyEnd":-1,"workerReady":-1,"sslEnd":-1,"pushStart":0,"requestTime":20226.65335,"sslStart":-1,"dnsStart":0,"sendEnd":31.6569999995409,"connectEnd":31.4990000006219,"connectStart":0,"sendStart":31.5860000009707,"dnsEnd":0,"receiveHeadersEnd":115.645999998378,"proxyStart":-1},"encodedDataLength":-1,"remotePort":80,"mimeType":"text/html","headersText":"HTTP/1.1 200 OK\r\nDate: Tue, 11 Oct 2016 14:08:47 GMT\r\nServer: Apache/2.2.22 (Ubuntu)\r\nAccept-Ranges: bytes\r\nCache-Control: max-age=300\r\nExpires: Tue, 11 Oct 2016 14:13:47 GMT\r\nVary: Accept-Encoding\r\nContent-Encoding: gzip\r\nContent-Length: 1957\r\nKeep-Alive: timeout=4, max=100\r\nConnection: Keep-Alive\r\nContent-Type: text/html\r\n\r\n","securityState":"neutral","requestHeadersText":"GET /teaching/cws/wws/webpage1.html HTTP/1.1\r\nHost: www.york.ac.uk\r\nConnection: keep-alive\r\nUpgrade-Insecure-Requests: 1\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate, sdch\r\nAccept-Language: en-GB,en-US;q=0.8,en;q=0.6\r\n\r\n","url":"http://www.york.ac.uk/teaching/cws/wws/webpage1.html","protocol":"http/1.1","fromDiskCache":false,"fromServiceWorker":false,"requestHeaders":{"Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8","Upgrade-Insecure-Requests":"1","Connection":"keep-alive","User-Agent":"Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36","Host":"www.york.ac.uk","Accept-Encoding":"gzip, deflate, sdch","Accept-Language":"en-GB,en-US;q=0.8,en;q=0.6"},"remoteIPAddress":"144.32.128.84","statusText":"OK","connectionId":11,"status":200},"loaderId":"3928.1","type":"Document","timestamp":20226.770012}}}
    ---------- bingo !!!!!!!!!!!!!! returned response for http://www.york.ac.uk/teaching/cws/wws/webpage1.html: 200
    ---------- bingo !!!!!!!!!!!!!! headers: {"Accept-Ranges":"bytes","Keep-Alive":"timeout=4, max=100","Cache-Control":"max-age=300","Server":"Apache/2.2.22 (Ubuntu)","Connection":"Keep-Alive","Content-Encoding":"gzip","Vary":"Accept-Encoding","Expires":"Tue, 11 Oct 2016 14:13:47 GMT","Content-Length":"1957","Date":"Tue, 11 Oct 2016 14:08:47 GMT","Content-Type":"text/html"}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Network.dataReceived","params":{"dataLength":2111,"requestId":"3928.1","encodedDataLength":1460,"timestamp":20226.770425}}}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Page.frameNavigated","params":{"frame":{"securityOrigin":"http://www.york.ac.uk","loaderId":"3928.1","id":"3928.1","mimeType":"text/html","url":"http://www.york.ac.uk/teaching/cws/wws/webpage1.html"}}}}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Network.dataReceived","params":{"dataLength":1943,"requestId":"3928.1","encodedDataLength":825,"timestamp":20226.782673}}}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Network.loadingFinished","params":{"requestId":"3928.1","encodedDataLength":2285,"timestamp":20226.770199}}}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Page.loadEventFired","params":{"timestamp":20226.799391}}}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Page.frameStoppedLoading","params":{"frameId":"3928.1"}}}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Page.domContentEventFired","params":{"timestamp":20226.845769}}}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Network.requestWillBeSent","params":{"request":{"headers":{"Referer":"http://www.york.ac.uk/teaching/cws/wws/webpage1.html","User-Agent":"Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36"},"initialPriority":"High","method":"GET","mixedContentType":"none","url":"http://www.york.ac.uk/favicon.ico"},"frameId":"3928.1","requestId":"3928.2","documentURL":"http://www.york.ac.uk/teaching/cws/wws/webpage1.html","initiator":{"type":"other"},"loaderId":"3928.1","wallTime":1.47619492768527E9,"type":"Other","timestamp":20226.848174}}}

status code: 200

answered Apr 20, 2018 by king_kenny
• 3,650 points
Thanks for the Sample and the Logic
We're glad it helped :)

Hi All,

I am facing an error in the below line. which is 'remove arguments to match JSONObject'

'JSONObject json = new JSONObject(entry.getMessage());'

Kindly help.

Thanks,
Mano

Hey @Mano! I executed the above code and got no errors.. Could your share your code so that I can look into what's causing this error?
I want to get the Page source of all the consecutive URLs that are being sent by the chrome driver.

For example:

if I execute "http://localhost/test/script.html" it loads a JavaScript which in turn loads few more URLs. How to get the Page source of all the URLs loaded.

I tried using the LoggingPreferences (LogType.PERFORMANCE) but it gives page source of only the first URL.

Please advice. Thanks.
@Ramesh, if what you want is Page Source, then there is an inbuilt library for that.

You can use driver.getPageSource() to get the page source from every page you open through WebDriver and store it in a variable.

Let me know what you think about this idea.
@vardhan

The in built library driver.getPageSource returns the pagesource of the first URL.

I am unable to get the pagesource of consecutives URL being loaded.
maybe there is an issue with the Driver instance being loaded for every URL. Can you please share the script? I'll be able to help with that.
The Entire Code:
import org.json.JSONException;
import org.json.JSONObject;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriverService;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.logging.LogEntries;
import org.openqa.selenium.logging.LogEntry;
import org.openqa.selenium.logging.LogType;
import org.openqa.selenium.logging.LoggingPreferences;
import org.openqa.selenium.remote.CapabilityType;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;

import java.io.File;
import java.io.IOException;
import java.util.logging.Level;

public class browser {

    private static ChromeDriverService service;
    private static WebDriver driver;

    public static void main(String[] args) {
        createAndStartService();
        createDriver();
        String url = "http://test.everesttech.net/amo/rlsa/prd/or1/rlsa10.html";
        DownloadPage(url);
    }

    public static void createAndStartService() {
        service = new ChromeDriverService.Builder()
                .usingDriverExecutable(new File("drivers/chromedriver.exe"))
                .usingAnyFreePort()
                .build();
        try {
            service.start();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    public static void createDriver() {
        ChromeOptions options = new ChromeOptions();
        options.setHeadless(true);

        LoggingPreferences logPrefs = new LoggingPreferences();
        logPrefs.enable(LogType.PERFORMANCE, Level.ALL);

        DesiredCapabilities cap = DesiredCapabilities.chrome();
        cap.setCapability(ChromeOptions.CAPABILITY, options);
        cap.setCapability(CapabilityType.LOGGING_PREFS, logPrefs);

        driver = new RemoteWebDriver(service.getUrl(), cap);
    }


    private static void DownloadPage(String url) {
        try {
            System.out.println("Navigate to " + url);
            driver.get(url);
            String currentURL = driver.getCurrentUrl();
            LogEntries logs = driver.manage().logs().get("performance");

            for (LogEntry entry : logs) {
                try {
                    JSONObject json = new JSONObject(entry.getMessage());
                    JSONObject message = json.getJSONObject("message");
                    String method = message.getString("method");
                    if ("Network.responseReceived".equals(method)) {
                        JSONObject params = message.getJSONObject("params");
                        JSONObject response = params.getJSONObject("response");
                        String messageUrl = response.getString("url");
                        int status = response.getInt("status");
                        System.out.println("Current URL: " + currentURL + "\tMessage URL: " + messageUrl + "\tStatus: " + status);
                        System.out.println("Message Headers: " + response.get("headers"));
                    }
                } catch (JSONException e) {
                    e.printStackTrace();
                }
            }
        } finally {
            if (driver != null) {
                driver.quit();
            }
        }
    }
}
Perfect! Let me get back to you in sometime.
this doesn't provide network call with a post method
Hey @Nevil Panchal, can you elaborate the error that you are getting.
Getting this error
invalid argument: log type 'performance' not found

Can anyone please look into it?

This error occurs while using RemoteWebdriver, So using a pure ChromeDriver object still produces logs of type "performance." 

Produces performance logs: 

desiredCapabilites = DesiredCapabilities.chrome() 

LoggingPreferences logPrefs = new LoggingPreferences() 

logPrefs.enable(LogType.PERFORMANCE, Level.ALL) desiredCapabilites.setCapability(CapabilityType.LOGGING_PREFS, logPrefs) 

driver = { new ChromeDriver(desiredCapabilites) }
@Abha Can you please copy the entire code here.

Thanks much!!
Hey, can you tell which webdriver you are using? Are you using a remote webdriver? And any specific url for which you need http respond code and performance log?
@Abha
Please provide a complete program for any sample page visit which is successfully generating  Performance logs hence i can verify the Status code with Selenium Finally.

Thanks.
Is there a way to get response body ?
The above mentioned code will provide only provide data that we see under headers tab in devtools
0 votes

The Below code is working for the Status Code extraction using Selenium.

import java.util.Iterator;
import java.util.logging.Level;
import org.json.JSONException;
import org.json.JSONObject;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.logging.LogEntries;
import org.openqa.selenium.logging.LogEntry;
import org.openqa.selenium.logging.LogType;
import org.openqa.selenium.logging.LoggingPreferences;
import org.openqa.selenium.remote.DesiredCapabilities;

public class TestResponseCode {
    public static void main(String[] args) {
        // simple page (without many resources so that the output is
        // easy to understand
        String url = "https://www.bing.com/";
        downloadPage(url);
    }

    private static void downloadPage(String url) {
        System.setProperty("webdriver.chrome.driver", "D:\\Softwares\\Selenium\\Drivers\\ChromeDrivers\\75.0.3770.140\\chromedriver.exe");
        ChromeDriver driver = null;
        try {
            ChromeOptions options = new ChromeOptions();
            // add whatever extensions you need
            // for example I needed one of adding proxy, and one for blocking
            // images
            // options.addExtensions(new File(file, "proxy.zip"));
            // options.addExtensions(new File("extensions",
            // "Block-image_v1.1.crx"));
            DesiredCapabilities cap = DesiredCapabilities.chrome();
            cap.setCapability(ChromeOptions.CAPABILITY, options);
            // set performance logger
            // this sends Network.enable to chromedriver
            /*
             * LoggingPreferences logPrefs = new LoggingPreferences();
             * logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
             * cap.setCapability(CapabilityType.LOGGING_PREFS, logPrefs);
             */

            LoggingPreferences logPrefs = new LoggingPreferences();
            logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
            options.setCapability("goog:loggingPrefs", logPrefs);
            driver = new ChromeDriver(options);
            // navigate to the page
            System.out.println("Navigate to " + url);
            driver.navigate().to(url);
            // and capture the last recorded url (it may be a redirect, or the
            // original url)
            String currentURL = driver.getCurrentUrl();
            LogEntries logs = driver.manage().logs().get("performance");
            int status = -1;
            System.out.println("\\nList of log entries:\\n");
            for (Iterator<LogEntry> it = logs.iterator(); it.hasNext();) {
                LogEntry entry = it.next();
                try {
                    JSONObject json = new JSONObject(entry.getMessage());
                    System.out.println(json.toString());
                    JSONObject message = json.getJSONObject("message");
                    String method = message.getString("method");
                    if (method != null && "Network.responseReceived".equals(method)) {
                        JSONObject params = message.getJSONObject("params");
                        JSONObject response = params.getJSONObject("response");
                        String messageUrl = response.getString("url");
                        if (currentURL.equals(messageUrl)) {
                            status = response.getInt("status");
                            System.out.println("---------- bingo !!!!!!!!!!!!!! returned response for " + messageUrl
                                    + ": " + status);
                            System.out.println("---------- bingo !!!!!!!!!!!!!! headers: " + response.get("headers"));
                        }
                    }
                } catch (JSONException e) {
                    // TODO Auto-generated catch block
                    e.printStackTrace();
                }
            }
            System.out.println("\nstatus code: " + status);
        } finally {
            if (driver != null) {
                driver.quit();
            }
        }
    }
}
answered Jul 30 by rvemula
• 140 points
Hey Rvemula, thanks for sharing the code. It would really help.
You are a killer!!! Thanks!

Related Questions In Selenium

+2 votes
2 answers

How can I press ENTER key with the execute_script in selenium python?

The below code containing Keys.ENTER might just ...READ MORE

answered Mar 27, 2018 in Selenium by nsv999
• 5,090 points
7,332 views
0 votes
1 answer

Identifying the page iframe in Selenium using Java Code

Open the page in a browser and ...READ MORE

answered May 24, 2018 in Selenium by Samarpit
• 5,130 points
120 views
0 votes
1 answer
0 votes
1 answer

Finding WebDriver element with Class Name in java

The better way to handle this element ...READ MORE

answered Apr 10, 2018 in Selenium by nsv999
• 5,090 points
777 views
0 votes
1 answer
0 votes
1 answer

Geo-location microphone camera pop up

To Allow or Block the notification, access using Selenium and you have to ...READ MORE

answered May 11, 2018 in Selenium by Samarpit
• 5,130 points
1,022 views
0 votes
1 answer

How to use such xpath to find web elements

It's much simpler to use: #MyDiv123 > div.super or ...READ MORE

answered Jun 14, 2018 in Selenium by Samarpit
• 5,130 points
724 views
+2 votes
1 answer

How can we handle authentication popup in Selenium WebDriver using Java

The Alert Method, authenticateUsing() lets us skip ...READ MORE

answered Apr 18, 2018 in Selenium by king_kenny
• 3,650 points
7,634 views
0 votes
1 answer

Using XPath with Selenium-Java code

There is a minor syntax error in ...READ MORE

answered Apr 13, 2018 in Selenium by king_kenny
• 3,650 points
70 views