Can we get the HTTP Response Code in Selenium with Java

Question

I have these Selenium WebDriver tests and I want to check if the HTTP Request returns any HTTP 403 Forbidden. Can I get these HTTP response status code with Selenium WebDriver? Maybe display it in the logger or save to a file?

Answer 1

It is indeed possible to get http request's response code when using Selenium with either Chrome or Firefox. But the catch is, you have to start them in logging mode. 
You have to tell chromedriver to do "Network.enable". This can be achieved by enabling Performance logging. Let me give you an example.

LoggingPreferences logPrefs = new LoggingPreferences();
logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
cap.setCapability(CapabilityType.LOGGING_PREFS, logPrefs);

Once the request is posted, you have to simply iterate the Perfomance logs and look for "Network.responseReceived" for requested url:

LogEntries logs = driver.manage().logs().get("performance");

Below is my complete code:

public class TestResponseCode
{
    public static void main(String[] args)
    {
        // simple page (without many resources so that the output is
        // easy to understand
        String url = "http://www.york.ac.uk/teaching/cws/wws/webpage1.html";

        DownloadPage(url);
    }

    private static void DownloadPage(String url)
    {
        ChromeDriver driver = null;

        try
        {
            ChromeOptions options = new ChromeOptions();
            // add whatever extensions you need
            // for example I needed one of adding proxy, and one for blocking
            // images
            // options.addExtensions(new File(file, "proxy.zip"));
            // options.addExtensions(new File("extensions",
            // "Block-image_v1.1.crx"));

            DesiredCapabilities cap = DesiredCapabilities.chrome();
            cap.setCapability(ChromeOptions.CAPABILITY, options);

            // set performance logger
            // this sends Network.enable to chromedriver
            LoggingPreferences logPrefs = new LoggingPreferences();
            logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
            cap.setCapability(CapabilityType.LOGGING_PREFS, logPrefs);

            driver = new ChromeDriver(cap);

            // navigate to the page
            System.out.println("Navigate to " + url);
            driver.navigate().to(url);

            // and capture the last recorded url (it may be a redirect, or the
            // original url)
            String currentURL = driver.getCurrentUrl();

            // then ask for all the performance logs from this request
            // one of them will contain the Network.responseReceived method
            // and we shall find the "last recorded url" response
            LogEntries logs = driver.manage().logs().get("performance");

            int status = -1;

            System.out.println("\nList of log entries:\n");

            for (Iterator<LogEntry> it = logs.iterator(); it.hasNext();)
            {
                LogEntry entry = it.next();

                try
                {
                    JSONObject json = new JSONObject(entry.getMessage());

                    System.out.println(json.toString());

                    JSONObject message = json.getJSONObject("message");
                    String method = message.getString("method");

                    if (method != null
                            && "Network.responseReceived".equals(method))
                    {
                        JSONObject params = message.getJSONObject("params");

                        JSONObject response = params.getJSONObject("response");
                        String messageUrl = response.getString("url");

                        if (currentURL.equals(messageUrl))
                        {
                            status = response.getInt("status");

                            System.out.println(
                                    "---------- bingo !!!!!!!!!!!!!! returned response for "
                                            + messageUrl + ": " + status);

                            System.out.println(
                                    "---------- bingo !!!!!!!!!!!!!! headers: "
                                            + response.get("headers"));
                        }
                    }
                } catch (JSONException e)
                {
                    // TODO Auto-generated catch block
                    e.printStackTrace();
                }
            }

            System.out.println("\nstatus code: " + status);
        } finally
        {
            if (driver != null)
            {
                driver.quit();
            }
        }
    }
}

And the log will look something like this:

Navigate to http://www.york.ac.uk/teaching/cws/wws/webpage1.html

List of log entries:

    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Page.frameAttached","params":{"parentFrameId":"172.1","frameId":"172.2"}}}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Page.frameStartedLoading","params":{"frameId":"172.2"}}}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Page.frameNavigated","params":{"frame":{"securityOrigin":"://","loaderId":"172.1","name":"chromedriver dummy frame","id":"172.2","mimeType":"text/html","parentId":"172.1","url":"about:blank"}}}}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Page.frameStoppedLoading","params":{"frameId":"172.2"}}}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Page.frameStartedLoading","params":{"frameId":"3928.1"}}}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Network.requestWillBeSent","params":{"request":{"headers":{"Upgrade-Insecure-Requests":"1","User-Agent":"Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36"},"initialPriority":"VeryHigh","method":"GET","mixedContentType":"none","url":"http://www.york.ac.uk/teaching/cws/wws/webpage1.html"},"frameId":"3928.1","requestId":"3928.1","documentURL":"http://www.york.ac.uk/teaching/cws/wws/webpage1.html","initiator":{"type":"other"},"loaderId":"3928.1","wallTime":1.47619492749007E9,"type":"Document","timestamp":20226.652971}}}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Network.responseReceived","params":{"frameId":"3928.1","requestId":"3928.1","response":{"headers":{"Accept-Ranges":"bytes","Keep-Alive":"timeout=4, max=100","Cache-Control":"max-age=300","Server":"Apache/2.2.22 (Ubuntu)","Connection":"Keep-Alive","Content-Encoding":"gzip","Vary":"Accept-Encoding","Expires":"Tue, 11 Oct 2016 14:13:47 GMT","Content-Length":"1957","Date":"Tue, 11 Oct 2016 14:08:47 GMT","Content-Type":"text/html"},"connectionReused":false,"timing":{"pushEnd":0,"workerStart":-1,"proxyEnd":-1,"workerReady":-1,"sslEnd":-1,"pushStart":0,"requestTime":20226.65335,"sslStart":-1,"dnsStart":0,"sendEnd":31.6569999995409,"connectEnd":31.4990000006219,"connectStart":0,"sendStart":31.5860000009707,"dnsEnd":0,"receiveHeadersEnd":115.645999998378,"proxyStart":-1},"encodedDataLength":-1,"remotePort":80,"mimeType":"text/html","headersText":"HTTP/1.1 200 OK\r\nDate: Tue, 11 Oct 2016 14:08:47 GMT\r\nServer: Apache/2.2.22 (Ubuntu)\r\nAccept-Ranges: bytes\r\nCache-Control: max-age=300\r\nExpires: Tue, 11 Oct 2016 14:13:47 GMT\r\nVary: Accept-Encoding\r\nContent-Encoding: gzip\r\nContent-Length: 1957\r\nKeep-Alive: timeout=4, max=100\r\nConnection: Keep-Alive\r\nContent-Type: text/html\r\n\r\n","securityState":"neutral","requestHeadersText":"GET /teaching/cws/wws/webpage1.html HTTP/1.1\r\nHost: www.york.ac.uk\r\nConnection: keep-alive\r\nUpgrade-Insecure-Requests: 1\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate, sdch\r\nAccept-Language: en-GB,en-US;q=0.8,en;q=0.6\r\n\r\n","url":"http://www.york.ac.uk/teaching/cws/wws/webpage1.html","protocol":"http/1.1","fromDiskCache":false,"fromServiceWorker":false,"requestHeaders":{"Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8","Upgrade-Insecure-Requests":"1","Connection":"keep-alive","User-Agent":"Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36","Host":"www.york.ac.uk","Accept-Encoding":"gzip, deflate, sdch","Accept-Language":"en-GB,en-US;q=0.8,en;q=0.6"},"remoteIPAddress":"144.32.128.84","statusText":"OK","connectionId":11,"status":200},"loaderId":"3928.1","type":"Document","timestamp":20226.770012}}}
    ---------- bingo !!!!!!!!!!!!!! returned response for http://www.york.ac.uk/teaching/cws/wws/webpage1.html: 200
    ---------- bingo !!!!!!!!!!!!!! headers: {"Accept-Ranges":"bytes","Keep-Alive":"timeout=4, max=100","Cache-Control":"max-age=300","Server":"Apache/2.2.22 (Ubuntu)","Connection":"Keep-Alive","Content-Encoding":"gzip","Vary":"Accept-Encoding","Expires":"Tue, 11 Oct 2016 14:13:47 GMT","Content-Length":"1957","Date":"Tue, 11 Oct 2016 14:08:47 GMT","Content-Type":"text/html"}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Network.dataReceived","params":{"dataLength":2111,"requestId":"3928.1","encodedDataLength":1460,"timestamp":20226.770425}}}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Page.frameNavigated","params":{"frame":{"securityOrigin":"http://www.york.ac.uk","loaderId":"3928.1","id":"3928.1","mimeType":"text/html","url":"http://www.york.ac.uk/teaching/cws/wws/webpage1.html"}}}}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Network.dataReceived","params":{"dataLength":1943,"requestId":"3928.1","encodedDataLength":825,"timestamp":20226.782673}}}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Network.loadingFinished","params":{"requestId":"3928.1","encodedDataLength":2285,"timestamp":20226.770199}}}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Page.loadEventFired","params":{"timestamp":20226.799391}}}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Page.frameStoppedLoading","params":{"frameId":"3928.1"}}}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Page.domContentEventFired","params":{"timestamp":20226.845769}}}
    {"webview":"3b8eaedb-bd0f-4baa-938d-4aee4039abfe","message":{"method":"Network.requestWillBeSent","params":{"request":{"headers":{"Referer":"http://www.york.ac.uk/teaching/cws/wws/webpage1.html","User-Agent":"Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36"},"initialPriority":"High","method":"GET","mixedContentType":"none","url":"http://www.york.ac.uk/favicon.ico"},"frameId":"3928.1","requestId":"3928.2","documentURL":"http://www.york.ac.uk/teaching/cws/wws/webpage1.html","initiator":{"type":"other"},"loaderId":"3928.1","wallTime":1.47619492768527E9,"type":"Other","timestamp":20226.848174}}}

status code: 200

Comments

Thanks for the Sample and the Logic

---

We're glad it helped :)

---

Hi All,

I am facing an error in the below line. which is 'remove arguments to match JSONObject'

'JSONObject json = new JSONObject(entry.getMessage());'

Kindly help.

Thanks,
Mano

---

Hey @Mano! I executed the above code and got no errors.. Could your share your code so that I can look into what's causing this error?

---

I want to get the Page source of all the consecutive URLs that are being sent by the chrome driver.

For example:

if I execute "http://localhost/test/script.html" it loads a JavaScript which in turn loads few more URLs. How to get the Page source of all the URLs loaded.

I tried using the LoggingPreferences (LogType.PERFORMANCE) but it gives page source of only the first URL.

Please advice. Thanks.

---

@Ramesh, if what you want is Page Source, then there is an inbuilt library for that.

You can use driver.getPageSource() to get the page source from every page you open through WebDriver and store it in a variable.

Let me know what you think about this idea.

---

@vardhan

The in built library driver.getPageSource returns the pagesource of the first URL.

I am unable to get the pagesource of consecutives URL being loaded.

---

maybe there is an issue with the Driver instance being loaded for every URL. Can you please share the script? I'll be able to help with that.

---

The Entire Code:

import org.json.JSONException;
import org.json.JSONObject;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriverService;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.logging.LogEntries;
import org.openqa.selenium.logging.LogEntry;
import org.openqa.selenium.logging.LogType;
import org.openqa.selenium.logging.LoggingPreferences;
import org.openqa.selenium.remote.CapabilityType;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;

import java.io.File;
import java.io.IOException;
import java.util.logging.Level;

public class browser {

    private static ChromeDriverService service;
    private static WebDriver driver;

    public static void main(String[] args) {
        createAndStartService();
        createDriver();
        String url = "http://test.everesttech.net/amo/rlsa/prd/or1/rlsa10.html";
        DownloadPage(url);
    }

    public static void createAndStartService() {
        service = new ChromeDriverService.Builder()
                .usingDriverExecutable(new File("drivers/chromedriver.exe"))
                .usingAnyFreePort()
                .build();
        try {
            service.start();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    public static void createDriver() {
        ChromeOptions options = new ChromeOptions();
        options.setHeadless(true);

        LoggingPreferences logPrefs = new LoggingPreferences();
        logPrefs.enable(LogType.PERFORMANCE, Level.ALL);

        DesiredCapabilities cap = DesiredCapabilities.chrome();
        cap.setCapability(ChromeOptions.CAPABILITY, options);
        cap.setCapability(CapabilityType.LOGGING_PREFS, logPrefs);

        driver = new RemoteWebDriver(service.getUrl(), cap);
    }


    private static void DownloadPage(String url) {
        try {
            System.out.println("Navigate to " + url);
            driver.get(url);
            String currentURL = driver.getCurrentUrl();
            LogEntries logs = driver.manage().logs().get("performance");

            for (LogEntry entry : logs) {
                try {
                    JSONObject json = new JSONObject(entry.getMessage());
                    JSONObject message = json.getJSONObject("message");
                    String method = message.getString("method");
                    if ("Network.responseReceived".equals(method)) {
                        JSONObject params = message.getJSONObject("params");
                        JSONObject response = params.getJSONObject("response");
                        String messageUrl = response.getString("url");
                        int status = response.getInt("status");
                        System.out.println("Current URL: " + currentURL + "\tMessage URL: " + messageUrl + "\tStatus: " + status);
                        System.out.println("Message Headers: " + response.get("headers"));
                    }
                } catch (JSONException e) {
                    e.printStackTrace();
                }
            }
        } finally {
            if (driver != null) {
                driver.quit();
            }
        }
    }
}

---

Perfect! Let me get back to you in sometime.

---

this doesn't provide network call with a post method

---

Hey @Nevil Panchal, can you elaborate the error that you are getting.

---

Getting this error
invalid argument: log type 'performance' not found

Can anyone please look into it?

---

This error occurs while using RemoteWebdriver, So using a pure ChromeDriver object still produces logs of type "performance." 

Produces performance logs: 

desiredCapabilites = DesiredCapabilities.chrome() 

LoggingPreferences logPrefs = new LoggingPreferences() 

logPrefs.enable(LogType.PERFORMANCE, Level.ALL) desiredCapabilites.setCapability(CapabilityType.LOGGING_PREFS, logPrefs) 

driver = { new ChromeDriver(desiredCapabilites) }

---

@Abha Can you please copy the entire code here.

Thanks much!!

---

Hey, can you tell which webdriver you are using? Are you using a remote webdriver? And any specific url for which you need http respond code and performance log?

---

@Abha
Please provide a complete program for any sample page visit which is successfully generating  Performance logs hence i can verify the Status code with Selenium Finally.

Thanks.

---

Is there a way to get response body ?
The above mentioned code will provide only provide data that we see under headers tab in devtools

Answer 2

The Below code is working for the Status Code extraction using Selenium.

import java.util.Iterator;
import java.util.logging.Level;
import org.json.JSONException;
import org.json.JSONObject;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.logging.LogEntries;
import org.openqa.selenium.logging.LogEntry;
import org.openqa.selenium.logging.LogType;
import org.openqa.selenium.logging.LoggingPreferences;
import org.openqa.selenium.remote.DesiredCapabilities;

public class TestResponseCode {
    public static void main(String[] args) {
        // simple page (without many resources so that the output is
        // easy to understand
        String url = "https://www.bing.com/";
        downloadPage(url);
    }

    private static void downloadPage(String url) {
        System.setProperty("webdriver.chrome.driver", "D:\\Softwares\\Selenium\\Drivers\\ChromeDrivers\\75.0.3770.140\\chromedriver.exe");
        ChromeDriver driver = null;
        try {
            ChromeOptions options = new ChromeOptions();
            // add whatever extensions you need
            // for example I needed one of adding proxy, and one for blocking
            // images
            // options.addExtensions(new File(file, "proxy.zip"));
            // options.addExtensions(new File("extensions",
            // "Block-image_v1.1.crx"));
            DesiredCapabilities cap = DesiredCapabilities.chrome();
            cap.setCapability(ChromeOptions.CAPABILITY, options);
            // set performance logger
            // this sends Network.enable to chromedriver
            /*
             * LoggingPreferences logPrefs = new LoggingPreferences();
             * logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
             * cap.setCapability(CapabilityType.LOGGING_PREFS, logPrefs);
             */

            LoggingPreferences logPrefs = new LoggingPreferences();
            logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
            options.setCapability("goog:loggingPrefs", logPrefs);
            driver = new ChromeDriver(options);
            // navigate to the page
            System.out.println("Navigate to " + url);
            driver.navigate().to(url);
            // and capture the last recorded url (it may be a redirect, or the
            // original url)
            String currentURL = driver.getCurrentUrl();
            LogEntries logs = driver.manage().logs().get("performance");
            int status = -1;
            System.out.println("\\nList of log entries:\\n");
            for (Iterator<LogEntry> it = logs.iterator(); it.hasNext();) {
                LogEntry entry = it.next();
                try {
                    JSONObject json = new JSONObject(entry.getMessage());
                    System.out.println(json.toString());
                    JSONObject message = json.getJSONObject("message");
                    String method = message.getString("method");
                    if (method != null && "Network.responseReceived".equals(method)) {
                        JSONObject params = message.getJSONObject("params");
                        JSONObject response = params.getJSONObject("response");
                        String messageUrl = response.getString("url");
                        if (currentURL.equals(messageUrl)) {
                            status = response.getInt("status");
                            System.out.println("---------- bingo !!!!!!!!!!!!!! returned response for " + messageUrl
                                    + ": " + status);
                            System.out.println("---------- bingo !!!!!!!!!!!!!! headers: " + response.get("headers"));
                        }
                    }
                } catch (JSONException e) {
                    // TODO Auto-generated catch block
                    e.printStackTrace();
                }
            }
            System.out.println("\nstatus code: " + status);
        } finally {
            if (driver != null) {
                driver.quit();
            }
        }
    }
}

Comments

Hey Rvemula, thanks for sharing the code. It would really help.

---

You are a killer!!! Thanks!

---

I am getting an error at the performance

logPrefs.enable(LogType.PERFORMANCE, Level.ALL);

---

Hey, could you please post the error message that you are getting.

---

Good grief... whoever says Java is nice, get outta here... such verbosity, but I guess there is no other way to make this work.. Thanks !!!!!!