Python web scraping without browser

Question

I am trying to webs scrape a lot of pages. Because the browser is opening the page, it is taking too much time. Is there a way I can scrape the page without opening a browser?

Answer 1

Yes, you can scape the web without a browser. You can use the request library to get the html data and then use BeautifulSoup to find elements or extract data. To get html data without opening the browser, you can use the following code:

url = "<Enter the page url here>"
page=requests.get(url)
soup = BeautifulSoup(page.text, 'html.parser')