Web-scraping with beautifulsoup in different siblings

0 votes

I've tried different ways to scrape Answer1 and Answer2 from a website through BeautifulSoup, urllib and Selenium, but without success. Here's the simplified version:

<div class="div1">
  <p class="p1"></p>
  <p class="p2">
    <span>Question1</span>
    <strong>Answer1</strong>
    <br>
    <span>Question2</span>
    <strong>Answer2</strong>
    <br>

In selenium, I try to find Question1, then go to its parent and scrape Answer1. Below is the code I use, although it's not correct.

browser.find_elements_by_xpath("//span[contains(text(), 'Question1')]/parent::p/following::strong")

I believe bs is more efficient than selenium in this case. How would you do this in bs? Thanks!

Mar 27, 2018 in Selenium by code_ninja
• 5,900 points
134 views

1 answer to this question.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
0 votes

The tags are not closed in your HTML. Check the code below...i've closed the <p> and <div> tags:

from bs4 import BeautifulSoup as BS
html = """
<div class="div1">
  <p class="p1"></p>
  <p class="p2">
    <span>Question1</span>
    <strong>Answer1</strong>
    <br>
    <span>Question2</span>
    <strong>Answer2</strong>
    <br>
    </p>
</div>
"""
soup = BS(html,'lxml')
QA = {x.text:y.text for x,y in zip(soup.select('span'),soup.select('strong'))}
print(QA)
answered Mar 27, 2018 by nsv999
• 5,050 points

Related Questions In Selenium

0 votes
1 answer
0 votes
1 answer

XPath for the elements with no ID or Name in Selenium Webdriver

Use like this or similar to this: //div[@id="top-level-menu-item-3"]/div[@class="filter-label"] //div[@id="top-level-menu1"] ...READ MORE

answered Apr 17, 2018 in Selenium by Meci Matt
• 9,400 points
841 views
0 votes
1 answer

How to Create Object Repository in Selenium Web Driver?

Try POM Model (Page object model). Refer ...READ MORE

answered Apr 17, 2018 in Selenium by adam
421 views
0 votes
1 answer

Searching for the element with &nbsp; symbol in selenium

Use Below: //div[@class='Tips' and text()='\u00a0'] READ MORE

answered Apr 20, 2018 in Selenium by Vardy
• 2,280 points
1,050 views
0 votes
1 answer

Finding WebDriver element with Class Name in java

The better way to handle this element ...READ MORE

answered Apr 10, 2018 in Selenium by nsv999
• 5,050 points
131 views
0 votes
1 answer
0 votes
1 answer

How to perform web scraping with python?

Hey, there are various libraries used in ...READ MORE

answered Apr 20, 2018 in Python by aayushi
• 750 points
181 views
0 votes
1 answer

Geo-location microphone camera pop up

To Allow or Block the notification, access using Selenium and you have to ...READ MORE

answered May 11, 2018 in Selenium by Samarpit
• 5,130 points
519 views
+2 votes
2 answers

How can I press ENTER key with the execute_script in selenium python?

The below code containing Keys.ENTER might just ...READ MORE

answered Mar 27, 2018 in Selenium by nsv999
• 5,050 points
4,431 views

© 2018 Brain4ce Education Solutions Pvt. Ltd. All rights Reserved.
"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.