Force Python Scrapy not to encode URL

0 votes

There are some URLs with [] in it like

http://www.website.com/CN.html?value_ids[]=33&value_ids[]=5007

But when I try scraping this URL with Scrapy, it makes Request to this URL

http://www.website.com/CN.html?value_ids%5B%5D=33&value_ids%5B%5D=5007

How can I force scrapy to not to urlenccode my URLs?

Jul 24, 2019 in Python by ana1504.k
• 7,910 points
1,398 views

1 answer to this question.

0 votes

When creating a Request object scrapy applies some url encoding methods. To revert these you can utilize a custom middleware and change the url to your needs.

You could use a Downloader Middleware like this:

class MyCustomDownloaderMiddleware(object):

    def process_request(self, request, spider):
        request._url = request.url.replace("%5B", "[", 2)
        request._url = request.url.replace("%5D", "]", 2)
answered Jul 24, 2019 by SDeb
• 13,300 points

Related Questions In Python

0 votes
1 answer

How to write "not equal" in python?

The keywords is and is not are ...READ MORE

answered Sep 18, 2018 in Python by SayantiniDeb
• 200 points
2,365 views
+1 vote
3 answers
+1 vote
7 answers

How to use not equal operator in python

To check if operands are not equal ...READ MORE

answered Nov 30, 2021 in Python by Charry
370,629 views
0 votes
1 answer

Not able to take input in python

You are using python 2 in which ...READ MORE

answered Jan 31, 2019 in Python by Omkar
• 69,230 points
3,571 views
0 votes
1 answer

Section postgresql not found in the database.ini file

Python doesn't know what $FILEDIR is. Try an absolute path ...READ MORE

answered Oct 3, 2018 in Python by Priyaj
• 58,100 points
3,567 views
0 votes
1 answer

Iterating over dictionaries using 'for' loops

key is just a variable name. for key ...READ MORE

answered Oct 8, 2018 in Python by SDeb
• 13,300 points
955 views
0 votes
1 answer

Conflicting dependencies of pypyodbc and blpapi

I figured out that pypyodbc only works ...READ MORE

answered Oct 9, 2018 in Python by Priyaj
• 58,100 points
701 views
0 votes
1 answer

How to get travis to fail if tests do not have enough coverage for python?

if you add the --fail-under switch to ...READ MORE

answered May 10, 2019 in Python by SDeb
• 13,300 points
1,080 views
0 votes
1 answer

How to check the version of Python?

you can check the version of python ...READ MORE

answered Sep 25, 2018 in Python by SDeb
• 13,300 points
752 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP