Force Python Scrapy not to encode URL

0 votes

There are some URLs with [] in it like

http://www.website.com/CN.html?value_ids[]=33&value_ids[]=5007

But when I try scraping this URL with Scrapy, it makes Request to this URL

http://www.website.com/CN.html?value_ids%5B%5D=33&value_ids%5B%5D=5007

How can I force scrapy to not to urlenccode my URLs?

Jul 24 in Python by ana1504.k
• 7,870 points
39 views

1 answer to this question.

0 votes

When creating a Request object scrapy applies some url encoding methods. To revert these you can utilize a custom middleware and change the url to your needs.

You could use a Downloader Middleware like this:

class MyCustomDownloaderMiddleware(object):

    def process_request(self, request, spider):
        request._url = request.url.replace("%5B", "[", 2)
        request._url = request.url.replace("%5D", "]", 2)
answered Jul 24 by SDeb
• 13,160 points

Related Questions In Python

0 votes
1 answer

How to write "not equal" in python?

The keywords is and is not are ...READ MORE

answered Sep 17, 2018 in Python by SayantiniDeb
• 200 points
111 views
0 votes
2 answers
0 votes
1 answer

How to use not equal operator in python

Use !=. See comparison operators. For comparing object identities, ...READ MORE

answered Dec 20, 2018 in Python by charlie_brown
• 7,720 points
112 views
0 votes
1 answer

Not able to take input in python

You are using python 2 in which ...READ MORE

answered Jan 31 in Python by Omkar
• 67,620 points
52 views
0 votes
1 answer

Section postgresql not found in the database.ini file

Python doesn't know what $FILEDIR is. Try an absolute path ...READ MORE

answered Oct 3, 2018 in Python by Priyaj
• 56,900 points
193 views
0 votes
1 answer

Iterating over dictionaries using 'for' loops

key is just a variable name. for key ...READ MORE

answered Oct 8, 2018 in Python by SDeb
• 13,160 points
51 views
0 votes
1 answer

Conflicting dependencies of pypyodbc and blpapi

I figured out that pypyodbc only works ...READ MORE

answered Oct 9, 2018 in Python by Priyaj
• 56,900 points
93 views
0 votes
1 answer

How to get travis to fail if tests do not have enough coverage for python?

if you add the --fail-under switch to ...READ MORE

answered May 10 in Python by SDeb
• 13,160 points
57 views
0 votes
1 answer

How to check the version of Python?

you can check the version of python ...READ MORE

answered Sep 25, 2018 in Python by SDeb
• 13,160 points
57 views