python - web scraping of web that blocks access for scripts -

-

is a website that I used to use a Pyro script (urllib). It seems that the website is now blocking my requests and whenever I request a web page using a script I get a html with some JS but without the normal data. Accessing the website from your browser works well. I tried to change the 'user-agent' to fit one who used my browser but it was not used. I saw that a strange behavior is that after accessing a page from my browser, I can access it from the script too.

So my questions are:

  1. How can the server detect it is not a browser (can I change user-agent)?
  2. What kind of access can be the reason for the strange behavior of a web page after the web page has loaded? Is it caching? If yes, where is caching?
  3. Any idea how to move forward? (I do not have a very elegant solution to open every page before loading my browser, but it takes a lot of time)

Thanks!

To go without too many details, it seems like the site to include JavaScript loaders Has been updated to urlib can not process Javascript, so it is unable to continue (pure speculation here)

A site preventing a scraper from reaching it , Which includes updating some javascript sets or cookies, or anyone This is the first test to modify the session in a similar manner. It is entirely up to the site, so you have to check it manually.

The general solution is to use a javascript-linked scraper like selenium , which is actually installed locally firefox , chrome Click or IE to open the page, and simulate the items you can also use PhantomJS to take action on the downloaded page. .

There are a lot of posts about this, but here's one that can give you a starting point:


Comments

Post a Comment

Popular posts from this blog

apache - 504 Gateway Time-out The server didn't respond in time. How to fix it? -

-
Using a form submission on an embedded iframe , the customer downloads a compressed log file Requested. The request was sent to the server, which is the compressed log files, perform some database operations and returns a compressed file. After just 2 minutes, 504 gateway time-out server did not respond timely message on browser net panel How to fix this error? The log files were taking a long time to compress, and timeout was set to 2min The error was fixed by extending the file file: # # timeout: The number of seconds before getting the time out. # # # Timeout 120 timeout 600
Read more

c# - .net WebSocket: CloseOutputAsync vs CloseAsync -

-
We have a working ASP.NET Web API REST service, which is one of the methods of our controller, HTTPTTEX. .) Socket handler code looks something like this ... Public async task socket handler (AspNetWebSocketContext context) {_webSocket = context.WebSocket; ... while (! Cts.IsCancellationRequested) {WebSocketReceiveResult Results = _webSocket.ReceiveAsync (Input Segment, cts.Token) .Result; WebSocketsStateCollusocketState = _webSocket.State; If (result.MessageType == WebSocketMessageType.Close || currentSocketState == WebSocketState.CloseReceived) {// What should I use. CloseAysnc () or. CloseOutputAsync ()? _webSocket.CloseOutputAsync (WebSocketCloseStatus.NormalClosure, "Client Requested", cts.Token). Wait (); } If (currentSocketState == WebSocketState.Open) {...}}} .What is the difference between .CooseAsync () and CloseOutputAysnc ()? I tried both of them and they both seemed to work fine but some difference should be the same they both describe very similar to...
Read more

c++ - How to properly scale qgroupbox title with stylesheet for high resolution display? -

-
I am trying to apply a stylesheet for QGroupBox (QT4.8), which works on the normal screen ( 96 dpi) high resolution screen (Yoga 2 Pro, 3200x1800, 275 dpi, windows 8.1). The following style looks good on the 275 dpi screen, but the top margins on a regular 96 dpi screen are far too big. QGroupBox {border: 1px solid red; Range radius: 7px; Margin-top: 12x; } QGroupBox :: Title {subcontrol-origin: margin; Subcontrol-position: left above; Padding-left: 10px; Padding-right: 10px; } Changing the top-margin has an effect, but I did not get a proper setting which works on both screens. If I shorten the value, the content of the group box overlaps with the title on 275 dpi screen. I was also playing with units "East", "PX", "MX", "PT". Reading the document I would have guessed, "2 X" was the correct solution, which should be scaled with different screen resolutions. Without the stylesheet, the groupbox works well on both screens. ...
Read more