last day (14 days later) » 

7:12 AM
0
A: How should I modify to parse Google news search article title & preview & URL?

F. KleinHow to select the right elements (using chrome) First step: disable javascript in you browser (for example using a add on like uMatrix for convenience), so you see the same result as jsoup. Now right click on a element and choose inspect or open up the dev tools with Ctrl+Shift+I. When you hove...

 
many red lines :.referrer("https://www.google.com/").get(); select text() attr("href"); I use ctrl shift i to import all libraries
 
Depending on your IDE you might need write it in one line (here it is split for readability). Try: doc = Jsoup.connect(searchUrl).userAgent(userAgent).data("q", searchTerm).data("tbm", "nws").data("start", "" + i).method(Method.GET).referrer("google.com/").get().‌​get());
 
I m using netbean 8.0
 
Needed imports: import org.jsoup.Jsoup; import org.jsoup.Connection.Method; import org.jsoup.nodes.Document; import org.jsoup.nodes.Element;
 
already imported .still red
Even in one line still red
 
7:12 AM
Have you tried it with the one liner posted in my first comment? Compare the imports, to make sure, you have the right ones.
post you code here.
including imports
 
package javaapplication1;

import java.io.IOException;
import java.io.UnsupportedEncodingException;
import javax.lang.model.element.Element;
import org.jsoup.Connection.Method;
import org.jsoup.Jsoup;
import org.w3c.dom.Document;

import org.jsoup.nodes.*;

/**
*
* @author s1106630
*/
public class JavaApplication1 {

/**
* @param args the command line arguments
*/
public static void main(String[] args) throws UnsupportedEncodingException, IOException {

String userAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36";
 
Wrong document
import org.jsoup.nodes.Document; not import org.w3c.dom.Document;
Wrong Element
import org.jsoup.nodes.Element; not import javax.lang.model.element.Element;
 
omg
yes
becoz i just use the short-cut key to import the libraries
one red error
"Picked up JAVA_TOOL_OPTIONS: -Djava.vendor="New Oracle""
is it fine?
 
that happens easily, hence my comment to compare to the needed imports.
I haven't seen this error yet, but I'm also using another IDE.
 
String userAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36"
one question
 
7:19 AM
But is it running now?
 
is it real network setup
wt if i havnt install chrome
can it parse the content
yes i works
thanks bro
 
For development reasons, I would recommend it (see chrome portable from portableapps for example), but the user agent is just telling the server for what kind of browser the content should be provided, for example smartphone contain something like Android in the user agent so the mobile version gets provided. Some pages will even send a 404 if you haven't set a user agent to avoid (poorly written) bots.
Using the developer tools and monitoring the network tab while sending the request in the browser is also immensely useful: you see which header fields were send and for stable crawlers, you should do it as the website would do it, so acquire needed cookies, etc.
 
in simple words , setting user agent is used for telling the server what kind of browser the content should be provided . Hence ,the content will be best fit to the browser .If i set the user agent is Chrome , the content to see in smartphone would be unfit or 404 .Is it ur point ?
Hence , it is better to set useragent to deal with the network case .But ,without setting user agent still work logically
is it correct?
 
7:35 AM
My message here is: always set the user agent and in most cases (for crawling) use a user agent for a normal browser not a mobile one.
"still work logically": as I said, some pages might return a 404 or other messages/redirects, if there is no user agent. The user agent is send as a header field, so a page can directly see if it is missing and react accordingly.
 
7:50 AM
Rly one last question . For your code , if i would like to get page 1 to page 10 content , should i make change in this int numberOfResultpages = 2; -->int numberOfResultpages = 9;?
 

last day (14 days later) »