Getting Started with Java Web Scraping: Basic Jsoup Usage and Two Download Examples
Introduction
At the moment, this only covers very basic scraping for articles and images. Anything beyond that is outside the scope here.
Maven dependencies
These are the JARs used in the examples below:
<dependencies>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.2</version>
</dependency>
<dependency>
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<version>1.10.1</version>
</dependency>
<dependency>
<groupId>com.cloudhopper</groupId>
<artifactId>ch-commons-io</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.7</version>
</dependency>
</dependencies>
A few basic ways to use Jsoup
Here are several entry-level examples, plus two test cases that can be used directly.
Parsing a URL and reading the page title
@Test
public void testUrl() throws Exception{
//解析url地址，第一个参数为访问url，第二个是访问时候超时时间
Document document=Jsoup.parse(new URL("http://99887766554433221100.cn/"),3000);
//使用标题选择器
String title=document.getElementsByTag("title").first().text();
System.out.println("titile--->"+title);
}
This example fetches a page with Jsoup.parse, using a URL and a timeout value, then pulls the content of the <title> tag.
Selecting elements from a local HTML file
@Test
public void test() throws Exception{
//id
Document document=Jsoup.parse(new File("D:\\qianduan\\1\\index.html"),"utf-8");
Element e_id=document.getElementById("xiaonanlashi");
System.out.println("e_id---->"+e_id);
System.out.println("e_id.test()---->"+e_id.text());
//获取第一个span
Element e_span_first=document.getElementsByTag("span").first();
System.out.println("e_span_first---->"+e_span_first);
System.out.println("e_span_first.test()---->"+e_span_first.text());
//类元素
Element e_class_first=document.getElementsByClass("class1").first();
System.out.println("e_class_first---->"+e_class_first);
System.out.println("e_class_first.test()---->"+e_class_first.text());
//属性
Element e_shuxing=document.getElementsByAttribute("abc").first();
System.out.println("e_shuxing---->"+e_shuxing);
System.out.println("e_shuxing.test()---->"+e_shuxing.text());
//根据属性和属性的值
Element e_shuxing_VALUE=document.getElementsByAttributeValue("href","http://99887766554433221100.cn/").first();
System.out.println("e_shuxing_VALUE---->"+e_shuxing_VALUE);
System.out.println("e_shuxing_VALUE.test()---->"+e_shuxing_VALUE.text());
}
This demonstrates several common selection methods:
- find an element by
id - get the first tag of a given type, such as
span - select by
class - select by attribute name
- select by both attribute name and attribute value
Using CSS selectors
@Test
public void xiaonan() throws Exception{
Document document=Jsoup.parse(new File("D:\\qianduan\\1\\index.html"),"utf-8");
//Elements es=document.select("span");
//for (Element e:es){
// System.out.println("span--->"+ e.text());
//}
//用id选择器必须加#号
//Elements es=document.select("#test");
//for (Element e:es){
// System.out.println("id--->"+ e.text());
//}
//用class选择器必须加.
Element e=document.select(".class1").first();
System.out.println("class--->"+e.text());
System.out.println(new Date());
}
The key point here is the selector syntax:
- use
#for anid - use
.for aclass
Two practical scraping examples
Both examples below are complete enough to run as-is, and both focus on image downloading.
Example 1
This one loops through search result pages, extracts image addresses, and saves them to a local folder.
One issue noted in the code is that a normal URLConnection request can fail with 403, because the server blocks Java-based requests. The workaround used here is to set a User-Agent through HttpURLConnection.
//链接分析
//https://wall.alphacoders.com/search.php?search=landscape&lang=Chinese
//https://wall.alphacoders.com/search.php?search=landscape&lang=Chinese&page=1
//Element clss_ss=doc.select(".boxgrid>a>picture>img").first();
System.out.println("开始时间为-->"+new Date());
int id=1;
for (int i=1;i<=999;i++){
String url="https://wall.alphacoders.com/search.php?search=landscape&lang=Chinese&page=";
url+=i;
Document doc=Jsoup.parse(new URL(url),2000);
Elements imgs=doc.select(".boxgrid>a>picture>img");
for (Element img:imgs){
String img_url=img.attr("src");
URL target=new URL(img_url);
//URLConnection urlConnection=target.openConnection();
//InputStream inputStream=urlConnection.getInputStream();
//Exception in thread "main" java.io.IOException: Server returned HTTP response code: 403 for URL:
//抓取失败，服务器禁止Java程序，使用下面三行代码就可以了
HttpURLConnection httpConnection = (HttpURLConnection) new URL(img_url).openConnection();
httpConnection.setRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt)");
InputStream inputStream=httpConnection.getInputStream();
System.out.println("已经下"+id+"张图片了");
id++;
OutputStream outputStream=new FileOutputStream("D:\\imgs\\"+id+".png");
int flag=0;
while ((flag=inputStream.read())!=-1){
outputStream.write(flag);
}
outputStream.close();
inputStream.close();
}
}
System.out.println("结束时间为-->"+new Date());
A few details worth noting from the code itself:
- page numbers are appended to the base search URL
- images are selected with
.boxgrid>a>picture>img - each image is saved under
D:\imgs\ - progress is printed after each download
- start and end times are printed with
new Date()
Example 2
The second example follows the same overall pattern: inspect the target URL structure, determine the selector, then loop through the pages and save the images locally.
//链接分析
//https://www.photocome.com/search?q=%E9%A3%8E%E6%99%AF&page=1
//https://www.photocome.com/search?q=%E9%A3%8E%E6%99%AF&page=2
//初步分析
//Document doc=Jsoup.parse(new URL("https://www.photocome.com/search?q=%E9%A3%8E%E6%99%AF&page=1"),5000);
//Elements clss_ss=doc.select("._2gLVL _33Mfr");
//Element clss_ss=doc.select("._33Mfr>a>img").first();
//System.out.println(clss_ss.attr("src"));
//alifei01.cfp.cn/creative/vcg/veer/612/veer-375537960.jpg
System.out.println("开始时间为-->"+new Date());
int id=1;
for (int i=1;i<=78;i++){
String url="https://www.photocome.com/search?q=%E9%A3%8E%E6%99%AF&page=";
url+=i;
Document doc=Jsoup.parse(new URL(url),5000);
Elements imgs=doc.select("._33Mfr>a>img");
for (Element img:imgs){
String img_url=img.attr("src");
img_url="https:"+img_url;
//System.out.println(img_url);
URL target=new URL(img_url);
URLConnection urlConnection=target.openConnection();
InputStream inputStream=urlConnection.getInputStream();
System.out.println("已经下"+id+"张图片了");
id++;
OutputStream outputStream=new FileOutputStream("D:\\img\\"+id+".png");
int flag=0;
while ((flag=inputStream.read())!=-1){
outputStream.write(flag);
}
outputStream.close();
inputStream.close();
}
}
System.out.println("结束时间为-->"+new Date());
Compared with the first example, one extra step appears here: the extracted src is not a complete URL at first, so https: is added before downloading.
The selector used is:
._33Mfr>a>img
Images are then written to:
D:\img\
The overall approach in both examples is simple:
- analyze the page URL pattern
- inspect the page structure to find the right selector
- pull the image link from each matched element
- open an input stream from the image URL
- write the file out with a
FileOutputStream
If all you need is straightforward article or image scraping with Java, this is enough to get started.