630 lines
20 KiB
HTML
630 lines
20 KiB
HTML
|
|
<!doctype html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
|
<html>
|
|
<head><META http-equiv="Content-Type" content="text/html; charset=utf-8">
|
|
<title>Start HTTP Crawler (STRHTTPCRL)</title>
|
|
<link rel="stylesheet" type="text/css" href="../rzahg/ic.css">
|
|
</head>
|
|
<body bgcolor="white">
|
|
<script language="Javascript" src="../rzahg/synch.js" type="text/javascript"></script>
|
|
<a name="STRHTTPCRL.Top_Of_Page"></a>
|
|
<h2>Start HTTP Crawler (STRHTTPCRL)</h2>
|
|
<table width="100%">
|
|
<tr>
|
|
<td valign="top" align="left"><b>Where allowed to run: </b>All environments (*ALL)<br>
|
|
<b>Threadsafe: </b>No
|
|
</td>
|
|
<td valign="top" align="right">
|
|
<a href="#STRHTTPCRL.PARAMETERS.TABLE">Parameters</a><br>
|
|
<a href="#STRHTTPCRL.COMMAND.EXAMPLES">Examples</a><br>
|
|
<a href="#STRHTTPCRL.ERROR.MESSAGES">Error messages</a></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<div> <a name="STRHTTPCRL"></a>
|
|
<p>The Start HTTP Crawling (STRHTTPCRL) command allows you to create or append to a document list by crawling remote web sites, downloading files found, and saving the path names in the document list specified.
|
|
</p>
|
|
<p>To create a document list, specify *CRTDOCL for the <b>Option (OPTION)</b> parameter.
|
|
</p>
|
|
<p>To update a document list, specify *UPDDOCL for the OPTION parameter.
|
|
</p>
|
|
</div>
|
|
<table width="100%">
|
|
<tr><td align="right"><a href="#STRHTTPCRL.Top_Of_Page">Top</a></td></tr>
|
|
</table>
|
|
<hr size="2" width="100%">
|
|
|
|
<div>
|
|
<h3><a name="STRHTTPCRL.PARAMETERS.TABLE">Parameters</a></h3>
|
|
<table border="1" cellpadding="4" cellspacing="0">
|
|
<!-- col1="10" col2="15" col3="30" col4="10" -->
|
|
<tr>
|
|
<th bgcolor="aqua" valign="bottom" align="left">Keyword</th>
|
|
<th bgcolor="aqua" valign="bottom" align="left">Description</th>
|
|
<th bgcolor="aqua" valign="bottom" align="left">Choices</th>
|
|
<th bgcolor="aqua" valign="bottom" align="left">Notes</th>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top"><a href="#STRHTTPCRL.OPTION"><b>OPTION</b></a></td>
|
|
<td valign="top">Option</td>
|
|
<td valign="top">*CRTDOCL, *UPDDOCL</td>
|
|
<td valign="top">Required, Positional 1</td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top"><a href="#STRHTTPCRL.METHOD"><b>METHOD</b></a></td>
|
|
<td valign="top">Crawling method</td>
|
|
<td valign="top">*OBJECTS, <b><u>*DETAIL</u></b></td>
|
|
<td valign="top">Optional</td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top" rowspan="3"><a href="#STRHTTPCRL.OBJECTS"><b>OBJECTS</b></a></td>
|
|
<td valign="top">URL and options objects</td>
|
|
<td valign="top"><i>Element list</i></td>
|
|
<td valign="top" rowspan="3">Optional</td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top">Element 1: URL object</td>
|
|
<td valign="top">
|
|
<i>Character value</i></td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top">Element 2: Options object</td>
|
|
<td valign="top">
|
|
<i>Character value</i></td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top"><a href="#STRHTTPCRL.DOCLIST"><b>DOCLIST</b></a></td>
|
|
<td valign="top">Document list file</td>
|
|
<td valign="top"><i>Path name</i></td>
|
|
<td valign="top">Optional</td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top"><a href="#STRHTTPCRL.DOCDIR"><b>DOCDIR</b></a></td>
|
|
<td valign="top">Document storage directory</td>
|
|
<td valign="top"><i>Path name</i>, <b><u>'/QIBM/USERDATA/HTTPSVR/INDEX/DOC'</u></b></td>
|
|
<td valign="top">Optional</td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top"><a href="#STRHTTPCRL.LANG"><b>LANG</b></a></td>
|
|
<td valign="top">Language of documents</td>
|
|
<td valign="top">*ARABIC, *BALTIC, *CENTEUROPE, *CYRILLIC, *ESTONIAN, *GREEK, *HEBREW, *JAPANESE, *KOREAN, *SIMPCHINESE, *TRADCHINESE, *THAI, *TURKISH, <b><u>*WESTERN</u></b></td>
|
|
<td valign="top">Optional</td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top"><a href="#STRHTTPCRL.URL"><b>URL</b></a></td>
|
|
<td valign="top">URL</td>
|
|
<td valign="top"><i>Character value</i></td>
|
|
<td valign="top">Optional</td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top"><a href="#STRHTTPCRL.URLFTR"><b>URLFTR</b></a></td>
|
|
<td valign="top">URL filter</td>
|
|
<td valign="top"><i>Character value</i>, <b><u>*NONE</u></b></td>
|
|
<td valign="top">Optional</td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top"><a href="#STRHTTPCRL.MAXDEPTH"><b>MAXDEPTH</b></a></td>
|
|
<td valign="top">Maximum crawling depth</td>
|
|
<td valign="top">0-100, <b><u>3</u></b>, *NOMAX</td>
|
|
<td valign="top">Optional</td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top"><a href="#STRHTTPCRL.ENBROBOT"><b>ENBROBOT</b></a></td>
|
|
<td valign="top">Enable robots</td>
|
|
<td valign="top"><b><u>*YES</u></b>, *NO</td>
|
|
<td valign="top">Optional</td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top"><a href="#STRHTTPCRL.PRXSVR"><b>PRXSVR</b></a></td>
|
|
<td valign="top">Proxy server for HTTP</td>
|
|
<td valign="top"><i>Character value</i>, <b><u>*NONE</u></b></td>
|
|
<td valign="top">Optional</td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top"><a href="#STRHTTPCRL.PRXPORT"><b>PRXPORT</b></a></td>
|
|
<td valign="top">Proxy port for HTTP</td>
|
|
<td valign="top">1-65535</td>
|
|
<td valign="top">Optional</td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top"><a href="#STRHTTPCRL.PRXSVRSSL"><b>PRXSVRSSL</b></a></td>
|
|
<td valign="top">Proxy server for HTTPS</td>
|
|
<td valign="top"><i>Character value</i>, <b><u>*NONE</u></b></td>
|
|
<td valign="top">Optional</td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top"><a href="#STRHTTPCRL.PRXPORTSSL"><b>PRXPORTSSL</b></a></td>
|
|
<td valign="top">Proxy port for HTTPS</td>
|
|
<td valign="top">1-65535</td>
|
|
<td valign="top">Optional</td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top"><a href="#STRHTTPCRL.MAXSIZE"><b>MAXSIZE</b></a></td>
|
|
<td valign="top">Maximum file size</td>
|
|
<td valign="top">1-6000, <b><u>1000</u></b></td>
|
|
<td valign="top">Optional</td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top"><a href="#STRHTTPCRL.MAXSTGSIZE"><b>MAXSTGSIZE</b></a></td>
|
|
<td valign="top">Maximum storage size</td>
|
|
<td valign="top">1-65535, <b><u>100</u></b>, *NOMAX</td>
|
|
<td valign="top">Optional</td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top"><a href="#STRHTTPCRL.MAXTHD"><b>MAXTHD</b></a></td>
|
|
<td valign="top">Maximum threads</td>
|
|
<td valign="top">1-50, <b><u>20</u></b></td>
|
|
<td valign="top">Optional</td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top" rowspan="3"><a href="#STRHTTPCRL.MAXRUNTIME"><b>MAXRUNTIME</b></a></td>
|
|
<td valign="top">Maximum run time</td>
|
|
<td valign="top">Single values: *NOMAX<br>Other values: <i>Element list</i></td>
|
|
<td valign="top" rowspan="3">Optional</td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top">Element 1: Hours</td>
|
|
<td valign="top">
|
|
0-1000, <b><u>2</u></b></td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top">Element 2: Minutes</td>
|
|
<td valign="top">
|
|
0-59, <b><u>0</u></b></td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top"><a href="#STRHTTPCRL.LOGFILE"><b>LOGFILE</b></a></td>
|
|
<td valign="top">Logging file</td>
|
|
<td valign="top"><i>Path name</i>, <b><u>*NONE</u></b></td>
|
|
<td valign="top">Optional</td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top"><a href="#STRHTTPCRL.CLRLOG"><b>CLRLOG</b></a></td>
|
|
<td valign="top">Clear logging file</td>
|
|
<td valign="top"><b><u>*YES</u></b>, *NO</td>
|
|
<td valign="top">Optional</td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top"><a href="#STRHTTPCRL.VLDL"><b>VLDL</b></a></td>
|
|
<td valign="top">Validation list</td>
|
|
<td valign="top"><i>Name</i>, <b><u>*NONE</u></b></td>
|
|
<td valign="top">Optional</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table width="100%">
|
|
<tr><td align="right"><a href="#STRHTTPCRL.Top_Of_Page">Top</a></td></tr>
|
|
</table>
|
|
</div>
|
|
<div> <a name="STRHTTPCRL.OPTION"></a>
|
|
<h3>Option (OPTION)</h3>
|
|
<p>Specifies the document list task to perform.
|
|
</p>
|
|
<p>This is a required parameter.
|
|
</p>
|
|
<dl>
|
|
<dt><b>*CRTDOCL</b></dt>
|
|
<dd>Create a document list. If the file already exists, it will be replaced.
|
|
</dd>
|
|
<dt><b>*UPDDOCL</b></dt>
|
|
<dd>Append additional document paths to a document list.
|
|
</dd>
|
|
</dl>
|
|
</div>
|
|
<table width="100%">
|
|
<tr><td align="right"><a href="#STRHTTPCRL.Top_Of_Page">Top</a></td></tr>
|
|
</table>
|
|
<div> <a name="STRHTTPCRL.METHOD"></a>
|
|
<h3>Crawling method (METHOD)</h3>
|
|
<p>Specifies the crawling method to use.
|
|
</p>
|
|
<dl>
|
|
<dt><b><u>*DETAIL</u></b></dt>
|
|
<dd>Use specific values for crawling remote web sites such as the document storage directory, a URL, and a URL filter. These are the same values that are contained in a URL object and an options object.
|
|
</dd>
|
|
<dt><b>*OBJECTS</b></dt>
|
|
<dd>Use a URL object and an options object for crawling web sites. These objects contain specific values used in the crawling process.
|
|
</dd>
|
|
</dl>
|
|
</div>
|
|
<table width="100%">
|
|
<tr><td align="right"><a href="#STRHTTPCRL.Top_Of_Page">Top</a></td></tr>
|
|
</table>
|
|
<div> <a name="STRHTTPCRL.OBJECTS"></a>
|
|
<h3>URL and options objects (OBJECTS)</h3>
|
|
<p>Specifies the objects to use for crawling. Both must be specified. Use the Configure HTTP Search (CFGHTTPSCH) command to create the objects.
|
|
</p>
|
|
<p><b>Element 1: URL object</b>
|
|
</p>
|
|
<dl>
|
|
<dt><b><i>character-value</i></b></dt>
|
|
<dd>Specify the name of the URL object to use.
|
|
</dd>
|
|
</dl>
|
|
<p><b>Element 2: Options object</b>
|
|
</p>
|
|
<dl>
|
|
<dt><b><i>character-value</i></b></dt>
|
|
<dd>Specify the name of the options object to use.
|
|
</dd>
|
|
</dl>
|
|
</div>
|
|
<table width="100%">
|
|
<tr><td align="right"><a href="#STRHTTPCRL.Top_Of_Page">Top</a></td></tr>
|
|
</table>
|
|
<div> <a name="STRHTTPCRL.DOCLIST"></a>
|
|
<h3>Document list file (DOCLIST)</h3>
|
|
<p>Specifies the document list file to hold the path names of the documents found by crawling remote web sites.
|
|
</p>
|
|
<dl>
|
|
<dt><b><i>path-name</i></b></dt>
|
|
<dd>Specify the document list file path name.
|
|
</dd>
|
|
</dl>
|
|
</div>
|
|
<table width="100%">
|
|
<tr><td align="right"><a href="#STRHTTPCRL.Top_Of_Page">Top</a></td></tr>
|
|
</table>
|
|
<div> <a name="STRHTTPCRL.DOCDIR"></a>
|
|
<h3>Document storage directory (DOCDIR)</h3>
|
|
<p>Specifies the directory to use to store the documents that are downloaded.
|
|
</p>
|
|
<dl>
|
|
<dt><b><u>'/QIBM/USERDATA/HTTPSVR/INDEX/DOC'</u></b></dt>
|
|
<dd>This directory is used to store the downloaded documents.
|
|
</dd>
|
|
<dt><b><i>path-name</i></b></dt>
|
|
<dd>Specify the document storage directory path name.
|
|
</dd>
|
|
</dl>
|
|
</div>
|
|
<table width="100%">
|
|
<tr><td align="right"><a href="#STRHTTPCRL.Top_Of_Page">Top</a></td></tr>
|
|
</table>
|
|
<div> <a name="STRHTTPCRL.LANG"></a>
|
|
<h3>Language of documents (LANG)</h3>
|
|
<p>Specifies the language of the documents that are to be downloaded. These language choices are similar to the character sets or encodings that can be selected on a browser.
|
|
</p>
|
|
<dl>
|
|
<dt><b><u>*WESTERN</u></b></dt>
|
|
<dd>The documents are in a Western language such as English, Finnish, French, Spanish, or German.
|
|
</dd>
|
|
<dt><b>*ARABIC</b></dt>
|
|
<dd>The documents are in Arabic.
|
|
</dd>
|
|
<dt><b>*BALTIC</b></dt>
|
|
<dd>The documents are in a Baltic language such as Latvian or Lithuanian.
|
|
</dd>
|
|
<dt><b>*CENTEUROPE</b></dt>
|
|
<dd>The documents are in a Central European language such as Czech, Hungarian, Polish, Slovakian, or Slovenian.
|
|
</dd>
|
|
<dt><b>*CYRILLIC</b></dt>
|
|
<dd>The documents are in a Cyrillic language such as Russian, Ukranian, or Macedonian.
|
|
</dd>
|
|
<dt><b>*ESTONIAN</b></dt>
|
|
<dd>The documents are in Estonian.
|
|
</dd>
|
|
<dt><b>*GREEK</b></dt>
|
|
<dd>The documents are in Greek.
|
|
</dd>
|
|
<dt><b>*HEBREW</b></dt>
|
|
<dd>The documents are in Hebrew.
|
|
</dd>
|
|
<dt><b>*JAPANESE</b></dt>
|
|
<dd>The documents are in Japanese.
|
|
</dd>
|
|
<dt><b>*KOREAN</b></dt>
|
|
<dd>The documents are in Korean.
|
|
</dd>
|
|
<dt><b>*SIMPCHINESE</b></dt>
|
|
<dd>The documents are in Simplified Chinese.
|
|
</dd>
|
|
<dt><b>*TRADCHINESE</b></dt>
|
|
<dd>The documents are in Traditional Chinese.
|
|
</dd>
|
|
<dt><b>*THAI</b></dt>
|
|
<dd>The documents are in Thai.
|
|
</dd>
|
|
<dt><b>*TURKISH</b></dt>
|
|
<dd>The documents are in Turkish.
|
|
</dd>
|
|
</dl>
|
|
</div>
|
|
<table width="100%">
|
|
<tr><td align="right"><a href="#STRHTTPCRL.Top_Of_Page">Top</a></td></tr>
|
|
</table>
|
|
<div> <a name="STRHTTPCRL.URL"></a>
|
|
<h3>URL (URL)</h3>
|
|
<p>Specifies the name of the URL (Universal Resource Locator) to crawl.
|
|
</p>
|
|
<dl>
|
|
<dt><b><i>character-value</i></b></dt>
|
|
<dd>Specify the URL to crawl.
|
|
</dd>
|
|
</dl>
|
|
</div>
|
|
<table width="100%">
|
|
<tr><td align="right"><a href="#STRHTTPCRL.Top_Of_Page">Top</a></td></tr>
|
|
</table>
|
|
<div> <a name="STRHTTPCRL.URLFTR"></a>
|
|
<h3>URL filter (URLFTR)</h3>
|
|
<p>The domain filter to limit sites crawled to those within the specified domain.
|
|
</p>
|
|
<dl>
|
|
<dt><b><u>*NONE</u></b></dt>
|
|
<dd>No filtering will be done base on domain.
|
|
</dd>
|
|
<dt><b><i>character-value</i></b></dt>
|
|
<dd>Specify the domain filter to limit crawling.
|
|
</dd>
|
|
</dl>
|
|
</div>
|
|
<table width="100%">
|
|
<tr><td align="right"><a href="#STRHTTPCRL.Top_Of_Page">Top</a></td></tr>
|
|
</table>
|
|
<div> <a name="STRHTTPCRL.MAXDEPTH"></a>
|
|
<h3>Maximum crawling depth (MAXDEPTH)</h3>
|
|
<p>The maximum depth to crawl from the starting URL. Zero means to stop crawling at the starting URL site. Each additional layer refers to following referenced links within the current URL.
|
|
</p>
|
|
<dl>
|
|
<dt><b><u>3</u></b></dt>
|
|
<dd>Referenced links will be crawled three layers deep.
|
|
</dd>
|
|
<dt><b>*NOMAX</b></dt>
|
|
<dd>Referenced links will be crawled regardless of depth.
|
|
</dd>
|
|
<dt><b><i>0-100</i></b></dt>
|
|
<dd>Specify the maximum crawling depth.
|
|
</dd>
|
|
</dl>
|
|
</div>
|
|
<table width="100%">
|
|
<tr><td align="right"><a href="#STRHTTPCRL.Top_Of_Page">Top</a></td></tr>
|
|
</table>
|
|
<div> <a name="STRHTTPCRL.ENBROBOT"></a>
|
|
<h3>Enable robots (ENBROBOT)</h3>
|
|
<p>Specifies whether to enable support for robot exclusion. If you select to support robot exclusion, any site or pages that contain robot exclusion META tags or files will not be downloaded.
|
|
</p>
|
|
<dl>
|
|
<dt><b><u>*YES</u></b></dt>
|
|
<dd>Enable support for robot exclusion.
|
|
</dd>
|
|
<dt><b>*NO</b></dt>
|
|
<dd>Do not enable support for robot exclusion.
|
|
</dd>
|
|
</dl>
|
|
</div>
|
|
<table width="100%">
|
|
<tr><td align="right"><a href="#STRHTTPCRL.Top_Of_Page">Top</a></td></tr>
|
|
</table>
|
|
<div> <a name="STRHTTPCRL.PRXSVR"></a>
|
|
<h3>Proxy server for HTTP (PRXSVR)</h3>
|
|
<p>Specifies the HTTP proxy server to be used.
|
|
</p>
|
|
<dl>
|
|
<dt><b><u>*NONE</u></b></dt>
|
|
<dd>Do not use an HTTP proxy server.
|
|
</dd>
|
|
<dt><b><i>HTTP-proxy-server</i></b></dt>
|
|
<dd>Specify the name of the HTTP proxy server.
|
|
</dd>
|
|
</dl>
|
|
</div>
|
|
<table width="100%">
|
|
<tr><td align="right"><a href="#STRHTTPCRL.Top_Of_Page">Top</a></td></tr>
|
|
</table>
|
|
<div> <a name="STRHTTPCRL.PRXPORT"></a>
|
|
<h3>Proxy port for HTTP (PRXPORT)</h3>
|
|
<p>Specifies the HTTP proxy server port.
|
|
</p>
|
|
<dl>
|
|
<dt><b><i>1-65535</i>.</b></dt>
|
|
<dd>Specify the number of the HTTP proxy server port. This parameter is required if a proxy server name is specified for the <b>Proxy server for HTTP (PRXSVR)</b> parameter.
|
|
</dd>
|
|
</dl>
|
|
</div>
|
|
<table width="100%">
|
|
<tr><td align="right"><a href="#STRHTTPCRL.Top_Of_Page">Top</a></td></tr>
|
|
</table>
|
|
<div> <a name="STRHTTPCRL.PRXSVRSSL"></a>
|
|
<h3>Proxy server for HTTPS (PRXSVRSSL)</h3>
|
|
<p>Specifies the HTTPS proxy server for using SSL support.
|
|
</p>
|
|
<dl>
|
|
<dt><b><u>*NONE</u></b></dt>
|
|
<dd>Do not use an HTTPS proxy server.
|
|
</dd>
|
|
<dt><b><i>character-value</i></b></dt>
|
|
<dd>Specify the name of the HTTPS proxy server for SSL support.
|
|
</dd>
|
|
</dl>
|
|
</div>
|
|
<table width="100%">
|
|
<tr><td align="right"><a href="#STRHTTPCRL.Top_Of_Page">Top</a></td></tr>
|
|
</table>
|
|
<div> <a name="STRHTTPCRL.PRXPORTSSL"></a>
|
|
<h3>Proxy port for HTTPS (PRXPORTSSL)</h3>
|
|
<p>Specifies the HTTPS proxy server port for SSL support.
|
|
</p>
|
|
<dl>
|
|
<dt><b><i>1-65535</i></b></dt>
|
|
<dd>Specify the number of the HTTPS proxy server port for SSL support. This is required if an SSL proxy server is also specified. This parameter is required if a proxy server name is specified for the <b>Proxy server for HTTPS (PRXSVRSSL)</b> parameter.
|
|
</dd>
|
|
</dl>
|
|
</div>
|
|
<table width="100%">
|
|
<tr><td align="right"><a href="#STRHTTPCRL.Top_Of_Page">Top</a></td></tr>
|
|
</table>
|
|
<div> <a name="STRHTTPCRL.MAXSIZE"></a>
|
|
<h3>Maximum file size (MAXSIZE)</h3>
|
|
<p>Specifies the maximum file size, in kilobytes, to download.
|
|
</p>
|
|
<dl>
|
|
<dt><b><u>1000</u></b></dt>
|
|
<dd>Download files that are no greater than 1000 kilobytes.
|
|
</dd>
|
|
<dt><b>*NOMAX</b></dt>
|
|
<dd>Files will be downloaded regardless of size.
|
|
</dd>
|
|
<dt><b><i>1-6000</i>.</b></dt>
|
|
<dd>Specify the maximum file size to download, in kilobytes.
|
|
</dd>
|
|
</dl>
|
|
</div>
|
|
<table width="100%">
|
|
<tr><td align="right"><a href="#STRHTTPCRL.Top_Of_Page">Top</a></td></tr>
|
|
</table>
|
|
<div> <a name="STRHTTPCRL.MAXSTGSIZE"></a>
|
|
<h3>Maximum storage size (MAXSTGSIZE)</h3>
|
|
<p>Specifies the maximum storage size, in megabytes, to allocate for downloaded files. Crawling will end when this limit is reached.
|
|
</p>
|
|
<dl>
|
|
<dt><b><u>100</u></b></dt>
|
|
<dd>Up to 100 megabytes of storage will be used for downloaded files.
|
|
</dd>
|
|
<dt><b>*NOMAX</b></dt>
|
|
<dd>No maximum storage size for downloaded files.
|
|
</dd>
|
|
<dt><b><i>1-65535</i>.</b></dt>
|
|
<dd>Specify the maximum storage size, in megabytes, for downloaded files.
|
|
</dd>
|
|
</dl>
|
|
</div>
|
|
<table width="100%">
|
|
<tr><td align="right"><a href="#STRHTTPCRL.Top_Of_Page">Top</a></td></tr>
|
|
</table>
|
|
<div> <a name="STRHTTPCRL.MAXTHD"></a>
|
|
<h3>Maximum threads (MAXTHD)</h3>
|
|
<p>Specifies the maximum number of threads to start for crawling web sites. Set this value based on the system resources that are available.
|
|
</p>
|
|
<dl>
|
|
<dt><b><u>20</u></b></dt>
|
|
<dd>Start up to 20 threads for web crawling.
|
|
</dd>
|
|
<dt><b><i>1-50</i>.</b></dt>
|
|
<dd>Specify the maximum number of threads to start.
|
|
</dd>
|
|
</dl>
|
|
</div>
|
|
<table width="100%">
|
|
<tr><td align="right"><a href="#STRHTTPCRL.Top_Of_Page">Top</a></td></tr>
|
|
</table>
|
|
<div> <a name="STRHTTPCRL.MAXRUNTIME"></a>
|
|
<h3>Maximum run time (MAXRUNTIME)</h3>
|
|
<p>Specifies the maximum time for crawling to run, in hours and minutes.
|
|
</p>
|
|
<p><b>Single values</b>
|
|
</p>
|
|
<dl>
|
|
<dt><b>*NOMAX</b></dt>
|
|
<dd>Run the crawling session until it completes normally or is ended by using the ENDHTTPCRL (End HTTP Crawler) command.
|
|
</dd>
|
|
</dl>
|
|
<p><b>Element 1: Hours</b>
|
|
</p>
|
|
<dl>
|
|
<dt><b><u>2</u></b></dt>
|
|
<dd>Run the crawling session for 2 hours plus the number of minutes specified.
|
|
</dd>
|
|
<dt><b><i>0-1000</i>.</b></dt>
|
|
<dd>Specify the number of hours to run the crawling session.
|
|
</dd>
|
|
</dl>
|
|
<p><b>Element 2: Minutes</b>
|
|
</p>
|
|
<dl>
|
|
<dt><b><u>0</u></b></dt>
|
|
<dd>Run the crawling session for the number of hours specified.
|
|
</dd>
|
|
<dt><b>*SAME</b></dt>
|
|
<dd>Use this value when you are updating the options object, but want to use the same maximum number of minutes to run.
|
|
</dd>
|
|
<dt><b><i>0-59</i>.</b></dt>
|
|
<dd>Specify the number of minutes to run the crawling session. The crawling session will run for the number of hours specified in the first element of this parameter plus the number of minutes specified.
|
|
</dd>
|
|
</dl>
|
|
</div>
|
|
<table width="100%">
|
|
<tr><td align="right"><a href="#STRHTTPCRL.Top_Of_Page">Top</a></td></tr>
|
|
</table>
|
|
<div> <a name="STRHTTPCRL.LOGFILE"></a>
|
|
<h3>Logging file (LOGFILE)</h3>
|
|
<p>Specifies the activity logging file to be used. This file contains information about the crawling session plus any errors that occur during the crawling session. This file must be in a directory.
|
|
</p>
|
|
<dl>
|
|
<dt><b><u>*NONE</u></b></dt>
|
|
<dd>Do not use an activity log file.
|
|
</dd>
|
|
<dt><b><i>path-name</i></b></dt>
|
|
<dd>Specify the path name of the logging file.
|
|
</dd>
|
|
</dl>
|
|
</div>
|
|
<table width="100%">
|
|
<tr><td align="right"><a href="#STRHTTPCRL.Top_Of_Page">Top</a></td></tr>
|
|
</table>
|
|
<div> <a name="STRHTTPCRL.CLRLOG"></a>
|
|
<h3>Clear logging file (CLRLOG)</h3>
|
|
<p>Specifies whether to clear the activity log file before starting the crawling session.
|
|
</p>
|
|
<dl>
|
|
<dt><b><u>*YES</u></b></dt>
|
|
<dd>Always clear the activity log file before each crawling session.
|
|
</dd>
|
|
<dt><b>*NO</b></dt>
|
|
<dd>Do not clear the activity log file.
|
|
</dd>
|
|
</dl>
|
|
</div>
|
|
<table width="100%">
|
|
<tr><td align="right"><a href="#STRHTTPCRL.Top_Of_Page">Top</a></td></tr>
|
|
</table>
|
|
<div> <a name="STRHTTPCRL.VLDL"></a>
|
|
<h3>Validation list (VLDL)</h3>
|
|
<p>Specifies the validation list to use for SSL sessions. Use the Configure HTTP Search (CFGHTTPSCH) command to create a validation list object.
|
|
</p>
|
|
<dl>
|
|
<dt><b><u>*NONE</u></b></dt>
|
|
<dd>Do not use a validation list object.
|
|
</dd>
|
|
<dt><b><i>name</i></b></dt>
|
|
<dd>Specify the name of the validation list.
|
|
</dd>
|
|
</dl>
|
|
</div>
|
|
<table width="100%">
|
|
<tr><td align="right"><a href="#STRHTTPCRL.Top_Of_Page">Top</a></td></tr>
|
|
</table>
|
|
<hr size="2" width="100%">
|
|
<div><h3><a name="STRHTTPCRL.COMMAND.EXAMPLES">Examples</a> </h3>
|
|
<p>
|
|
<pre>
|
|
STRHTTPCRL OPTION(*CRTDOCL) DOCLIST('/mydir/my.doclist')
|
|
URL('http://www.ibm.com') MAXDEPTH(2)
|
|
</pre>
|
|
</p>
|
|
<p>This command starts a new crawling session finding referenced links 2 layers from the starting URL at www.ibm.com. The document list will be created in '/mydir/my.doclist' and will contain sets of a local directory path, for example, '/QIBM/USERDATA/HTTPSVR/INDEX/DOC/www.ibm.com/us/index.html' and the actual URL to the page 'http://www.ibm.com/us/'. Use the Configure HTTP Search (CFGHTTPSCH) command to create an index using this document list.
|
|
</p>
|
|
</div>
|
|
<table width="100%">
|
|
<tr><td align="right"><a href="#STRHTTPCRL.Top_Of_Page">Top</a></td></tr>
|
|
</table>
|
|
<hr size="2" width="100%">
|
|
<div><h3><a name="STRHTTPCRL.ERROR.MESSAGES">Error messages</a> </h3>
|
|
<p><b><u>*ESCAPE Messages</u></b>
|
|
</p>
|
|
<dl>
|
|
<dt><b>HTP160C</b></dt>
|
|
<dd>Request to create or append to a document list failed. Reason &1.
|
|
</dd>
|
|
<dt><b>HTP166E</b></dt>
|
|
<dd>Request to print the status of a document list failed. Reason &1.
|
|
</dd>
|
|
</dl>
|
|
</div>
|
|
<table width="100%">
|
|
<tr><td align="right"><a href="#STRHTTPCRL.Top_Of_Page">Top</a></td></tr>
|
|
</table>
|
|
</body>
|
|
</html>
|