Crawl(File / File)

Operation Name

Crawl(File / File)

Function overview

Passing files to Web pages and writes the acquired data files.

Data model

Data model of this component is XML type.

Properties

For information about using variables, refer to "variables".

Basic settings

Item name	Required / Optional	Use of Variables	Description	Remarks
Name	Required	Not available	Enter the name on the script canvas.

Required Settings

Item name	Required / Optional	Use of Variables	Description	Remarks
Destination	Required	Not available	Select Global Resources. Refer to "Global Resource Properties" to set up a global resource. [Add]: Adds new global resource. [Edit list]: Global resource settings can be edited with "Edit Resource list".
Path	Optional	Available	Enter the path to be connected. If relative path ("/" Path does not start) is specified, "/" is automatically added to the beginning.
Method	Required	Not available	Select the HTTP method. [POST]:(default )
Content-Type	Required	Available	Select or enter Content-Type for POST data . [text/plain]: (default) plain text [text/html]: HTML [text/xml]: XML [application/octet-stream]: Binary
POST file path	Required	Available	Enter the file path. Click [Browse] button to activate the file Selectr and Select the file.	Specify the absolute path of the DataSpider file system as the file path. Constrained characters of DataSpider File System cannot be used except for path separator "/".
Output file	Required	Available	Enter the path of the output file. Click [Browse] button to activate the file Selectr and Select the file.	Specify the absolute path of the DataSpider file system as the file path. Constrained characters of DataSpider File System cannot be used except for path separator "/".
XHTML format	Optional	Available	Select the data obtained whether the XHTML format. [false]: (default) Do not store the data acquired in the form of XHTML. [true]: Data obtained for the XHTML format.	If omitted, "false" is set. For more details on when selecting [true], refer to "Saving in XHTML Format".

Authentication

Item name	Required / Optional	Use of Variables	Description
Authentication mode	Required	Not available	Select the authentication mode. [None]: (default) Do not use authentication. [Basic authentication]: Basic authentication is used.
User name	Optional	Available	Enter the user name used for basic authentication. [Authentication mode] in [Basic authentication] is selected to activate.
Password	Optional	Available	Enter the password of basic authentication. [Authentication mode] in [Basic authentication] is selected to activate.

Cookie

Item name	Required / Optional	Use of Variables	Description	Remarks
Send cookie of subdomain to server	Optional	Not available	Select whether to send subdomains Cookie. [Checked]: (default) Send Subdomain Cookie saved. [Not Checked]: Does not send Cookie subdomain saved.	The Cookie saving process will run automatically after each Web adapter process , and it is used for subsequent process of Web adapter. when using the same global resources for subsequent processing in the adapter Web, even if [Not Checked] is selected cookie of the same domain will be also sent automatically . [Checked] learn more about the "sub-domain for transmission Cookie".

Comment

Item name	Required / Optional	Use of Variables	Description	Remarks
Comment	Optional	Not available	You can write a short description of this adapter. The description will be reflected in the specifications.

Schema

Input Schema

None.

Output Schema

Varies depending on the Web page connecting.

Loading schema in Mapper

Schema needs to be loaded manually.

For any details, please refer to the "Editing Schema"

Mass Data Processing

Mass data processing is not supported.

Transaction

Transaction is not supported.

PSP Usage

PSP is not supported.

Available component variables

Component Variable name	Description	Remarks
message_category	Stroes the category to which corresponding message code belongs to, when an error occurs.	The value defaults to null.
message_code	Stores its corresponding message code of occured error.	The value defaults to null.
message_level	Stores the severity of the corresponding message code of the occured error.	The value defaults to null.
error_type	Returns the error type when error occurred.	The value defaults to null. Error is represented in the format depicted below. Example:java.io.FileNotFoundException The message may vary depending on the DataSpider Servista version.
error_message	Return the error message when error occurred.	The value defaults to null. The message may vary depending on the DataSpider Servista version.
error_trace	Return trace information when error occurred.	The value defaults to null. The message may vary depending on the DataSpider Servista version or the client application used.

About subdomain of Cookie sending

Cookie subdomain compared the property of cookie saved to destination information, and will be sent when all of the following conditions are met.

The domain attribute value matches the destination host
It satisfies only when the following conditions are considered .
- domain attribute "."If the host name that adhere
- If the IP address of domain attributes
The path attribute value is higher than the one of destination or the same.
Value of secure property matches with the method of communication of destination(HTTPS/HTTP).
The date of expire day is a date after now.

Example

Destination URL	Cookie Attributes				Send / Do not send
Destination URL	domain	path	secure	expire	Send / Do not send
http://test.example.com/aaa/	.example.com	/aaa/	false	Indefinite period	Send to satisfy all conditions.
http://example.com/aaa/	.example.com	/aaa/	false	Indefinite period	Does not send message because the property of domain does not match with cookie.
http://test.example.com/aaa/	.example.com	/aaa/bbb/	false	Indefinite period	Does not send because the property of path is below the path.
http://test.example.com/aaa/	.example.com	/aaa/	true	Indefinite period	Does not send because the property of secure is a value of secret communication.
http://test.example.com/aaa/	.example.com	/aaa/	false	Yesterday's date	Does not send because the property of expire is older than now.

If the same attribute domain, path attributes, secure attribute, name attribute has been saved the last stored in the processing of Web Cookie will be sent to the adapter.

Saving in XHTML Format

If saving acquired data in XHTML format, it is necessary to identify encoding for the data in order to convert data properly.
Encoding will be searched in following procedure and use the first one found.
1. Charset parameter included in Content-Type header for HTTP response
2. charset parameter with meta element for obtained HTML and included "Content-Type" as http-equiv attribute.
If non can be found, JISAutoDetect will be used.
JISAutoDetect is a function which automatically distinguish between JIS, Shift_JIS, or EUC-JP.

Specification Limits

Mass data processing is not supported. The information may not be gotten correctly if the structure of HTML document specified by URL is different from that of setting and running.

The information may not be gotten correctly if the structure of HTML document specified by URL is different from that of setting and running.

Cannot work in the environment where cannot connect to the Internet.

If you go beyond Firewall and use HTML as a data resource, you need to set http.proxyHost as well as http.proxyPort in DataSpiderServer or system.properties of DataSpiderClient.
Refer to "Property Reference" for details.

Character code, including HTML documents may be handled correctly.

Main exceptions

Exception Name	Causes	Solution
ResourceNotFoundException Resource definition could not be found.Name: []	[Destination] is not specified.	Please specify [Destination] .
ResourceNotFoundException Resource definition could not be found.Name: [<name of Global Resources>]	Resource definition selected in [Destination] is not found.	Please check the global resource specified in [Destination]
java.net.UnknownHostException	Exception if Web server is not found.	Please check the Web server configuration.
java.net.ConnectException	Exception if can not Connect to Web Server.	Check if port number or Web server is running.
org.apache.commons.httpclient.HttpConnection$ConnectionTimeoutException	Connection timed out during connecting with the Web server.	Confirm the state of network.Or, confirm [Time out] of global resource specified by [Destination].
org.apache.commons.httpclient.HttpRecoverableException java.net.SocketTimeoutException: Read timed out	Connection timed out after connecting to Web server.	Confirm the state of Web server.Or, confirm [Time out] of global resource specified by [Destination].

Notes

If the root certificate is not registered in Java
Java root certificate is not registered in the Web server when HTTPS connection, javax.net.ssl.SSLHandshakeException exception occurs, such as the connection will fail.
For information regarding how to import to the keystore, refer to "Import of certificate".
The certificate can also be retrieved from Internet Explorer. For more details, refer to "Download certificate using Internet Explorer".