Service Operation

The following sections describe the functions and the terminology that are necessary to understand for your service operations.
The sections below assume that reader has studied the Service Development and understands the concept of the service and functions.

Before operating the services

Registering the Windows Service

It is recommended to register DataSpiderServer to the Windows service if the production environment is built in Windows.
Here are some benefits of registering the Windows service. Please register to the Windows service when installing.
For more details, refer to "DataSpider Servista Installation Guide" offered in PDF.

Security

Depending on system requirements, transmission between HTTP triggers or Web service triggers and DataSpiderServer needs to be SSL.
In that case, you need to enable HTTPS settings in DataSpiderServer.
For more details about settings, refer to "HTTPS setting".

Deciding the Log Level

With Service execution tool, you can set "Log level" of executing script.
The recommended log level upon operating is "INFO".

The lower the log levels are, the more detailed the output contents to XML log will be, but that also increases the log file size.
If the disc capacity is oppressed due to the file ballooning, not only DataSpider Servista but also other systems are affected. Thus you need to select the appropriate log levels adjusting to the requirements and environments.
Even if you set not to output XML log, Dump file will be output. Therefore if you are anxious about the disc capacity, please consider about setting to not outputting XML log.

Server migration

Server migration is a function which enables the migration of the current server settings to different DataSpiderServer environment. Server migration can be useful when migrating settings used in a staging environment to a live environment.
No need for server migration if the live environment is the same as the development environment or the test environment.

Server migration: How To

Server migration can be completed in the following way.
  1. Exporting DataSpiderServer settings from the origin as a zip archive
  2. Install the same version of DataSpiderServista to the test environment or the live environment as the development environment.
  3. Configure settings to the destination DataSpiderServer.
    Please save the below settings to DataSpiderServer if required as those are not covered by server migration.
  4. Import the zip archive to the destination and unzip it as appropriate.

Settings migrated

Server settings which can be migrated are the followings. It is possible to selectively migrate the each of the following settings. When server settings are migrated, authority settings are too migrated as necessary. If the owner of files imported does not exists at the destination, the user who performed migration process becomes the owner.
See Server Migration(Export) for export, and for import, see Server Migration(Import).

Monitoring

Log monitoring

There are various log files used in DataSpider Servista.
You can use operation monitoring tools available in the market. For example, you can assess problems of DataSpiderServer and fatal errors based on certain messages and log levels appeared in the server log.
More specifically, the message "DataSpiderServer was resumed." which appears when resuming DataSpiderServer and log levels such as "FATAL" and "ERROR" should be inspected.
It is possible to change the standard of the log levels according to the operation policies and status.

The recommended log level for operation is "INFO". If you want to output only important information, you can set the log level to a higher level, "NOTICE". If you need more detailed information, set the log level to "FINFO", a one rank lower than "INFO".
If the log level is set as "INFO", a log to indicate caution in the operation, "WARN", is output. If the log level is "FINFO", a log to indicate a caution in a development phase, "FWARN" is output, in addition to "WARN". Logs which are output at a level of "WARN" or "FWARN" are not errors. However, please set those logs to be monitored when necessary in both an operation phase and a development phase.
See Logging Guide for logging facilities, and for all log levels supported by system, see Log Level.

Monitoring processes

By monitoring processes of DataSpiderServer, you can detect a sudden resume.
Please monitor the service name if registered to the Windows service. If not, the process ID.

Task Manager

Task Manager provides information about the scripts running on a computer and the total current memory usage reported.
You can check the amount of memory used in WebConsole. You can see a graphic image of the transition of memory usage in the Studio and the Studio for Web.
See Task Manager for any further details.

CLI Console

CLI stands for Command Line Interface. Several commands to support operations are available in Console.
Below is one of the examples.

Command name Description Remarks
imp Imports server information.
  • Administrator Command
exp Exports Server Information
  • Administrator Command
kill Requests to stop the process.  
myservices Shows the list of services.  
processcount Views and sets the number of processes and scripts in progress.  
ps Shows script processes.  
run Executes services.  
session Manages session information and shows status,
  • Administrator Command
shutdown Stops and restarts DataSpiderServer.
  • Administrator Command
threadstat Gets thread information.  
uname Shows system information.  

In Server CLI Console , these command can also be executed from batch programs.(CLI Batch)
For example, you can restart DataSpiderServer periodically from the Windows task scheduler using CLI batch.
See CLI Console for any details.

In the event of failure

My Logs

My Logs lets users view the event logs that store information about significant errors that happened within the components services are consisted of.
In Webconsole, you can browse the summary of the process. In the Studio and the Studio for Web, you can browse details with a graphical image for each component units by using a log viewer.
See My Logs for any details.

Message code

Each error has an unique corresponding message code. Message code is helpful when narrowing a problem down.
See Message Code for any details.

Log files

Depending on the error cases, you may have to view the logs output to each log file, not just my log.
Please Select the log file you are referring to according to the cases.
Most logs are output to a server log or a XML log. If there are some critical issues which would affect to DataSpiderServer, a message will be output to the standard error output.
See Logging Guide for any details.

Maintenance

Rolling up logs

There is a function called "Rotation", to delete logs when they become heavy at a certain level.
Records in the date directory are retained for 10 days, counted from the following day. Past records are automatically deleted.
A new log file is generated in the server log and in the Studio log when they reach to a certain amount of size. Old files are retained up to 5 by default.
It is recommended to configure rotation settings according to the operation policies and the disk capacity.
For information about changing rotations, refer to "DataSpider log settings" or "Property Reference".

Cleaning up Repository DB

The data source of settings and files stored in a repository DB will remain even after deleted.
In this case, delete the unnecessary data source by using the cleaning up function.
It is recommended to clean up data periodically in repository DB.
Before you use the cleaning up function, please make sure the service is not running.
For more details, refer to "Clean up Repository DB".

Cleaning up Temporary directory

The /tmp directory is used as a temporary directory for the system and components. The temporary files may not be deleted depending on the settings, and it may cause filling up the disk capacity.
To avoid this, it is recommended to delete temporary files periodically by using the cleaning up function.
Before you use the cleaning up function, please make sure the service is not running at the moment and also no other user is operating clients.
For more details, refer to "Cleaning up Temporary Directory".

Logging Guide

Types of log files, and what it contain is depicted below.

Types of log files

There are following log types available.

Category Name Description File path Remarks
Server Server Log The logging pertaining to DataSpiderServer.Events from Kernel and each service manager DataSpiderServer are logged to the Server Log.
$DATASPIDER_HOME/server/logs/server.log
Server Error Log Error events logged in DataSpiderServer process are redirected to the Server Error Log from standard error output. $DATASPIDER_HOME/server/logs/server.error.log
  • The redirection destination can be modified in DataSpiderServer.lax file.
  • The redirection destination file is created every time DataSpiderServer starts, deleting the existing log file with the same name, if any.
  • Log rolling up is not performed.
Execution Log Summaries for script execution status are logged to the Execution Log. $DATASPIDER_HOME/server/logs/<date in '(yyyyMMdd' format>/exec.log
  • Settings for log rotation based on date directory is discussed in section DataSpider Log Settings.
  • You can change the settings for output contents, destination and rotation in dslog.properties.
XML Log The logs of the components executed are stored in the XML Log.
The information output is depends on the Log Level set in the script executed.		 $DATASPIDER_HOME/server/logs/<date in 'yyyyMMdd' format>/execution/<Execution ID>.xml
Launch Log The logging pertaining to Launch External Application operation.Elapsed time since an external application has been started and the command line started the application are logged to the Launch Log. $DATASPIDER_HOME/server/logs/<date in 'yyyyMMdd' format>/launch.log
Launch Process Log Events from the launched external application that are ordinary logged to the standard output and the standard error output are redirected to the Launch Process Log. $DATASPIDER_HOME/server/logs/<date in 'yyyyMMdd' format>/launch.log
Server CLI Console Log The logging pertaining to Server CLI Console.Events logged in Server CLI Console operation, including command history, are logged to the Server CLI Console Log. $DATASPIDER_HOME/server/logs/cli.log
  • You can change the settings for output destination in dslog.properties.
  • If you execute "$DATASPIDER_HOME/server/bin/Shutdown", shutdown command history is output.
Access Log The log for accessing to DataSpiderServer from the client is output. $DATASPIDER_HOME/server/logs/access.log
  • You can change the settings for output destination in dslog.properties.
Client Studio Log The logging pertaining to DataSpider Studio. $DATASPIDER_HOME/client/logs/studio.log
  • You can change the settings for output destination in dslog.properties.
Client CLI Console Log The logging pertaining to Client CLI Console. Includes command history. $DATASPIDER_HOME/client/logs/cli.log
  • You can change the settings for output destination in dslog.properties.
Studio Standard Output The logging pertaining to DataSpider Studio process is redirected to the Studio Standard Output $DATASPIDER_HOME/client/logs/client.log
  • The redirection destination can be modified in DataSpiderStudio.lax file.
  • The redirection destination file is created every time DataSpider Studio is launched, deleting the existing log file with the same name, if any.
  • Log rolling up is not performed.
Studio Standard Error Output Error events logged in DataSpider Studio process are redirected to the Studio Standard Error Output from the standard error output. $DATASPIDER_HOME/client/logs/client.error.log

Other types of logs are those to which users specify their own log format arbitrarily.
See Application Log Output Settings for any details.

Server Log

The logging pertaining to DataSpiderServer.Events from Kernel and each service manager DataSpiderServer are logged to the Server Log.
Events occurred during execution of scripts are logged to XML Log.
The following events are logged to the Server Log destination.

An example of log output

10/22 10:28:49|INFO|ds.boot|Service of module [SystemModule Container] was successfully started.
10/22 10:28:49|INFO|ds.boot|Starting service of module [ScriptRunner Container]...
10/22 10:28:49|INFO|ds.boot|Starting service of module [DataProcessingComponent Manager]...
10/22 10:28:49|INFO|ds.boot|Service of module [DataProcessingComponent Manager] was successfully started.
10/22 10:28:49|INFO|ds.boot|Service of module [ScriptRunner Container] was successfully started.
10/22 10:28:49|INFO|ds.boot|System service was successfully started.
10/22 10:28:49|NOTICE|ds.boot|DataSpider Server was successfully started.
10/22 10:28:49|INFO|ds.boot|*****************************************************************
10/22 12:01:23|INFO|ds.kernel.mgr.callback|START|CALLBACK_THREAD|CLIENT=qm0000rg4dl1i0so2j7b72t8hb0l
10/22 12:01:30|FINFO|ds.kernel.login|SUCCEEDED|LOGIN|USER=root|SESSION_ID=lli947bl3u22oe1cu2ejet71md8lu4tf
10/22 12:03:45|FINFO|ds.kernel.mgr.dpc.pool|GC_START
10/22 12:03:45|FINFO|ds.kernel.mgr.dpc.pool|GC_END

Server Error Log

Error events logged in DataSpiderServer process are redirected to the Server Error Log from standard error output.
There are cases when events are not properly logged to the designated files but to the standard error output when the server is in severe condition due to critical problems occurred.

Execution Log

Summaries for script execution status are logged to the Execution Log.
The start and end of a script are logged correspondingly when a script is executed.
Those are the values of [Search result] found in My Logs.
This logging include the following items: Optionally, this logging can be separately made to $DATASPIDER_HOME/server/logs directory(the path is fixed and cannot be changed arbitrary) by setting in dslog.properties.This will become useful when the status of scripts being executed are to be monitored by other systems.

An example of log output

|2007/10/10 13:43:19|-|ID=j20000dg6il1i0go0c5iionihb21|VMID=pk0000ng0kl1iembiradhpv0l1b9|OWNER=root|PROJECT=null|SCRIPT=Script|TYPE=default
|2007/10/10 13:43:19|0|ID=j20000dg6il1i0go0c5iionihb21|TOTAL=1969|EXIT_STATUS=0
|2007/10/10 13:48:04|-|ID=be0000dg6il1i0go0c5rb722hb21|VMID=pk0000ng0kl1iembiradhpv0l1b9|OWNER=root|PROJECT=null|SCRIPT=Script|TYPE=default
|2007/10/10 13:48:05|1|ID=be0000dg6il1i0go0c5rb722hb21|TOTAL=547|EXIT_STATUS=-10101

Examples of log output format

|Date | End status | Execution ID | VMID | Execution user name | Project name | Script name | Type name
| date | Exit Status | Execution ID | Execution time | Exit status |

XML Log

Events logged in the operations of the components used in a script are logged to a XML Log, one file per script.

XML log file is created upon script execution, and when the operation is started and completed, logs will be output. The file name is the ID for the running script.
The Log viewer displays information derived form the XML Log.
The volume of log output and messages depends on the active log level threshold set in Designer, Trigger, ScriptRunner, and etc.

Dump file

Errors occurred during script execution may be logged to dump.xml file created under the directory that is named after the script execution ID.
Errors occurred within the Monitor Exception Operation components are not logged to the dump.xml file.
Write Table Operation and Execute Update SQL Operation of database adapters provide detailed information of the operations performed in the dump file.

Example: Script exited successfully

<?xml version="1.0" encoding="UTF-8" ?>
<exec>
  <executionStart time="1192189123891" production-version="2.4.0" />
  <scriptStart time="1192189124094" id="so0000fg10l1i0co28lhc5gaharl" version="2" name="script" project="root@project" type="default" enableTypeSwitch="false" repository="/home/root/projects/project" supportsLargeData="false" psp="false" />
  <start time="1192189124110" id="o20000fg3pl1i0ko50lkhvfghb4e" name="mapping" />
  <end time="1192189124251" id="o20000fg3pl1i0ko50lkhvfghb4e" />
  <scriptEnd time="1192189124251" status="0" />
  <executionEnd time="1192189124266" status="0" />
</exec>

Example: Script exited with an error

<?xml version="1.0" encoding="UTF-8" ?>
<exec>
  <executionStart time="1262591044666" production-version="2.4.0" />
  <scriptStart time="1262591044701" id="va0000bg70l1ie6moi7s2evoib8c" version="1" name="script" project="root@project" type="default" enableTypeSwitch="false" repository="/home/root/projects/project" supportsLargeData="true" psp="false" />
  <start time="1262591044708" id="si0000jg20l1ifemqt2klig8ibu2" name="csv_read" />
  <end time="1262591044759" id="si0000jg20l1ifemqt2klig8ibu2">
    <failure>
      <code><![CDATA[SCRIPT0003E]]></code>
      <type><![CDATA[com.appresso.ds.script.runtime.ComponentFailedException]]></type>
      <message><![CDATA[Failed to process [csv_read(File/CSV/Read CSV File)].Cause: [com.appresso.ds.common.dp.InvalidPropertyConfigurationException: File path is not specified.]]]></message>
      <detail><![CDATA[com.appresso.ds.script.runtime.ComponentFailedException: Failed to process [csv_read(File/CSV/Read CSV File)].Cause: [com.appresso.ds.common.dp.InvalidPropertyConfigurationException: File path is not specified.]
	... (skip)
]]></detail>
    </failure>
  </end>
  <scriptEnd time="1262591044768" status="-10101" />
  <executionEnd time="1262591044771" status="-10101">
    <failure>
      <code><![CDATA[SCRIPT0003E]]></code>
      <type><![CDATA[com.appresso.ds.script.runtime.ComponentFailedException]]></type>
      <message><![CDATA[Failed to process[csv_read(File/CSV/Read CSV File)].Cause: [com.appresso.ds.common.dp.InvalidPropertyConfigurationException: File path is not specified.]]]></message>
      <detail><![CDATA[com.appresso.ds.script.runtime.ComponentFailedException: Failed to process [csv_read(File/CSV/Read CSV File)].Cause: [com.appresso.ds.common.dp.InvalidPropertyConfigurationException: File path is not specified.]
	... (skip)
]]></detail>
    </failure>
  </executionEnd>
</exec>
Elements Attributes Description Remarks
executionStart time Operation executing starting time  
production-version DataSpiderServer version  
scriptStart time Start time of a script  
id script entry ID  
version The version of script  
name Script name  
project Project name  
type type identifier  
enableTypeSwitch Use the default type if the specified type is not found.(true or false)  
repository The location where projects are saved  
supportsLargeData Whether if the feature of handling large volume is enabled(true or false)  
psp Whether the script is PSP enabled(true or false)  
var id Script variable ID  
name Script variable name  
start time Start time of the operation performed by component  
id Component ID  
name Component icon name  
end time The end time of the operation performed by component  
id Component ID  
log level Log level  
log msg - Log message  
loopStart time The start time of the operation performed by component
  • Recorded in the order of Loop Operation, Conditional Loop Operation, Loop by Number of Data Operation.
id Component ID  
name Component icon name  
loopEnd time The end time of the operation performed by component
  • Recorded in the order of Loop Operation, Conditional Loop Operation, Loop by Number of Data Operation.
id Component ID  
count The number of iterations performed  
branchStart time The start time of the operation performed by component
  • Recorded before conditional operations
id Component ID  
name Component icon name  
branchEnd time The end time of the operation performed by component
  • Recorded after conditional operations
id Component ID  
eval id id of conditional operation  
name Condition icon name  
result The result of conditional operation(true or false)  
transactionStart - The start time of the operation performed by component
  • Recorded when transaction is started
transactionEnd time The end time of the operation performed by component
  • Recorded when transaction is ended.
type Transaction status(commit or rollback)  
tryStart time The start time of the operation performed by component
  • Recorded before Monitor Exception Operation is started.
id component ID  
name component icon name  
exceptionCaught time The start time of the operation performed by component
  • Recorded before Monitor Exception Operation is started.
id component ID  
name component icon name  
tryEnd time The end time of the operation performed by component
  • Recorded after Monitor Exception Operation is ended.
id component ID  
failure code - message code
  • Recorded only when the operation is abnormally terminated.
type - error type
  • Recorded only when the operation is abnormally terminated.
message - error messages
  • Recorded only when the operation is abnormally terminated.
detail - trace information
  • Recorded only when the operation is abnormally terminated.
scriptEnd time Script execution end time  
status Script execution status  
executionEnd time Process execution end time  
status Script end status  

Logs downloaded in text format

XML Log can be downloaded in Download in text form within from My Logs.

Example: Script exited successfully

06/20 19:34:39.760|EXEC_START  |
06/20 19:34:39.762|SCRIPT_START|SCRIPT=script, PROJECT=root@project, REPOSITORY=/home/root/projects/project, VERSION=2
06/20 19:34:39.763|START       |NAME=mapping 
06/20 19:34:39.763|END         |
06/20 19:34:39.763|SCRIPT_END  |EXIT_STATUS=0
06/20 19:34:39.763|EXEC_END    |EXIT_STATUS=0

Example: Script exited with an error

06/20 19:37:14.607|EXEC_START  |
06/20 19:37:14.609|SCRIPT_START|SCRIPT=script, PROJECT=root@project, REPOSITORY=/home/root/projects/project, VERSION=0
06/20 19:37:14.609|START       |NAME=csv_read [csv_adapter/get_data]
06/20 19:37:14.671|END         |
com.appresso.ds.script.runtime.ComponentFailedException: Failed to process [csv_read(File/CSV/Read CSV File)]. Cause:[com.appresso.ds.common.dp.InvalidPropertyConfigurationException: File path is not specified.]
	... (skip)
06/20 19:37:14.673|SCRIPT_END  |EXIT_STATUS=-10101
06/20 19:37:14.674|EXEC_END    |EXIT_STATUS=-10101
com.appresso.ds.script.runtime.ComponentFailedException: Failed to process [csv_read(File/CSV/Read CSV File)]. Cause:[com.appresso.ds.common.dp.InvalidPropertyConfigurationException: File path is not specified.]
	... (skip)
Elements Attributes Description Remarks
EXEC_START - Start time of the entire process  
SCRIPT_START - Operation executing starting time  
SCRIPT Script name  
PROJECT Project name  
REPOSITORY The location where projects are saved  
VERSION The version of script  
VARIABLE NAME Script variable name  
VALUE Value dereferenced by script variables  
START - Start time of the operation performed by component  
NAME Component icon name  
END - The end time of the operation performed by component  
ERROR
NOTICE
WARN
INFO
FWARN
FINFO
FINEST
DEBUG		 - The Log level used  
LOOP_START - Start time of the Loop, Conditional Loop, or Loop by Number of Data operation  
NAME Component icon name  
LOOP_END - End time of the Loop, Conditional Loop, or Loop by Number of Data operation  
COUNT Number of iteration  
BRANCH_START - Start time of the Conditional Branch operation  
NAME Component icon name  
BRANCH_END - End time of the Conditional Branch operation  
EVALUATION NAME Name of the condition  
RESULT Result of the predicate  
TX_START - Start time of a transactional process  
TX_END - End time of a transactional process  
COMMIT whether the transaction has been committed successfully; true or false.  
TRY_START - Start time of the Monitor Exception operation  
NAME Component icon name  
CATCH - Start time of the Monitor Exception operation  
NAME Component icon name  
TRY_END - End time of the Monitor Exception operation  
SCRIPT_END - Script execution duration  
EXIT_STATUS The exist status of the script  
EXEC_END - The execution duration of over all process.  
EXIT_STATUS The execution status  

Launch Log

Elapsed time since an external application has been started and the command line started the application are logged to the Launch Log.
This logging include the following items:

An example of log output

18:07:12|do0000rg7nl1i108hdf2ecash4qn|INFO|EXEC|COMMAND=/C:/test.bat
18:07:12|do0000rg7nl1i108hdf2ecash4qn|INFO|STARTED
18:07:12|do0000rg7nl1i108hdf2ecash4qn|DEBUG|PROCESS_MONITOR|STARTED
18:07:12|do0000rg7nl1i108hdf2ecash4qn|DEBUG|PROCESS_MONITOR|WAIT_FOR
18:07:12|do0000rg7nl1i108hdf2ecash4qn|DEBUG|STARTED|STREAM=java.io.BufferedInputStream@1f4ff23
18:07:12|do0000rg7nl1i108hdf2ecash4qn|DEBUG|STARTED|STREAM=java.io.FileInputStream@19a8c41
18:07:12|do0000rg7nl1i108hdf2ecash4qn|DEBUG|STDERR_MONITOR|ENDED|TOTAL=0
18:07:12|do0000rg7nl1i108hdf2ecash4qn|WARN|ENDED|EXIT_VALUE=0
18:07:12|do0000rg7nl1i108hdf2ecash4qn|DEBUG|STDOUT_MONITOR|ENDED|TOTAL=87

Launch Process Log

Events of the launched external application that are ordinary logged to the standard output and the standard error output are redirected to the Launch Process Log.
Execution ID of the script is logged to the file with the output items.

An example of log output

--------------------
18:07:12|do0000rg7nl1i108hdf2ecash4qn|INFO

C:\Program Files\DataSpiderServista\server>echo test output
test output

Server CLI Console Log

Events occurred in Server CLI Console operation, including command history, are logged to the Server CLI Console Log.
This logging include the following items:

An example of log output

10/14 22:58:00|INFO|ds.console.cli|USER=root|START help
10/14 22:58:00|INFO|ds.console.cli|USER=root|END help
10/14 22:58:11|INFO|ds.console.cli|USER=root|START pwd
10/14 22:58:11|INFO|ds.console.cli|USER=root|END pwd

An example of log output format

Date | Log level | Module name | User-name | Execution command

Access Log

Events logged in accessing to DataSpiderServer from clients are logged to the access log.
To enable the access log, configure settings in "dslog.properties" and "DataSpiderServer.lax".

Process

The following events will be logged to the access log: If those events are processed within DataSpiderServer, not within a client, they will not be logged to the access log.

Contents of log output

Below are the contents which can be logged:

An example of log output

03/27 21:44:32|root|192.168.0.1|Studio|SESSION|LOGOUT|jqlbjq2tp3034944s5ni38au52efkjun
03/27 21:44:37|root|192.168.0.1|Studio|SESSION|LOGIN|ipgm59kk1s3cskorubn5h675dmnl004m,en_US
03/27 21:44:48|root|192.168.0.1|Studio|PROJECT|DELETE|/home/root/projects/project
03/27 21:44:56|root|192.168.0.1|Studio|PROJECT|CREATE|/home/root/projects/project
03/27 21:44:56|root|192.168.0.1|Studio|PROJECT_PERMISSION|UPDATE|/home/root/projects/project
03/27 21:45:05|root|192.168.0.1|Studio|PROJECT|UPDATE|/home/root/projects/project,i80000bg4hl1id68g51039ocjcl1,0

An example of log output format

|Date|User name|IP address of the client machine|Name of the tool you used for access|Target name|Operation name|
|Date|User name|IP address of the client machine|Name of the tool you used for access|Target name|Operation name|Message
|Date|User name|IP address of the client machine|Name of the tool you used for access|Target name|Operation name|Message1,Message2,Message3
Target name Operation name Message Description Remarks
CALENDAR CREATE Calendar name Output when a calender is created.  
DELETE Calendar name Output when a calender is deleted.  
UPDATE Calendar name Output when a calender is modified.  
DIR COPY Copy source directory path, Copy destination directory path Output when a directory is copied.  
CREATE Directory path Output when a directory is created.  
DELETE Directory path Output when a directory is deleted.  
LOCK Directory path Output when a directory is locked.  
MOVE Source directory path, Destination directory path Output when a directory is moved.  
UNLOCK Directory path Output when a directory is unlocked.  
ENV_VAR CREATE Environment variable name Output when an environment variable is created.  
DELETE Environment variable name Output when an environment variable is deleted.  
RELOAD Environment variable name Output when an environment variable is reloaded.  
UPDATE Environment variable name Output when an environment variable is modified.  
FILE COPY Copy source directory path, Copy destination directory path Output when a file is copied.  
CREATE File path Output when a file is created.  
DELETE File path Output when a file is deleted.  
LOCK File path Output when a file is locked.  
MOVE Source file path, Source destination file path Output when a file is moved.  
READ File path Output when a file is loaded.  
UNLOCK File path Output when a file is unlocked.  
WRITE File path Output when a file is written in.  
FTPSERVER BOOT   Output when the FTP server is booted.
  • Output when the FTP server is booted or restarted in "FTP Server Settings" on the Control Panel.
SHUTDOWN   Output when the FTP server is stopped.
  • Output when the FTP server is stopped or restarted in "FTP Server Settings" on the Control Panel.
FTPSERVER_ACTIVE_PORT UPDATE Active mode port number Output when [Active mode Port Number] on the FTP server is modified.  
FTPSERVER_ANONYMOUS_LOGIN_ENABLED UPDATE true or false Output when [Allow anonymous login] on the FTP server is modified.
  • Output "true" if [Allow anonymous login] is checked. Output "false" if unchecked.
FTPSERVER_IDLE_TIMEOUT UPDATE Idle time out Output when [Idle time out] on the FTP server is modified.  
FTPSERVER_MAX_ANONYMOUS_LOGIN UPDATE Maximum anonymous login number Output when [Maximum anonymous login number] on the FTP server is modified.  
FTPSERVER_MAX_LOGIN UPDATE Maximum login number Output when [Maximum login number] on the FTP server is modified.  
FTPSERVER_PASSIVE_PORT UPDATE Passive mode port number Output when [Passive mode port number] on the FTP server is modified.  
FTPSERVER_PASSIVE_PORT_OPTION UPDATE 0 or 1 Output when [Automatic allocation] or [Specify range] of [Passive mode port number] on the FTP server is modified.
  • Output "0" if changed to [Automatic allocation]. Output "1' if changed to [Specify range].
FTPSERVER_PORT UPDATE Port number used for the FTP server Output when [Port number] on the FTP server is modified.  
FTPSERVER_START_OPTION_ENABLED UPDATE true or false Output when [Start FTP server when DataSpiderServer Start] is modified.
  • Output "true" if [Start FTP server when DataSpiderServer Start] is checked. Output "false" if unchecked.
GROUP CHANGE Path, Group Output when the group of the file/directory is modified.
  • In the Studio for Web, an access log is output when [OK] or [Apply] is clicked without changing the below properties.
    • Mount point properties in Mount settings
    • Service properties in My Services
    • Properties of a file or a directory on Explorer
  • In WebConsole, an access log is output when [Update] is clicked without changing the properties in Mount.
CREATE Group name, Group description Output when a group is created.  
DELETE Group name Output when a group is deleted.  
GROUP_DESC UPDATE Group name, Group description Output when a group description is modified.  
GROUP_OWNER UPDATE Group name, User name, true or false Output when the owner of the specified group is changed.
  • Output "true" if the user is the owner of the group. Output "false" if not.
GROUP_USER ADD Group name, User name Output when a user is added to a group.  
DELETE Group name, User name Output when a user is deleted from a group.  
HTTPS_ENABLED UPDATE true or false Output when [Enable HTTPS] is modified.
  • Output "true" if [Enable HTTPS] is checked. Output "false" if not.
HTTPS_PORT_NUMBER UPDATE Port number used HTTPS Output when the port number for HTTPS is modified.  
KEYSTORE_PASSWORD UPDATE   Output when the keystore password for HTTPS is modified.  
LOGGER CREATE Name of Log output destination setting Output when an Log output destination setting is created.  
DELETE Log output destination setting Output when an Log output destination setting is deleted.  
LOG_ARCHIVE_ENABLED UPDATE true or false Output when the log compress settings is modified.
  • Output "true" if [Compress log] is selected. Output "false" if [Not compress log] is selected.
LOG_ARCHIVE_RETENTION_PERIOD UPDATE Log retension period to be compressed Output when the log retension period to be compressed is modified.  
LOG_DELETE_ENABLED UPDATE true or false Output when the log delete settings is modified.
  • Output "true" if [Compress and delete log] is selected. Output "false" if [Not compress and delete log] is selected.
LOG_RETENTION_PERIOD UPDATE Log retension period to be deleted Output when the log retension period to be deleted o is modified.  
LOG_SEARCH_COND CREATE Log search condition name Output when a log search condition is created.  
DELETE Log search condition name Output when a log search condition is deleted.  
UPDATE Log search condition name Output when a log search condition is modified.  
MAX_EXEC_COUNT UPDATE Maximum execution count Output when the maximum execution count is modified.  
MEMORY CLEANUP   Output when memory is cleaned up.  
MODE CHANGE Path, Access Permission Mode Output when the access permission mode for a file/directory is modified.
  • In the Studio for Web, an access log is output when [OK] or [Apply] is clicked without changing the below properties.
    • Mount point properties in Mount settings
    • Service properties in My Services
    • Properties of a file or a directory on Explorer
  • In WebConsole, an access log is output when [Update] is clicked without changing the properties in Mount.
MOUNT_POINT CHANGE Current mount destination, New mount destination Output when a mount point is modified.  
CREATE Mount destination, Mount source Output when a mount point is created.
  • If the mount type is [Database File System], the source will not be output.
DELETE Mount destination Output when the mount point is unlocked.  
OWNER CHANGE Path, Owner Output when the owner of the file/directory is changed.
  • In the Studio for Web, an access log is output when [OK] or [Apply] is clicked without changing the below properties.
    • Mount point properties in Mount settings
    • Service properties in My Services
    • Properties of a file or a directory on Explorer
  • In WebConsole, an access log is output when [Update] is clicked without changing the properties in Mount.
PASSWORD UPDATE   Output when the login user password is modified.  
POOL RESET Global resource name, Type Output when resetting a pool is executed.
  • If the "-a" or the "--all" option is executed in the grp command in CLI, no message is output.
PORT_NUMBER UPDATE Port number Output when the port number of DataSpiderServer is modified.  
PROCESS END Execution ID Output when a script process is aborted.  
PROJECT CREATE Project path Output when a project is created.  
DELETE Project path Output when a project is deleted.  
DOWNLOAD Download source project path, Download destination project path, Version Output when a project is downloaded.
  • If a batch download is executed to all versions, no version is output.
  • If a batch download is executed to all versions in the Studio for Web, [CREATE] in [FILE] and [DIR], as well as [READ] and [WRITE] in [FILE] are output, while [DOWNLOAD] in [PROJECT] is not.
  • If a download is executed to a specific version in the Studio for Web, Download destination path is "/tmp/RelayServer/<Project name>" directory in DataSpider file system.
MOVE Source project path, Destination project path Output when a project is moved.
  • Projects are moved by executing below processes:
    • Share projects, or cancel the shared projects
    • Change the project name
UPDATE Project path, Script entry ID, New version of the script Output when a project is saved.  
UPLOAD Upload source project path, Upload destination project path Output when a project is uploaded from local.
  • In the Studio for Web, the upload source project path is same as the temporary path which is copied in the "/tmp/relayserver" directory in the DataSpider file system.
PROJECT_PERMISSION UPDATE Project Path Output when the project user or group permission is modified.  
PROJECT_TAG UPDATE Project path, Version, Tag Output when a project tag is modified.  
REPOSITORY CLEANUP   Output when the cleanup command in a repository DB is executed.  
REPOSITORY_DBMS UPDATE DBMS Output when the type of RDB used as a repository DB is modified.  
REPOSITORY_DRIVER_PATH UPDATE JDBC driver path Output when the JDBC driver path used for connecting to a repository DB is modified. /td>  
REPOSITORY_PASSWORD UPDATE Password Output when the password of a repository DB connection user is modified.  
REPOSITORY_POOL RELEASE   Output when the repository DB pool is released.  
REPOSITORY_POOLED_LIFE_TIME UPDATE Connection lifetime Output when the repository DB connection lifetime is modified.  
REPOSITORY_POOL_CHECK_INTERVAL UPDATE Check interval Output when the repository DB check interval is modified.  
REPOSITORY_POOL_INITIAL_CAPACITY UPDATE Initial connection number Output when the initial connection number of a repository DB is modified.  
REPOSITORY_POOL_MAX_CAPACITY UPDATE Connection upper bound number Output when the connection upper bound number of the repository DB is modified.  
REPOSITORY_RECONNECT_COUNT UPDATE Reconnect count Output when the the count of tries for reconnecting to the repository DB is modified.  
REPOSITORY_RECONNECT_INTERVAL UPDATE Reconnect interval Output when the interval of tries for reconnecting to the repository DB is modified.  
REPOSITORY_TESTCONNECT_COUNT UPDATE Test connect count Output when the generating connection try count is modified  
REPOSITORY_TESTCONNECT_INTERVAL UPDATE Test connect interval Output when the generating connection try interval is modified.  
REPOSITORY_URL UPDATE URL Output when the URL to connect the repository DB is modified.  
REPOSITORY_USER UPDATE User Output when the user connecting the repository DB is modified.  
RESOURCE CREATE Global resource name, Type Output when a global resource is created.  
DELETE Global resource name, Type Output when a global resource is deleted.  
UPDATE Global resource name, Type Output when a global resource is updated.
SCHEMA CREATE Global Schema name, Project, Script, Component, In-output Output when a global schema is registered or updated.
  • Log is output if a global schema is registered or updated from a designer or a mapper.
DELETE Global Schema name Output when global schema is deleted.  
UPDATE Global Schema name, Owner, Group, Access permission mode Output when global schema setting is updated.
  • Log is output if global schema setting is edited from control panel.
SCRIPT LOCK Project path, Script name Output when a script is locked.  
UNLOCK Project path, Script name Output when a script is unlocked.  
SERVER REBOOT Time for Executing Reboot (in millisecond) Output when DataSpiderServer is rebooted.  
SHUTDOWN Time for Executing Termination (in millisecond) Output when DataSpiderServer is stopped.  
SERVER_SETTING EXPORT Export destination path Output when the server setting is exported.
  • The export destination path is not output if the server setting is exported from the WebConsole.
IMPORT Import file path Output when server settings are imported.  
SERVICE COPY Copy source service name, Copy destination service name Output when a service is copied.  
DEPLOY Service name Output when a project is registered as a service.  
RENAME Old service name, New service name Output when a service name is modified.  
UNDEPLOY Service name Output when a service is deleted.  
SESSION LOGIN Session ID, Locale Output when logged in to DataSpiderServer.
  • Will not be output if login from the Studio for Web or the WebConsole is failed.
  • No session ID will be output if login to the Studio is failed.
LOGOUT Session ID Output when logged out from DataSpiderServer.
  • If logged out from the WebConsole, [TERMINATE] is output instead of [LOGOUT].
TERMINATE Session ID Output when the session is ended.  
TEAM_DEV_ENABLED UPDATE true or false Output when [Enable team development] is modified.
  • Output "true" if [Enable team development] is checked. Output "false" if not.
TMP_DIR DELETE   Output when a temporary directory is deleted.  
TRIGGER CREATE Trigger name Output when a trigger is created.  
UPDATE Trigger name Output when a trigger is modified.  
DELETE Trigger name Output when a trigger is deleted.  
TRIGGER_ENABLED UPDATE Trigger name, true or false Output when the trigger condition is modified.
  • Output "true" when the trigger is enabled. Output "false" if not.
  • If the trigger condition is changed in the Studio for Web, [UPDATE] in [TRIGGER] will be output instead of [UPDATE] in [TRIGGER_ENABLED].
USER CREATE User name, Primary Group Output when a user is created.  
DELETE User name Output when a user is deleted.  
USER_DESC UPDATE User name, Description Output when the description of the user is modified.  
USER_PERMISSION UPDATE User name, Access permission mode Output when the access permission of the user is modified.
  • The Access permission mode of message is outputted by an octal number.
USER_PRIMARY_GROUP UPDATE User name, Primary Group Output when the primary group of the user is modified.  

Date directory

Logs pertaining to the execution of script are found in the directories created with their names as current system date. Log
The date directories are the directories created with their names such as 'yyyyMMdd'. They are found under $DATASPIDER_HOME/server/logs.
The directory includes the following log files: The date directories are periodically zipped up for backup and archival purposes and are removed thereafter when the time specified for the removal of the archive elapses.

Creation,archive and removal of the date directory

When DataSpiderServer starts up or when script is executed, the date directory is created only if there is no precedently created directory with its name as current system date.
The archive and removal of the directory occurs at 12 hour intervals after DataSpiderServer is started.

In a case where a log file of more than 4GB(4294967295 bites) exists in the date directory, the system cannot compress log and data directory including the target file will be remained.
Please note that operations of DataSpiderServer will not be affected. Please manually compress or delete log.

Studio Log

Logging pertaining to DataSpider Studio.

An example of output

10/10 09:55:22|INFO|ds.boot|Starting system service...
10/10 09:55:22|INFO|ds.boot|System service was successfully started.
10/10 09:55:22|NOTICE|ds.boot|DataSpider Studio was successfully started.
10/10 09:55:22|INFO|ds.boot|*****************************************************************

Client CLI Console Log

Logging pertaining to Client CLI Console which includes command history and is output to the Client CLI Console Log.
This logging include the following items:

An example of log output

10/14 22:58:00|INFO|ds.console.cli|USER=root|START help
10/14 22:58:00|INFO|ds.console.cli|USER=root|END help
10/14 22:58:11|INFO|ds.console.cli|USER=root|START pwd
10/14 22:58:11|INFO|ds.console.cli|USER=root|END pwd

A formatting example of the log

Date | Log level | Module name | User-name | Execution command

Studio Standard Output

DataSpider Studio process are redirected to the Studio Standard Output from the standard output.
The same is logged to the Studio Log.

An example of output

--- |INFO|Starting system service...
--- |INFO|System service was successfully started.
--- |NOTICE|DataSpider Studio was successfully started.
--- |INFO|*****************************************************************

Studio Standard Error Output

Errors occurred in DataSpider Studio process are redirected to the Studio Standard Error Output from the standard error output.
No information will be output unless you encounter some serious problems.