DataSpider filesystem

DataSpider Servista provides its own implementation of a filesystem abstraction layer. The purpose of this abstraction layer is to allow for DataSpider Services to access different types of concrete filesystems in a uniform and platform independent way without having to know what type of filesystem they're accessing.

Concept of DataSpider filesystem

Platform independent
Regardless of which platform a script is made, whether it is UNIX/Linux or Windows, it can be executed on whichever platform the script is loaded,
because the filesystem abstraction layer offers the flexibility to change the platform a script is executed, without any modification to the implementation of file handling operations in a script.

ACL
DataSpider filesystem provides access control list for file and directory access privileges, thereby it is possible to protect against tamper or deletion if multiple users work simultaneously.

Data consistency
Compared to ordinary files, the risk of inconsistency occurring or data loss is reduced by the use of database whose integrity provides the validity and consistency of stored data.

DataSpider filesystem architecture

DataSpider filesystem architecture is represented in the diagram below.

Component Layer

Component layer consists of components which interact with DataSpider filesystem's abstraction layer.
These components include DataSpider Studio, CLI Console, and adapter components which fall into File adapter category.
All of these components perform indirect access to the filesystem through DataSpider filesystem's abstraction layer via its specified interface between the component layer and the platform filesystem or a database.

DataSpider filesystem Layer

DataSpider filesystem Layer provides a common file model that is capable of representing conceivable filesystem's general features and behavior.
There are three types of DataSpider filesystem, each controls data flow between the component layer and the physical layer by its own specific way.

Physical Layer

This is the layer that contains the actual data.
It could be the platform dependent filesystem or a database, or both, depending on how DataSpider filesystem is configured.

Types of DataSpider filesystem

There are 3 different types of DataSpider filesystem.

filesystem Types	Description	Notation used in [Type] attributes in Explorer
Local filesystem	Data is stored in the OS's filesystem.It performs the fastest, but it cannot save the file meta-information such as owner and permissions of a file, or whether a file is write-protected by the file lock feature or not. If there is no repository database used, Local filesystem it the only option available for the filesystem type.	LOCAL
Extended local filesystem	The file meta-information is stored in the repository database and data itself is stored in the OS's filesystem.It does not perform as fast as the Local filesystem. Can be used for the system with a repository database.	LOCAL\|EXTENDED
Database filesystem	Both the file meta-information and the file data are stored in the repository database table. the maximum file size supported varies depending on the DBMS used. Can be used for the system with a repository database.	none

DataSpider type of filesystem is synonymous with the type of mount.

OS local filesystem

OS local filesystem is a term we use to distinguish the actual filesystem of the OS from the aforementioned abstraction filesystem layer provided by DataSpider filesystem.

There are two terms we use for this OS local filesystem, accordingly to which part of the DataSpider Servista is running, whether it is DataSpiderServer or DataSpider Studio.

Server OS local filesystem
- Refers to the filesystem of the OS in which DataSpiderServer is running
Client OS local filesystem
- Refers to the filesystem of the OS in which DataSpider Studio is running
- Refers to the filesystem of the OS in which DataSpider Studio for Web and WebConsole is running

Mount

Mount is a mechanism used to associate the DataSpider filesystem with a particular point in the OS local filesystem's hierarchy or a database table. The associated filesystem directory in the OS local filesystem's hierarchy is called mount point

The association of a mount point and DataSpider filesystem can be done in Mount Settings in the Control Panel.

System Mount

There are number of directories in the OS's filesystem's hierarchy associated with the DataSpider filesystem by default when the system starts up:
System mount point is as follows.

When there is use of a repository database

Database filesystem
- /etc
- /home
- /share
Extended local filesystem
- /data
- /ftp
Local filesystem
- /samples
- /schema
- /tmp

When no repository database is used

Local filesystem
- /data
- /etc
- /ftp
- /home
- /samples
- /schema
- /share
- /tmp

All of these directories cannot not be associated redundantly with DataSpider filesystem by the user operations.

Restrictions on characters used in DataSpider filesystem

Use of the following characters and strings are disallowed in DataSpider filesystem.

\ / : ; * ? ' " < > | ~

Platform dependent characters and external characters

Use of characters and symbols that are not common in both Windows and UNIX/Linux platform are restricted.
Additionally, the tilde sign, ~ is disallowed as it represents a shot cut that points to a user's home directory in some platform.

Case sensitivity

The following table presents how in DataSpider filesystem case sensitivity plays a part.

Item	Filesystem Type	case sensitive or case insensitive	applicable fields
Files and directories found under /etc, /home, /share (when a repository database is used)	Database filesystem	Complies with the database you use and settings of that	Project name Service name Global Resource name Global Schema name Trigger name Holiday settings calendar name In the file path string to the log output destination. In the name of a user or a group(when a repository database is used)
Files and directories found under /etc, /home, /share (no repository database is used)	Local filesystem	Depends on OS platform in which DataSpiderServer is running Windows version: Case insensitive UNIX/Linux version: Case sensitive
Name of files and directories to be stored, except for /etc,/home,/share	Local filesystem and Extended local filesystem	Depends on OS platform in which DataSpiderServer is running Windows version: Case insensitive UNIX/Linux version: Case sensitive	-
Names stored in configuration files.	-	Case sensitive	Script names Folder names Script variable names Environment variable names Names of icons Names for log search criteria
Names of files and directories stored under associated mount points	Database filesystem	Case Sensitive	-
	Local filesystem and Extended local filesystem	Depends on OS platform in which DataSpiderServer is running Windows version: Case insensitive UNIX/Linux version: Case sensitive	-

It is encouraged to be mindful of case sensitivity issues when developing application in order to avoid case-related problems when performing server migrations to different OS platforms in the future.

Specifications may change depending on the version.

Specification Limitations

Limitations in mount operations
- Directories that are immediate children of the root "/" directory can only become mount points.
  Directories that are not associated with a mount point will be deleted at restart.
- It is discouraged to use the following directories as mount points.
  - /home,/home/<username>
  - Directories found under /home/<username>/projects and /home/<username>/projects
  - /etc and directories found under /etc
  - /share and directories found under /share
- A directory which is specified as a mount destination needs to have its parent directory.
- If you have booted DataSpiderServer using the Windows service and would like to mount a directory in the network drive, specify the user which has access permission to the network directory as an account name when you log on to the Windows service. After rebooted DataSpiderServer, you need to specify the directory path in the network drive in a UNC name format to the mount source.
No file can be created immediately under the '/' root directory of the DataSpider filesystem.

Any relative file path set in a settings of a component is relative to the directory that is selected in the file and directory Selectr or in Explorer.
It only affects the execution of the client tools such as designer .

The consistency of the data written to the same file by multi-threaded operation is not guaranteed.

Depending on the environment, when OS local filesystem is referenced by the directory Selectr from Studio, a dialog box warns about 'missing disk' may be prompted.
This can be ignored and the process is continued by pressing [Continue] button.

This issue has been confirmed, when DataSpider is hosted by Windows Vista running on VMware(not always)