Delta Watch: Import & Export module

  • Importing data like: Delta Link operational status, Total Station performance?
  • Exporting processed data to a third party database?
  • Importing data from geotechnical sensors?

The Import & Export module can take care of that.

This guide explains the ease of use and flexibility that we put in your hands.

Acceda o inscríbase en myTopcon para ver esta página:

Delta Watch: Import & Export module

Delta Watch has got a user configurable Import & Export module with lots of flexibility. With this module, as the name suggest, data can be imported and exported. Data export could use either preconfigured or a custom defined format. The last is particularly useful when there is a need to provide third parties with a continues flow of processed data of all sorts.

The module can be found under the Interface group in the Navigation bar  and is called Import & Export, see image on right.

All data Import & Export tasks are created and configured in this module, and data flow monitoring can easily be enabled. Tasks can be scheduled to run at intervals or, for exports, directly after a calculation. These tasks will run independent of other processing tasks.

The custom-export-format allows for configuring items like:

  • Dynamic filenames
  • (Custom) Headers
  • Date/time format
  • Column separator
  • Data value type; resolution and units

 

The IMPORT capabilities of this module are used for items like:

  • Weather station sensor import (*.ENV or *.THP)
  • Delta Link Control unit information (*.CUI, *.MDM)
  • Total station information (*.STNI and *.BUB)
  • Geotechnical sensors (Tilt, water level, inclinometer etc.)

The export options are endless and can be customized to fit the project specific needs.

Delta Watch also has a restful API interface that could be used to get data out of the DB instead of exporting files. Contact support for more details.

 

Additionally, alarms can be configured to trigger events, such as when no data has been sent or received for a certain amount of time. The Import & Export module also provides a status overview showing, for example, age of last action. The image below shows the status overview in the module. Similar, but less detailed, info is available in Delta Live! on the dashboard. This will only show the last time data was imported or exported.

What is in this article

  • The basic workflow. jump to
  • Configure Import task. jump to
  • Creating custom export format. jump to
  • Configure Automatic Export. jump to
  • Using the manual Export function. jump to
  • Troubleshooting. jump to

The basic workflow

The Import & Export module is always used in conjunction with other modules, and because of that it should not be the first item to configure when creating a project or wanting to import data. Sensors will always need to be configured in their respective modules first before their data can be Imported or Exported. Details and extra instruction for module specific import tasks can be found in the help or How to documentation for that particular module. That leaves use with the following guidelines for a basic workflow.

  1. Require an active project, even if just importing system diagnostics information.
  2. Prepare and configure the module of interest.
  3. For import:
    1. Create the import task, make sure there are some data files.
  4. For Export:
    1. Need to have data in the DB
    2. Decide on the export format
    3. Check if the correct export format exists, if not add a custom format
    4. Do a manual export, or preview, to check the format of the file and data
    5. Create the export task
REMEMBER: Delta Link data files always use UTC for any timestamp.

 Configure Import task

Import configurations are used to import data from one or more external sources at regular intervals, with options to monitor the data flow and raise alarm on specific values. This could be Delta Link specific data, or data from other geotechnical sensors. Below some examples.

 

Delta Link information Geotechnical sensors
  • Basic Weather station data (*.ENV)
  • Control unit information (*.CUI)
  • Modem information (*.MDM)
  • Station information (*.STNI)
  • Compensator data (*.BUB)
  • Tiltmeters
  • Water level sensors
  • Inclinometers
  • Extensometers

 

This article is limited to explaining the general configuration of an Import task. For more details on each of the above items, there are separate articles that look at the specific requirement for a sensor. So, here is the general workflow to create an import task.

  1. Open the Import & Export module, found under the Interfaces group.
  2. Click on Add button to add a new task.
  3. Follow the order of the fields in the table below to complete the configuration.
  4. When completed: Close the configuration window and click Apply to save the changes.

 

For most sensors, only one import definition per project and/or import folder could to be created. This will automatically process specified compatible files. Separate import tasks only have to be configured if:

  • Data arrives in separate folders.
  • Different alarm limits should be applied.
When importing Delta Link system diagnostics files, it is strongly advised to create an import task per data-type per unit. This allows for much more flexibility in regard to alarming or when a particular Delta Link or Total Station will move to a different network or project altogether.
 
Illustration below of import task configuration: A) Used for TS information, B) Used for geotechnical sensor data

Table with basic fields when configuring import task.

Overview of configurable fields for data import

Basic fields for import task configuration
Setting Value

Name

A user-chosen name for this import definition

Import/Export

Set to Import

Sensor type

Set to correct sensor type, see below table for more details.

Base directory

Directory where the data files can be found.

Time interval

At what interval to check for new data. If this is for CUI data and alarms are configured, then a relatively high interval of 5 or 10 minutes could be set.

Import type

Field changes based on selected sensor type above. See below table for extra details. Additional information can be found in sensor specific articles elsewhere on myTopcon.

Select project

The destination project for which this import is, not all Sensor type imports are project specific, see below table for details.

Select Limit

If a Limit was configured in the Limit Monitoring section of the Point and Limit management module, then select it here to be used. Data is checked upon import.

Select sensor

Depending on the imported Sensor type, specific sensors need to be selected.

Sensor types and their options
Sensor Type Import type Time interval Selection
Control unit (info) Control unit info (*.cui) Selectable Sensor only
Modem log file (*.mdm)
Total station (hardware info) Total station info (*.stni) Selectable Sensor only
Compensator (*.bub
Inclinometer Simple csv (*.csv) Selectable Project+Sensor
Extensometer Geoscope customized Selectable Project+Sensor
Comma separated values (*.csv)
Water level Geoscope customized Selectable Project+Sensor
Comma separated values (*.csv)
Tilt meter Geoscope customized Selectable Project+Sensor
Senceive (*.csv)
VMT data format (*.csv)
Strain gauges Geoscope customized Selectable Project+Sensor
VMT data format (*.csv)
Meteorological Meteorological (*.env) Fixed 10 minutes Sensor only
Levelling data Simple csv (*.csv) Selectable Project+Sensor
Sixense 1D [in] (*.txt)

Extended settings (optional)

These settings allow a little more control and configuration of the data, especially if the data is older data or created with local timestamps instead of the standard expected UTC time.

Extended settings
Setting Value
Active allows activating/deactivating this import task
Start time Specify the time for the start of the data. Normally this is in the future but could be a time in the past matching the date/time from which point onwards the data needs to be processed. Any data from before this time will not be processed.
Data time zone If the timestamp in the data files is NOT in the UTC time zone, then specify here the correct time zone so that Delta Watch can convert the timestamp information correctly. By default, Delta Watch assumes any timestamp is in UTC time zone.

Monitor data transfer (optional)

Configure to get an email message triggered by the data flow. For example: An email could be sent when data has not arrived for a certain amount of time. Click on  to Add one of the project recipients and define the amount of time after which the message should go out. If the list of recipients is empty, then configure these first in the Contacts section of the Project module.

 


Finished

Once all is configured and saved to the database, Delta Watch might directly start with the import, depending on the start time and interval settings. The Import status can always be seen in this module in the columns Latest datatransfer and Total datatransfer. Below figure shows an example of this.


 

Creating Custom export format

The import and Export module allow for easy creation of so called flat-files, or CSV files. A configuration is made for a specific sensor type, and one or more of the following can be configured:

  • Filename layout
  • Multiple custom date and timestamp formats
  • Custom order of columns
  • Automatic or custom header
  • Export units
  • Resolution of values
  • Fixed text fields can be added
  • Column and decimal separators

There are a few pre-defined export formats available, these are mostly used for backwards compatibility with other systems that require a very specific format.

Section A) shows how to configure the file format, and section B) shows how to define custom filename and timestamp format strings.

 

Upgraded from a Delta Watch pre v3.x?

Make sure you have disabled ALL network exports before upgrading!  Alternatively, login to Delta Watch as Administrator and navigate to the System Configuration section. Under the Support section, use the function to Deactivate all exports from geodetic networks. Most data export was done from Automated networks, but if you were exporting from other modules, then you will find each format back in the export module once you select the Sensor type.


 

A.  Configure the format

A format definition can be seen as a template that defines what data should go in which columns, the type of units and number of decimal places that should be used. A format definition can be used by multiple export tasks. This section shows how to define a Custom format.

  1. Open the Import & Export module, found in the Interfaces group.
  2. Click the Custom Format button in the toolbar. A new dialog window opens, like the illustration below on the left, showing all current available custom formats.

 

Want to see an output example?

Select the item of interest under Configurations and the overview will update with the specific details. Click on Preview to generate a dummy file.

  1. Click either Add Edit or Duplicate Add
  2. A configuration screen opens where the new format can be configured. This will look something like the in the above illustration on the right. All fields need to be configured; the below table gives further details.
  3. Click OK to save the configuration. Now it can be used for any exports.

extended field details
Settings Details

A user-chosen name for this format definition.

Currently only .CSV export possible.

Selecting the correct type will ensure only applicable data fields are shown and that the definition only shows when creating an export for this sensor type. The following options are currently available:

  1. Geodetic sensor
  2. Meteorological
  3. Structural monitoring
  4. Convergence section

Defines the format type. Currently only Export can be selected.

Select an existing formatting using the drop-down control, or add/edit a format.
To Add or Edit, click on Add format and a dialog window will open where formats can be managed.

When clicking on EditEdit or AddAdd an editor will show, as per image.
Options vary depending on the Sensor type of the current Export format definition.

custom filename definition for data export

Specify the character to use. Type [TAB] for the tab character

Select dot or comma

Select an existing formatting using the drop-down control, or add/edit a format.
To Add or Edit, click on Add format and a dialog window will open where formats can be managed.

When clicking on EditEdit or AddAdd an editor will show, as per image. Timestamp format can now be defined using standard ISO codes.

Custom timestamp format configuration for exporting the data

If enabled, then a header will be written as the first line of each data file. This can be an automatic header based on the chosen data columns or a custom header.

When creating a custom header, do not forget to include the column separator and unit indicators.

 

By default, only 5 columns will show, and the first one will always be the timestamp.

  1. Add/Remove columns as required, the action is always performed on the last column. Inserting or selective delete is not possible.
  2. For each column, were applicable, specify:
    1. Column type
    2. Unit
    3. Decimal places
    4. Text (only when fixed text is selected)

 

 


Configure Automatic EXPORT

The Export tasks are replacing all automated exports from pre Delta Watch v2.15 versions.

Once the module that is processing the source data is configured, and preferably has already done some processing, it is time to create the export task. Follow the below steps to do so.

 

  1. Add a new task definition by clicking on the AddAdd. See below illustration
  1. A configuration form will show where the new tasks can be defined. The below illustration shows 2 typical configuration examples, and the table below it clarifies the available fields.
  2. Click OK to save the new configuration, then click Apply to write to the database.
 
Illustration left: Example of Geodetic data export, performed directly after an epoch calculation.
Illustration right: Export definition example for (any) geotechnical sensor, exported at hourly intervals.
Available field when configuring any data export
Setting Details

Memorable name for this task.

When naming tasks keep in mind that a task can only export from one project but can export multiple Point groups at the same time!

Set to Export

The following options are currently available:
  1. Geodetic sensor
  2. Meteorological
  3. Structural monitoring
  4. Convergence section
  5. Rail track
  6. Tilt meter

Specify the destination folder.

Ensure a unique export folder per task so that data is not accidentally mixed or overwritten.

Together, these fields specify when data export should happen.
Enable Extended settings to see all options.
The Start time has two functions:

  1. Set the date-part in the future so that the task would not start until it reaches that date.
  2. Specifying the time-part to control base time of the export.

    For example;

    • With a Start Time of 00:10:00
    • A Time Interval of 60 minutes
    • The task will export at:
      • 00:10:00
      • 01:10:00
      • 02:10:00
      • 03:10:00
      • …….
      • 23:10:00

Export after processing is available for some export types and, when enabled, the Time Interval field is ignored, and export is triggered by the respective module after processing new data.

 

By default, the timestamp of the exported data is in UTC. Change this field if the date/time stamp should reflect a different timezone.

A list with export formats. Any custom export formats for the selected sensor type will also be visible here.

Select from which project the data will be exported

Depending on which sensor type is exported, either a sensor-group or a sensor-field might need to be selected.

Multiple entries can be selected, but if selecting point groups all the data will be written to the same file unless the filename definition has the [Point Group] field enabled. Otherwise this can give unwanted results, like duplicate entries.

Complete this section to get a message on certain actions.

 

The mail server needs to be configured and the project must have one or more Contacts.

 

The following triggers are available:

  1. Data transfer
  2. Incorrect Export
  3. Confirmation (that data was exported)

 

 

 

Using Manual Export function

With manual exports, the user can perform a one-time export while defining things such as the time frame and the specific (user-defined) format. Or just re-export a certain dataset.

  1. Click the Manual Export button in the toolbar of the Import & Export module, see below illustration.
  1. A configuration window opens. Make the configuration for the following fields:
    1. Select Project
    2. Sensor type
    3. Export type
    4. Export folder
    5. Used Timezone
      (Timestamp of the Exported data will be in this timezone)
  2. Depending on the selected Sensor type and the current definitions in your DB, for items like convergence, the dialog box shows different options. These could be point groups or sensor fields. Multiple point-groups can be select at once, but all the data will be written to the same file, unless the filename definition had the field [Point Group].
  3. Select the Network and Time Interval, different fields will show depending on the selection here.
  4. Configure the options and click on Display to load all available epochs for selection.
  5. Use the mouse right-click context menu to select the Start and End epochs.
  6. Click OK.

Exporting will start, when completed a message will show indicating how many files have been written.


Troubleshooting - Data Import

All troubleshooting for Imports will be because there appears to be No data. Below some points to check: 

  • Age of last data transfer for each task as shown in the Import and Export module.
    • Very recent...
      If not long ago, then the data file might have been empty, or charts are not scaled correctly when attempting to view the data. Locate the file in the pro-ject Archive folder and check its content.
    • Long ago...
      More problematic, could be anything that stops data flow from Delta Link turned OFF to FTP server not running, or no internet connection.
  • Check the message archive and look for import failures, how?
    Open the Message archive and check if there are any messages relating to data import. If there were incompatible, faulty or corrupted files, then a warning could be found here. Depending on the last import more than 100 messages need to be loaded. Make sure to check and use the column filters.
  • Are the data files waiting in the import folder:
    • Yes...
      • If this is the first time to import, be patient and let Delta Watch execute the task as per time interval setting.
      • Is the import task set to Active?
      • Does the import task points at the right import folder.
      • Is the Start date equal or older than the date of the first data file.
      • Depending on import type, is the correct project and/or sensor selected
      • Is the filename correct, for example, no uppercase file extensions!
      • Does the file have actual data?
      • Check the file format and sensor names, use a previously accepted file from the archive to compare with if possible.
      • Is the file corrupt, like unexpected and of file.
    • No...
      • Check the data delivery or, in case of a Delta Link, attempt to connect to it using the web-portal. If no connection can be made, then a site visit to investigate is required. Check the last records from the CUI file to see if the power source might have starting to fail.
      • Files might have been archived but not processed. Most likely reason is a change in name or other details somewhere. Check the full filename, text is case sensitive.

Troubleshooting - Data Export

Troubleshooting for data Exports can be narrowed down to one of the following:

  • No (new) data
    • No data to calculate an epoch.
    • Epoch calculation failed.
    • Check age of last export, if not recent but epochs exist, then might point to missing reference points or other epoch calculation failure.
  • Destination folder was moved/renamed/deleted
  • External drive is not available
  • User had no rights to write a file
  • Export task checks
    • was de-activated
    • Has a start time in the future
    • Filename definition is wrong, i.e. each export overwrites the same file
  • When exporting Geodectic data
    • Point group deleted or adjusted.
    • Points in the point group were not calculated this time.
    • Local coordinate system wrongly created or assigned to the point group
  • Export did happen, accoording to message archive
    • File was already consumed by another program (and failed processing)
    • File was accessed by another program while still being written