Background

This issue describes requirements to the GUI for the functionality implemented via #2721.

Approach

Admin and owner shall have the ability to configure the transition rules for object storages (currently, only for AWS s3).

• Create/Edit rule
• View rules
• Rule logs
• Notifications

Separately, there shall be the ability to restore files, previously transferred according to the transition rules, back to the s3 storage.

• Restoring files
  • Restore file
  • Restore whole folder

Create/Edit rule

Ability to manage storage rules shall be available via the Create/Edit data storage form.
A new tab "Transition rules" shall be added:

Note: as unnecessary, the controls that previously managed storage lifecycle - STS/LTS duration shall be removed from this form

By click this new tab, a new form shall be opened, e.g.:

This form shall contain:

• button to add a new transition rule for the storage
• a list of all transition rules configured for the current storage (empty list by default)

To create a new rule, admin/owner shall click the +Create button.
A new form shall appear, e.g.:

Here, the rule settings shall be specified:

• Root path (mandatory) - it is the path inside the storage, from which the search for files to transition (corresponding to the specified glob pattern) will be performed. Example: /, /some_folder/sub_folder/, /some_folder/*
• Glob (mandatory) - glob pattern. Pattern that shall define files that will be transferred by the current rule. Pattern shall be applied in the storage path specified as root. Example: *.csv - any CSV file in the current root path, /**/*.csv - any CSV file in any subfolder of the root path.
• Method (mandatory) - method that defines the transition process. The control shall be a dropdown list, multi-select is disabled. Possible variants:
  • One by one (default) - each file matches the glob pattern will be transferred separately. For each such file separate notification will be sent (if notifications are enabled)
  • By the earliest file - all files match the glob pattern will be transferred simultaneously - when the earliest file from them corresponds to all conditions of the transition rule
  • By the latest file - all files match the glob pattern will be transferred simultaneously - when the latest file from them corresponds to all conditions of the transition rule
• Condition (mandatory) - additional condition that shall be met to initiate the transition procedure. The control shall be a dropdown list, multi-select is disabled. Possible variants:
  • Default - no additional conditions are required
  • Files matches - if this condition is selected, a new field appears - to specify additional glob pattern. In this case, transition of the files that match main glob pattern can be performed only after in the root path the files will appear that match this additional glob pattern. Example - rule for transition of CSV files in case of TSV files appear:
    
• Transitions - there, shall be a list of destinations where exactly files shall be transferred and in which dates. By default, at least one item shall be in the list. Each item shall include fields:
  • Destination (mandatory) - destination where files matches the glob pattern shall be transferred. The control shall be a dropdown list, multi-select is disabled. Possible variants: S3 Glacier Instant Retrieval, S3 Glacier Flexible Retrieval (formerly S3 Glacier), S3 Glacier Deep Archive and Deletion
  • Date (mandatory) - date when files matches the glob pattern shall be transferred. There are two variants (only one can be selected for the rule):
    • specifying the count of days (from the file creation) after which the transition will be performed. For example, if in this field the count 30 is specified - so files match the glob pattern in the root path will be transferred to S3 Glacier Instant Retrieval after 30 days after their appearance in the current storage:
      
    • specific date. If the specific date is selected - files match the glob pattern in the root path will be transferred exactly in that date:
      

Additional destinations can be created by the "+ Add" button. There, 4 destinations can be maximum (according to the currently available possible AWS destinations). When user adds a new destination - in the corresponding Destination dropdown list, only destinations shall be available that were not used in the previous items. After, there are all possible destinations in the list, "+ Add" button shall disappear. There, shall be the ability to remove destination item from the list. Example of the several destinations in the list:

• Notify - here, the settings for messages about files transition can be configured. In based scenario, recipients shall receive notifications about files transition before the period specified as "notice period". Users can click Prolongate button from the notification (message body) - in this case, transition will be delayed for days specified as Prolongation period, after prolongation period is over - a new notification shall be sent again. Prolongation can be repeated any times.
  • Disable all notifications for the current rule checkbox - if it is set, notifications for the current rule shall not be sent at all (regardless of the performed/prolonged transitions). If it is set, all controls below in the section shall be disabled
  • Recipients (optional) - list of notification recipients. If this list is empty - no one will receive notifications (except those users that are specified in the email notification settings)
  • Notice period (optional) - period in days before the date specified in Date field - when the notification about further transition will be sent. Note: if it is not specified - the default value shall be used from the email notification settings during the rule performing
  • Prolongation period (optional) - period in days for which the transition will be delayed in case when user clicks the Prolongate button. This prolongate period shall be applied explicitly to files for which notification is received. Note: if it is not specified - the default value shall be used from the email notification settings during the rule performing
  • Notification - expanding section that shall include fields for templates of the notification subject and body. By default, these fields shall be hidden. Instead of them, the enabled checkbox "Use default template" shall be shown. In this case, values from the email notification settings will be used when building message. Once user unselects the checkbox "Use default template", fields for notification subject and body shall appear, these fields shall be filled in by the corresponding values from the email notification settings. Additionally, user shall have the ability to view/edit these fields and also preview - how the notification will be displayed in the email. If the notification subject/body will be changed via this section - it shall impact only to the ending view of the current rule notifications and shall not change the default templates in the email notification settings. Expanded section shall have the same view as it was implemented for the EMAIL NOTIFICATIONS tab of the System Settings, e.g.:
    

When the new rule is created, it will appear in the list, e.g.:

This form shall contain the list of all transition rules configured for the current storage.
For each rule, there shall be:

• short info (Root path, Glob, Transition destinations)
  Note: if for the same rule, there are several destinations were specified - only the first shall be shown in the table, all other shall be displayed as a hyperlink in the format + <additional_count> more, where <additional_count> = count of specified destinations for the rule - 1. For 3 destinations for the same rule, see an example on the image above. By click such link, the list of additional destinations specified for the rule shall appear in a tooltip
• the control to edit a rule. By click this button, a form like described above shall be opened with the rule settings
• the control to view rule logs - logs of all transitions/prolongations performed via the current rule (see details below)
• the control to remove a rule

Example of the several rules for a storage:

View rules

If user opens any folder in the storage, and for that folder any rule is configured - that info shall be displayed in the Attributes panel of the folder, e.g.:

• a rule should be considered to belong to this folder if the folder is specified as Root path in that rule
• if no rule has the current folder specified as Root path - this info label shall be hidden

Rule logs

By click the Logs button for any rule - a new form shall appear. This form shall include logs of all transitions/prolongations performed via the current rule, e.g.:

That form shall contain:

• info fields - short info about the current rule (Root path, Glob)
• Action type dropdown list - to filter logs table by performed action (it may be transition/deletion/prolongation):
  • If the rule has transition destination S3 Glacier Instant Retrieval, S3 Glacier Flexible Retrieval, S3 Glacier Deep Archive, this dropdown list shall have only items Transition and Prolongation. Empty value is allowed - in this case, logs table shall not be filtered
  • If the rule has transition destination Deletion, this dropdown list shall have only items Deletion and Prolongation. Empty value is allowed - in this case, logs table shall not be filtered
• logs table with columns:
  • Date - datetime of the event
  • Action - it may be one of the system actions - Transition or Deletion, or manual user action - Prolongation
  • User - in case of system actions, there should be system stub. In case of Prolongation, there should be a user name who has prolonged transition for a file(s)
  • Path - path to the file/folder, with which the action was performed
  • Destination - transition destination
  • Prolongation - period in days for which transition/deletion was delayed (Note: only for prolonged files)
  • Renewed transition - a new date when transition/deletion shall take place (Note: only for prolonged files)

Notifications

A new notification type shall be added to Email Notifications (e.g. DATASTORAGE_LIFECYCLE_ACTION).
Default template of the notification shall include subject, text, recipients. Template shall contain a link to delay data transition.

Also via the notification settings, there shall be the ability:

• to define default notice period (i.e. when data is close up to N days to transition). This date can be overridden by the specific rule field
• to define default prolongation period. This period can be overridden by the specific rule field

Restoring files

If there are files in the storage folder that were previously transferred to any S3 Glacier destination according to some transition rule - admin or owner shall have the ability to restore such files.
This ability shall be available:

• for separate files
• for whole folder. In this case, all previously transferred files in that folder shall be restored identically. Also, all transferred files in sub-folders of the folder, should be restored too - recursively.

Restore separate file

To restore a file - user should select it.
In the file management menu (above the files list), a new button shall appear, e.g.:

By click this button, a pop-up shall appear, e.g.:

Here, user shall specify:

• the period duration for which the file shall be restored. Restore duration period can not be empty. Default value for this field shall be received from the preference
• recipients who shall be notified about file restoring process (by email notifications). This field can be left empty - in this case, file will be restored without notifications
• additionally, only in case when the storage is versioning storage, user shall have the ability to define - restore only the latest file version (default behavior) or all versions - via the corresponding checkbox:
  

After period and recipients are specified - user shall confirm to proceed.

Note: user shall have the ability to restore several files simultaneously in described way. In this case, user shall select desired files and click the same button as for the single file. Pop-up to confirm restoring shall look like:

All selected files will be restored for the same period

If restoring for a file is requested, in the file row, the restoring status shall be shown.
There are possible states:

• RESTORING - when restore operation is in progress. This status shall be shown after the request for the restore was submitted and until the file is restored successfully or restoring is failed. Status shall be shown as additional label in the file row, e.g.:
  
• RESTORE FAILED - when restore operation is failed. This status shall not be shown at the GUI
• RESTORED TILL <DATE> - when restore operation is completed. This status shall be shown after the file is restored successfully and till the date when the recovery period is finished. Status shall be shown as additional label in the file row, e.g.:
  

Note: when hovering such status label, the tooltip with the corresponding status text shall appear

Example of statuses:

When restore period is over, restore status shall disappear.

Restore whole folder

Restoring of the whole folder (i.e. recursively restoring of all files in that folder, previously transferred to some S3 Glacier) shall be available via additional control in the Attributes panel of the folder, e.g.:

By click this control, a pop-up shall appear, e.g.:

Note: options and settings for that pop-up shall be similar to the pop-up of the single file restoring

After restoring for a folder is requested, in the Attribute panel, the restoring status shall be shown.
There are possible statuses:

• Restore in progress - when restore operation is in progress
• Restore was failed - when restore operation is failed. This status shall not be shown at the GUI
• Restore is completed. Folder is restored till <DATE> - when restore operation is completed. <DATE> - the end date of the recovery period

Example:

When restore period is over, restore status shall disappear.

Other options

• All S3 objects lifecycle events (transitions, prolongations, deletions) shall be additionally logged in the System logs of the platform. Restoring operations shall be logged in the System logs too. For details see Extend Security Logs with Storage Lifecycle Events #2797
• To the Cloud Regions settings, add a new field - "Storage Lifecycle Service properties" - to save config of the lifecycle service. It shall be a text input field, that waiting for input in JSON format, e.g.:
  
  Details:
  • that field shall be in settings form of each region
  • value of that field shall be optional, field may be left empty
  • if some properties were previously saved for that region - they shall be displayed in the field when opening form
  • API details see in Issue 2721 Storage Lifecycle Service: move config to region #2830