Valido 4.6: new integrations and algorithm updates

TL;DR

The Validio 4.6 release is now generally available! This release makes the setup of sources and validators more scalable, introduces new integrations, and the option to decide the adaptation rate for dynamic thresholds. You can find all the details in the changelog, but here’s a snapshot:

One-click setup for sources and validators

Following the improved validator workflow in the 4.5 release, we’ve made additional improvements to streamline, speed up, and scale the process of creating sources and validators.

Now you can:

• Automatically create sources per table from data warehouse sources, and add metadata validators as part of the source creation flow.
• Bulk-convert catalog assets: convert multiple catalog assets into sources and metadata validators at the same time.
• Bulk-create validators: for example, add null count validator and then multi-select from a list of fields in the schema to apply null checks to each selected field.

Email notifications

If you prefer email over chat, you can now receive the latest notifications about incidents over email. The email can include graphs or just text, and be sent directly to the stakeholder who needs to know about data issues.

Integration with Sigma

Validio's integration with Sigma’s business intelligence tool enables organizations to seamlessly validate and monitor data from Sigma's BI platform. Validio fetches catalog and lineage information from Sigma to display directly in Validio's data insights and lineage.

Integration with Atlan

Atlan stores tables, views, columns, and lineage as assets in its data catalog. The Validio integration sends data quality metrics at regular intervals to Atlan, where you can view the metrics alongside each asset.

Flexible adaption rate for dynamic thresholds

The dynamic thresholds are a core part of Validio’s data observability, using a combination of algorithms to automatically detect anomalies in your data. The threshold model infers trends, seasonality, and also adapts to shifts in your data. Based on customer feedback from using Validio to monitor business metrics, we’ve added the option to configure how quickly the dynamic threshold's bounds adapt to changes in your data or metrics. This is controlled by the adaption rate parameter, which offers two settings:

1. Fast: The bounds of the dynamic threshold will adapt quickly to recent changes in the data.
2. Slow: The bounds of the dynamic threshold will adapt more gradually, placing more emphasis on historical data.

The best choice between Fast and Slow depends on your unique use case and the characteristics of your data, but the best part is that you can easily switch between the two. 

Bounds of dynamic threshold with different adaption rates. Bounds for Fast widen and narrow more rapidly with changes in data, while the bounds for Slow are more persistent in following the downward trend and triggering alarms when the trend changes.