Unmatched Performance
Here is the performance benchmark using snowflake warehouse.
How we achieved
- DQ rules are push down. We are not sucking the data out of warehouse for validating rules.
- DQ rules are specially fine tune for snowflake.
- All the DQ rules execute natively on snowflake.
- DQ rules are execute in parallel depending on the size of the hardware.
Data Sample
| Item | Description |
|---|---|
| Profiling | 16 columns |
| Data Quality rules | 18 rules |
Execution Timings
| Row count | Size | Compute Warehouse | Time taken |
| 6 M | 160MB | Snowflake x-small | 20 secs |
| 60 M | 1.6GB | Snowflake medium | 1 min |
| 600 M | 16GB | Snowflake large | 4 mins |