We're pleased to announce the release of Pandora 5.9.5.
We're pleased to announce the release of Pandora 5.9.4.
We are pleased to announce the release of Pandora 5.9.2.
This Pandora release introduces improvements to our export functionality, ensuring that rule validation occurs before export. This allows customers to fully automate the export of governance reports to an external database in one easy step. We have also added a feature so that you can persist changes when you reorder, hide, show, transform columns or add new columns in the column view.
Full list of new features:
This functionality allows users to fully automate the export of rule validation reports to an external database. Previously the way Pandora processed validations meant that customers could not automatically update and then export a drilldown with validation rules in it. The rules would not process and would display "Not evaluated".
Projects and sub projects are now listed in a parent/child format rather than listed out in one view. This is to make it consistent with behaviour elsewhere in the product and we believe it should be expected behaviour.
When making a saved drilldown auto-load as a new table, there are now options for what schema and project the new table should be put in.
Customers requested more choice on what they can export via JDBC. Previously, Pandora did not allow 'system' drill-downs, i.e. those that are not generated from user data such as the tables list, to be saved to a JDBC target. This restriction has been universally lifted.
The AUG data set and its associated DataPlus has been added to Pandora.
We will no longer support this driver as there are currently no live users.
Score and threshold columns have been moved so that they appear after column profile information by default. This means more valuable information will be displayed to users in the default view.
Unable to insert or update into JDBC database which have lower case column names
Drilling into validation rule results does not work correctly
Usernames can't start with a number with LDAP authentication
Rules over time intermittently incorrect
When loading a new version of a table, the low/high validation thresholds with decimals are rounded down
Ignore nulls does not work for Rules that use Table Expressions
Issue with mappings disappearing after editing
Scripts triggered from alerts require the admin user to be logged in
First List Value and Last List Value transformations fails with "Index: 0"
Strings containing backslashes used in Decision function causing validations to fail
After a Pandora Upgrade the SQL Connection details are lost
Server setting "Delete Import Files After Load" not functioning correctly
**Some Invert Validation Functions not working as expected **
Upsert Performance Issue
Database connection refresh dialog UI issues
Updated JDBC drivers
We are pleased to announce the release of Pandora 5.9.
The latest version of Pandora groups together several features improving the usability and extensibility of the product. We have added the ability to schedule and load multiple tables, which you can now shuffle to change the loading order. This is a powerful new set of functionality that expands the use cases for Pandora and significantly eases the use of Pandora in large scale migrations and projects. We have also added the ability to call an external web service from within Pandora which means customers can use external web resources to enrich their content.
Full list of new features (details on each available on the Pandora 5.9 documentation pages):
The web service call function allows connections to most types of web services enabling users of Experian Pandora to call the web services directly in Pandora. Functionality includes:
Results caching to improve performance
Exposing cached data for visualisation and searching outside of Experian Pandora.
Web services proxy setup in the Pandora GUI
Basic and OAuth2.0 (client credentials) authorization support
Logging to make it easier to debug
Any Web Service, that returns information in an organised and standardised structure, can be used. Experian Pandora can match JSON, XML, SOAP or Plain Text providing the response can be templated through configuration.
In our documentation, we have included an example web call to What3words that shows how you can pull data in from an external source to enrich your address data.
The Columns and Tables drilldowns now have columns displaying a table's projects, schema, source it was loaded from and the source type. This is to help in regulatory reporting where customers may want to be able to export the source and file lineage to data governance.
In previous versions of Pandora, a user would need to create a new rule threshold every time they wanted to validate a rule. A reusable 'Rule Threshold' can now be created and saved under a name, storing a low and high threshold value. For example, a user can create and name thresholds according to the particular quality metric required for a particular regulation.
The Columns and Tables drilldowns now have columns displaying the low and high validation threshold and threshold name for rules.
A user can now access a simple self-contained screen within Pandora that allows them to search and filter tables by name and multiselect tables to add them to a scheduled load. This feature vastly improves the usability of this functionality in the product (previously tables had to be manually entered by text). Tables can then be found with minimum effort and added or removed from the schedule as required. In addition to this, a user can now shuffle the order the tables are loaded in to better manage system resources and schedule.
A prefix and suffix to a table name can now be added when configuring one or more tables allowing, for example, users to create a group name or version across several tables. This can also be done for multiple flat files and manual JDBC loads, in the configuration and load dialog.
In previous versions of Pandora, when exporting data via JDBC, customers would have to map all the columns from their table in Pandora to the source. The target source columns would have to exactly match the columns in the table in Pandora or it would fail.
You can now map individual columns to the target which means you can update single columns. This is a huge improvement in export functionality that will allow clients much more flexibility in using Pandora to export via JDBC.
**Rename the "Leading Zero as a number" in Configure Columns menu
The name of this standardisation option has been changed to "Treat numeric values starting with zero as text (keep leading zeros)" as the original text was not user friendly.
When an audit type is enabled or disabled, an entry in the server log will detail the change. This improves reporting for compliance as previously this could not be tracked in the logs.
If directory rename fails post defragmentation, the product will wait and retry for up to 30 minutes. This allows external processes access time to finish with the newly written files and remove any locks on the directory. We have also improved logging such that each stage of defragmentation is logged on completion (rather than at the end of the whole defragmentation process).
When connecting to an Oracle database all columns are returned twice on upload
JDBC connection held open continuously
Client freezes when refreshing connections with associated drilldowns
Decision functions that use error(s) as conditions ignore them rather than returning an error
Linking drilldowns don't cancel quickly enough on large links if the user hits back button whilst linking is ongoing
Settings in paths.properties file are not honoured
Date format not inherited correctly when table created from another table/drilldown
Extract Date/Time Element not applying Time Zone offset correctly
Column Language Configuration not respected at File Preview
Cell functions not marked as Dynamic
This release groups together a collection of features improving on the theme of improving Experian Pandora for GDPR requirements.
Features include:
GDPR validation expressions
Global Batch Address Verification improvements
Improvements to the ability to leverage the REST API
These features are important for customers for the following reasons:
For those currently using Batch integration with Experian Pandora – the new Batch experience is faster and easier to use
Processing time for address verification cut from hours to minutes.
Manipulating data post-Batch is faster – providing customers with improved time to value
Customers with Personally Identifiable information subject to GDPR now have added expressions and functions that allow them to validate their data and report on data quality more effectively
Customers can now use Experian Pandora with third party solutions in a more streamlined way through API improvements
We have added commands in the API, and improved its documentation, to make Experian Pandora more intuitive for use with partner applications
Improved integration of Experian Pandora into a customer's automated GDPR workflow decreasing time to value
National Insurance number format validation is now available
Dynamic Minimum Age check validation is now available
Email address format validation is now available
Numerous improvements have been made to the integration with Batch Address Verification, to aid configuration and troubleshooting. Additionally, the Batch process has been sped up and is now caching results, making for a much more responsive experience for users. Data can then be manipulated more quickly after a Batch job is run. We have also added a progress indicator so that customers can see the status of the job.
Address Validation within Experian Pandora, details:
Missing json files added to aid configuration; json files created for ADS sets – this allows you to create and configure a layout in Experian Pandora rather than having to leave the GUI and directly edit the qaworld.ini file
Input field configuration in the "Clean Address Settings" (Home > System > Right Click > Clean Address Settings) is simplified for users
Experian Pandora now displays the correct country flags for non-ISO datasets (SGF, GBB etc.) Previously it did not display a flag
Allow layout configuration for all data, even if unlicensed, or where the data has expired (customers will not be able to display the data until a license is purchased but they can configure the layout of the address)
A user can now configure how often they are alerted to batch data expirations
Batch Address Verification data and license expiry handling
All valid address layouts are available for selection (even if the data files have expired) when the Batch feature is used
When the Batch feature is used an explicit error is given if the data or license for Batch has expired. This also includes the message "Not Licensed" if the Batch feature is missing from the Experian Pandora license
"Repeat Interval for Batch and Disk Warnings" setting default updated from every 10 minutes to 8 Hours and is now fully configurable by customers
In the Validate and Enrich dialogue the output lines configuration now displays each element name when there are multiple address elements configured on the same address line, improving ease of use
Added a verbose Match Type field
You can now create or edit Batch layouts in the Client when connecting to a Linux Pandora server
We have improved our API to enable customers and partners to better utilize Experian Pandora outside of the user interface. Connections to the application can now be created and deleted from the REST API. It is also now possible to catalogue the tables available within a connection and to initiate a table load against a connection once established.
Third-party software partners will be able to utilize these new commands to help them to integrate new services with Experian Pandora such as data discovery. Customers will be able to operate Experian Pandora seamlessly from within their own workflow tools.
The following commands have been added:
GET object/datasource returns a list of connections
GET object/datasource/<name> returns details for connection <name>
POST object/datasource/<name> creates connection <name> with the parameters in the request body
DELETE object/datasource/<name> deletes connection <name>
GET object/datasource/<name>/list catalogues the tables available in connection <name>
POST object/datasource/<name>/load loads a table identified in the request body from the connection <name>
We have now created Swagger documentation (as well as our existing user guide) to help customers utilize these features.
The ability to pre-select current table in the table lookup function
Metro2: Configurable date handling added. (Metro 2 is a proprietary file format used by financial services companies in the U.S.A. We have added support for ingestion of these files in recent versions of Experian Pandora to support clients using our Data360 solution)
Enhanced rule validation
Create a specific view for the Table Expression drilldown
Introduction of a view of table expressions across all version of a given table where previously you could only see a specific table
Introduction of a view of table expressions across the latest versions of all tables
Updated Progress Hive driver with support for Knox authentication
A bug was found in versions 5.7.0 and 5.7.1 where large numbers of table version groups were created erroneously in Experian Pandora leading to performance issues.
Version 5.7.2 introduces the following improvements:
Fix to the table version groups bug
An automatic repair tool for this bug that will repair affected repositories on install
This release of Experian Pandora introduces the following enhancements:
Metro2 handling improvements:
Improved Metro2™ file handling: all known correct Metro formats are now handled by Experian Pandora
Improved Metro2™ error logging: configurable error logging is now available for Metro2™ files and Pandora produces error messages for incorrect Metro2™ file formats. All errors are logged.
Amazon S3 buckets: Experian Pandora can now successfully access files inside folders in Amazon S3 buckets.
Improvement to JDBC SQL connection when connecting via Windows Authentication
Experian Pandora now handles exporting of an SQL table which has non-alphabetically ordered composite keys
This release of Experian Pandora introduces the following enhancements:
We've also fixed a number of bugs.
It is now possible to convert files from the Metro2 (Packed 366 or Character 426) file formats into a CSV ready for loading into Experian Pandora. Although this functionality is fully tested, test data has been in short supply and it is possible that some aspects of the standard have not been fully catered for. Any such issues will be dealt with should they arise.
Metro2 files should be copied into the drop folder (C:\pandora\data\metro2 by default but this is configurable). They will be automatically converted and, when successful, will be available in Experian Pandora's list of files ready to be loaded. Any errors which occur during the loading process will be logged; successful loads will have the input file removed from the drop directory whilst failures will be left in place. Three attempts will be made to load a file after which time it will be blacklisted until a server restart.
Once converted, records loaded into Experian Pandora will contain a column for all fields of the base record and all fields for each optional segment whether or not they are populated in the source. Whilst there are plans to update the functionality in the future, at present any record containing multiple J1 and J2 segments will see only the last populated instance of each segment written to the destination.
The setting to enable raw format means that numbers dates will not be processed on conversion and will just be stored as a string of digits. This is provided as a diagnostic tool for users who are having difficulty getting their data to convert correctly.
Better management and visibility of the maximum number of versions that can be kept of each table.
The default value for the maximum number of versions of a table that will be kept remains at 32 but now can be configured by the user. Once set, this default will be applied to all new and existing tables that do not have the maximum individually set. Furthermore, the maximum can now be set across a number of tables at the same time.
It is now possible to view the maximum version count for each table as it appears along with the number of versions actually kept in the table drilldown.
In versions of Experian Pandora prior to v5.6.1, the handling of dates and times when interchanging values between database and Experian Pandora has been inconsistent and unreliable. This is in fact a very complex area because often, it is not clear which timezone a date and time relates to. The design goal for this change has been to hide the complexity of timezones from the user by attempting, as far as possible, to leave displayed times unchanged whilst maintaining the integrity of the data.
Specifically, the following points are now true:
When logging load table actions, the repository id and table version are recorded.
Table expression cache start/completed actions are now logged. Log entries include table name, version, id and expression name.
JDBC connection strings will now only be shown in errors when the JDBC Debug option is turned on.
A log message will now appear to show that a period of database disconnection has begun.
Tooltips have been added and updated on the JDBC connection Automation/Refresh Details tabs. An Experian Pandora documentation page has also been created.
The phone number validation libraries have been updated so validation for international phone numbers should now be more accurate. This requires a full installation (you can't upgrade to this version to get the improved phone validation).
Scheduled tasks will not restart after a non working day.
Any task not scheduled to run on a working day would fail to run after a non-working day unless other scheduled tasks continued to run in the meantime.
Language settings are incorrectly referenced when loading dates in the format YYYY-mm-dd.
This date format is now invariably parsed as year, month, day regardless of the locale of the source data.
Files with associated DDL load yyyy-mm-dd dates incorrectly, and changing the language does not fix it.
The fix for 40165 also applies in this situation where the file is described by a DDL file.
"Default Standardisation Rules" server setting appears to require a server restart but doesn't prompt.
A server restart is now prompted when this value is edited.
JDBC Connection period does not work
A number of fixes made to ensure that the connection period is respected
Cannot export nulls to sql server varchar/char field
Experian Pandora does not distinguish between nulls and empty strings and attempts to hide the concept from the user. An Experian Pandora null (which is a true null or an empty string) will be written to a database field as null if the field is nullable, or otherwise as an empty string. Experian Pandora strings comprising only spaces are always written as is.
Note that this means that if data is read from, and subsequently written back to, SQL server then any empty strings in nullable fields will be converted to null.
Prevent Ghost Table versions appearing
When the Maximum Table Versions setting is changed for a table, ensure the table information accurately reflects the number of table versions stored.
Disabled Connections allow tables to be loaded and exported to
Prevented loads from occurring from disabled connections
Prevent drilldown exports to a disabled connection - including loads/exports that are scheduled for later.
JDBC connection refresh does not show new tables for particular database types (e.g. Cassandra & Mongo)
New tables are now detected with a connection refresh
Unable to update records via Experian Pandora for various JDBC Drivers
It is now possible to insert/update/delete records for Mongo and Cassandra databases
JDBC Load Query keeps running when Load job is stopped in Experian Pandora - SQ
When loading a table from a SQLdatabase, if you stop the job on Experian Pandora, the SQL query now stops in the source SQL database.
Port error when attempting to connect to SQLServer Data Direct Driver
This driver now automatically sets the allowPortWithNamedInstance property such that a connection is still made when a named instance connection is supplied as well as a port number.
Client fails to startup on initial attempts if configured database connections time out
Regardless of the connection status of database connections, the client now starts up on the first attempt.
DNS caching never refreshes
The IP address for a hostname is no longer cached indefinitely for database connections so if the IP of a hostname changes, it will now be picked up.
"Drilldowns (All Versions)" drilldown on a table duplicates all drilldown names
Ghost versions of drilldowns are no longer displayed in the "Drilldowns (All Versions)" panel.
The major features in this release are as follows:
Experian Pandora will now ship with high-performance database drivers to allow out-of-the-box connections to a wide range of common database servers.
The new Drivers fully comply with the JDBC 4.0 specification and are fully supported. Pandora users will no longer need to download and configure JDBC drivers provided by database vendors.
List of drivers shipped with first version
Oracle
SQL Server (Supports SQL Server, SQL Azure)
DB2
MongoDB
PostgreSQL
Hive (Supports Apache Hadoop Hive, HortonWorks, Cloudera CDH Hive, Amazon EMR Hive, IBM BigInsights Hive, MapR Hive, Pivotal HD Hive)
MySQL
Greenplum (Supports Pivotal Greenplum, Pivotal HAWQ)
Cassandra (Supports Apache Cassandra, DataStax Enterprise)
Informix
Salesforce (Supports Salesforce.com, Force.com Applications, Financial Force, Veeva CRM)
Spark SQL
Sybase
Progress OpenEdge
Oracle Service Cloud
Amazon Redshift
The major features in this release are as follows:
The Pandora Data Quality summary reports (column summaries) now automatically associate the data quality scores and cost/risk amounts to each Business Term in the Business Glossary.
Column summaries now include the Business Glossary Categories and Terms to which each data column is associated, alongside the existing rule scores and rule categories.
The major features in this release are as follows:
Table Expression Cache builders now initialize in groups, controlled by the server setting Maximum Expression Cache Concurrent Builds.
Explaining drilldown SQL Queries is faster.
Caching means that when an object is requested for the drilldown, it's only had to be called once for the entire query.
The major features in this release are as follows:
All communication between the client and server now uses SSL by default.
Support for user-supplied certificates.
The major features in this release are as follows:
A new Save As script-type has been added. This can be used to save a modified drilldown or table as a new table, or new version of an existing table.
Scripts can be written using any of the following supported languages:
JavaScript
Python
Ruby
Groovy
Scala
R
Template scripts, to help get started with scripting, are available via the Template button in the code editor
Script logs can now be viewed in the UI by right-clicking on a script and selecting View Logs
Permissions (read, modify, delete, execute) can be controlled on a per-script basis. The Script user capability controls script usage at the Role level
Export scripts can now export via script to a JDBC target as well as to files
The Character As Alphanumeric server setting now applies to all data formats, not just JDBC
The new Strict CSV Processing server setting gives additional control over how files with .csv extension are handled on load. If set to false, .csv files will not be treated as strictly comma separated values
The new server setting End Of Line Precedence can be turned on to give newline characters precedence over quote characters and field delimiters when loading a file.
There is a new option to change a file's extension on the server, through the Client. This can be done by right clicking a fixed-length, comma-separated or otherwise delimited file in the Explorer and selecting Change File Format.
New functions have been added for parsing, formatting and validating international phone numbers
Format Phone Number – Takes a column or value as an input and then will either output a correctly formatted phone number, extract information from the phone number (for example the Number Type: Fixed Line, Mobile, Toll-free), or return a Not A Number error.
Phone Number Match Code – Compares two values as phone numbers and returns the match code (For example NSN Match, Exact Number, No Match)
Phone Numbers Match – Tests if two phone numbers are an exact match for a given country ISO, taking into account valid formats, country codes, etc.
Validate Phone Number – Validates a phone number using country-specific validation tests based on a supplied county ISO code.
Example Phone Number – Creates an example number for a particular country in a specified format
The Join Lists function has been renamed Combine Lists, and for clarity the various list join operations been renamed as follows:
Matched is now Intersection
Combine is now Union
Right Unmatched is now Remove Left
Left Unmatched is now Remove Right
Left and Right Unmatched is now Difference
The major features in this release are as follows:
Scripts can now be defined as one of a number of different types:
Export - a script that can provide custom export functionality
Batch Script - a script that can be executed in a Server job
Server - a general Server-side script that can be invoked by the Client
Server Model -a script that is available to other Server-side scripts
Client UI - a script that can optionally display UI components in a Client desktop window
Client Module - a script that is available to other Client-side scripts
RESTful - a script that is executed in a web container on the Client
Scripts now support parameters, which can be defined by the user at invoke time.
Refer to the Scripting and REST Guide.pdf for full details on the new scripting functionality.
The process of uploading files to the Server import area has been simplified: clicking on the Upload button on the quick toolbar now immediately shows a file chooser dialog rather than the previous File Manager dialog.
Support for uploading metadata files (DDL, COB and XML) to the metadata directory.
