Executive Summary

Many organisations talk about achieving a “single view” of an entity, but the practical reality is much harder than the phrase suggests. Customer, citizen, legal entity, and supplier records are often spread across multiple systems, created for different operational purposes, and maintained with varying levels of quality. This leads to duplicates, incomplete records, inconsistent formats, missing identifiers, and conflicting versions of the truth.

Entity resolution is the discipline of determining which records refer to the same real-world entity and then linking, clustering, or consolidating those records into a trusted view. The webinar emphasised that this is not just a technical matching exercise. Effective entity resolution depends on strong data quality foundations, careful standardisation, configurable match strategies, and input from subject matter experts who understand the data and its business context.

The value of a successful single view is substantial. Organisations can make better decisions, reduce operational inefficiency, improve service delivery, support compliance obligations such as KYC and AML, and identify risks including fraud and duplicate activity. In both financial services and public sector settings, the ability to connect fragmented data into a trustworthy entity view has direct operational and societal value.

The discussion also highlighted the role of AI and machine learning. These technologies can improve speed, recall, and classification of ambiguous matches, especially when deterministic rules alone are not enough. However, they should augment rather than replace governance, explainability, and expert review. Entity resolution remains a human-guided discipline supported by automation, not an unsupervised black box.

1. Introduction: Why “Single View” Matters

In data management and governance, the term single view is widely used: single customer view, single citizen view, single supplier view, and more. The webinar opened by challenging the simplicity of the phrase. A single view sounds straightforward, but in practice it means reconciling data that has been created in different places, for different reasons, at different times, and often with inconsistent standards.

�鶹���� described this challenge as common across industries. In financial services, firms need to align customer and counterparty data across systems to support compliance, onboarding, and risk management. In government and public services, organisations face the equivalent challenge with citizen data, vulnerable persons, case records, and service interactions. In each case, the goal is not simply to store data in one place, but to create a trustworthy, usable representation of the real-world entity behind the records.

The webinar positioned entity resolution as a core capability within broader data management: profiling data, detecting outliers, cleansing and standardising records, then applying matching logic to resolve duplicates and improve trust.

2. The Core Problem: Why Entity Resolution Is Difficult

Achieving a single view is difficult because enterprise data is rarely born clean. Records may be duplicated across systems, entered manually, updated at different times, or captured without a common identifier. Even when identifiers exist, they may be missing, outdated, or inconsistent across sources.

Common sources of complexity include:

• Variation in names, addresses, and descriptive fields
• Misspellings, abbreviations, and transposed characters
• Incomplete or missing attributes
• Different formatting standards between systems
• Conflicting versions of the same entity record
• Absence of universal identifiers across datasets

The transcript and related materials reinforce that matching is especially challenging for person and organisation data, where ambiguity is natural. For example, people matching must contend with naming conventions, incomplete personal details, and data quality gaps. Legal entity matching often depends on names, addresses, country information, and identifiers such as LEIs, which still require cleansing and standardisation before reliable matching is possible.

This is why the webinar repeatedly linked entity resolution to data quality. Matching poor-quality data directly tends to create false positives, false negatives, and high manual review effort. Better input data produces better resolution outcomes.

3. Entity Resolution Explained

Entity resolution is the process of identifying records that refer to the same real-world entity and deciding how they should be linked or merged. Depending on the use case, the outcome may be:

• Match: determine whether two or more records refer to the same entity
• Link: connect related records across systems without physically merging them
• Cluster: group multiple records believed to represent the same entity
• Golden record: create a trusted, canonical representation from multiple inputs

The supporting documentation describes a layered approach to this process. First, data is ingested and improved through cleansing and normalisation. Next, configurable match strategies are applied using combinations of exact, fuzzy, phonetic, numeric, regex-based, and semantic approaches. Finally, results are reviewed, scored, and either accepted automatically or escalated for expert validation.

This approach matters because no single match technique is sufficient for every domain. Exact matching may work well for strong identifiers; fuzzy or phonetic matching can help with spelling variation; NLP or semantic similarity can add value when unstructured or messy text is involved. Effective entity resolution combines these techniques rather than relying on only one.

Key principle: Entity resolution is best treated as a configurable, data-aware process, not a one-size-fits-all algorithm.

4. The Business Value of a Single View

The webinar made clear that entity resolution matters because it improves decisions and outcomes. When organisations can trust that records are correctly linked, they gain a clearer understanding of customers, citizens, suppliers, or counterparties and can act with greater confidence.

Key benefits include:

• Better decision-making: users can work from a more complete and accurate entity view
• Operational efficiency: less time spent reconciling duplicates or validating conflicting data
• Improved compliance: supports AML, KYC, customer due diligence, and regulatory reporting processes
• Fraud and risk detection: better visibility of suspicious duplication, hidden relationships, or fragmented records
• Service improvement: public sector organisations can reduce duplication and improve case visibility for citizen services

Internal reference material extends these examples. In financial services, matching supports single client view, anti-fraud measures, and cross-vendor mapping of instruments and identifiers. In the public sector, similar techniques are used for citizen data, case management, and vulnerable individual records. The same underlying challenge appears in different forms across domains.

5. A Practical Approach to Solving the Problem

The webinar presented entity resolution as a practical discipline rather than an abstract aspiration. Organisations should start by acknowledging that this is both a data and a business problem.

A sensible approach includes:

1. Prepare the data: profile, cleanse, standardise, and normalise source records before matching.
2. Design a match strategy: choose appropriate techniques for the entity type and attributes involved.
3. Apply scoring and thresholds: use confidence scores across relevant attributes such as name, address, country, or identifier.
4. Review ambiguous cases: route lower-confidence outcomes to trained analysts or domain experts.
5. Refine over time: use expert feedback to improve rules, thresholds, and model behaviour.

This practical model is consistent with existing �鶹���� documentation, which describes flexible match strategies, scalable in-memory processing, and workflows that combine deterministic techniques with machine learning support.

6. Why Human Expertise Still Matters

One of the clearest messages from the webinar was that entity resolution is not a fully automated exercise. Sean McConnell described it as an art that requires input from multiple directions. Domain specialists such as crime registrars, analysts, or line-of-business experts understand the subtleties of their data and the consequences of getting a match wrong.

Technology and domain expertise must work together. Tools can accelerate matching, expose patterns, and reduce manual burden, but subject matter experts provide the context needed to judge edge cases, validate outcomes, and ensure the process reflects real business meaning.

Entity resolution requires the expertise of both subject matter specialists and the analysts and engineers configuring the tools. The balance between these perspectives is essential to achieving a trusted outcome.

This is especially important in high-stakes domains such as financial crime, citizen services, and vulnerable-person records, where false matches or missed links can have material consequences.

7. The Role of AI and Machine Learning

The webinar addressed AI directly and took a balanced position. AI introduces opportunities for greater speed, intelligence, and potentially improved accuracy, but also introduces risks, including overconfidence and poor explainability if applied carelessly.

Supporting material shows several areas where AI and ML can strengthen entity resolution:

• Classifying low-confidence matches when deterministic rules are inconclusive
• Learning from expert-reviewed match decisions to reduce future manual effort
• Improving semantic matching using NLP or embedding-based approaches
• Supporting data preparation tasks such as labelling, feature engineering, and standardisation

�鶹���� internal materials note that machine learning can reduce manual intervention significantly in some entity resolution workflows, and that low-confidence classifiers can learn from thousands of expert-reviewed examples. At the same time, strategy documents point toward newer embedding-based methods that can augment or replace older NLP approaches for better semantic matching.

Important: AI should augment governed matching processes, not replace them. Human review, auditability, and clear thresholds remain critical.

8. Architecture and the “Golden Record” Question

In the closing discussion, the webinar considered whether organisations should aim for a centralised architecture with one harmonised system of record. The answer was pragmatic: in an ideal world, deduplicated and controlled data would flow into a central platform, but real-world estates are more complex.

Entity resolution and architecture are not opposing choices. Matching can serve several architectural patterns:

• Creating a new golden record from multiple upstream systems
• Linking records across systems while leaving source systems intact
• Supporting a hub-and-spoke model where canonical identifiers are shared back to the estate
• Enriching governance platforms with trusted entity outcomes and trust scores

This is echoed in integration material involving Microsoft Purview, where �鶹���� performs advanced matching, clustering, and golden-record creation, then pushes canonical IDs and trust scores back into the governance layer. The implication is clear: a single view is as much an operational capability as it is a data architecture decision.

9. Use Cases Across Industries

These examples reinforce that entity resolution is not confined to a single vertical. The data patterns differ, but the core challenge remains the same: connect fragmented records accurately enough to support real decisions.

10. Conclusion

The webinar’s central argument was simple: a single view is valuable, but it is not achieved by wishful thinking or architecture alone. It requires disciplined data quality, flexible match strategies, domain expertise, and controlled use of AI where it adds measurable value.

Entity resolution is foundational for organisations that need to trust their data across systems. Whether the goal is a single customer view, citizen view, supplier view, or legal entity view, the path forward is practical: improve the data, match it intelligently, involve experts, and operationalise the result.

For organisations dealing with fragmented and duplicate entity data, the opportunity is not just cleaner records. It is better decisions, stronger compliance, improved service, and a more trustworthy data foundation overall.

Next Steps

You can watch a full recording of the webinar below. Need more information? Simply hit ‘Get in touch’ at the top of the page.

The post Entity Resolution: The Long Read appeared first on �鶹����.

London, 20th May, 2026 –��, the enabler of real-time��clearing and embedded��banking,��has strengthened Financial Services��Compensation Scheme (FSCS)��compliance��through a partnership with data readiness firm��, supporting rapid client onboarding.��

The bank’s Embedded Banking Partner (EBP) model, which supports partners such as Revolut, Chip, Tide and Raisin, introduces additional complexity into FSCS submissions, especially as the bank continues to scale. 

ClearBank selected �鶹���� based on its industry-proven FSCS capability, including work with Investec and Bank of Ireland, its ability to provide rapid, on-demand regulatory support, and its strong fit with ClearBank’s agile operating model. 

Through the partnership, �鶹���� rapidly integrated FSCS outputs with ClearBank’s internal data, segmenting issues by EBP, allowing the bank to focus on specific sections of the FSCS submission and broader regulatory engagement.  

Chris Kemp, Head of Product Compliance at��ClearBank, said:��“FSCS readiness��isn’t��just a compliance requirement��for us,��it’s��an opportunity for growth, supporting faster onboarding for partners and customers.��Working with���鶹������has been a strong cultural fit,��with their in-depth��expertise��helping��to��future-proof our FSCS compliance, effectively acting as internal auditors,��validating��processes��and��providing us with assurance with our submissions.��This has strengthened our data and controls, while improving visibility across embedded��banking partners, helping us stay regulator-ready and focus more on driving value for��customers.����”��

The partnership has also provided ClearBank with strategic insights, shedding additional light on the downstream impact of controls such as identification and verification (ID&V) and financial crime processes, and providing clearer visibility into Single Customer View file data quality across EBPs. 

Stuart Harvey, CEO of �鶹����, commented: “The FSCS process operates under tight turnaround windows, and the financial and reputational impact of FSCS failures can be significant, so banks need audit-ready data to meet regulatory demands quickly. Especially for financial institutions like ClearBank, operating with an embedded banking model, collecting and managing accurate data across multiple systems at pace is critical.” 

“Banks must view regulatory readiness as a growth multiplier, starting with strong data quality and governance foundations. In this aspect, it’s clear to us that ClearBank is setting the standard.” 

The post Clear Bank strengthens FSCS compliance with �鶹���� Data Readiness appeared first on �鶹����.

Hubert Składanowski, Machine Learning Manager at �鶹���� recently penned an article for Techerati on Data Privacy Day. Click the link below the graphic to read the article in full.

Read the article .

The post AI Data Readiness: A Privacy Day special appeared first on �鶹����.

Head on over to to read the full article.

The post AI Unhappy? Stuart Harvey on data’s critical path to success appeared first on �鶹����.

Read the article in full .

And to explore the benefits that �鶹���� FSCS Data Readiness can offer, head to /get-data-readiness-fscs/.

The post Coverage in i-Invest Online: FSCS Data Readiness appeared first on �鶹����.

London, 14th March 2025

�鶹���� has developed a digital data validation sandbox for the Financial Services Compensation Scheme (FSCS) designed to help assess data systems and processes prior to regulatory audits.

, a leading international financial services provider, adheres to multiple regulatory standards, including those set by the FSCS, which protects depositors by guaranteeing savings of up to £85,000 per person if a regulated firm is unable to meet claims.

The Prudential Regulation Authority (PRA) conducts regular assessments of Investec’s compliance with the FSCS rules, which necessitates quick access to accurate data.

The �鶹���� sandbox is designed to help streamline Investec’s data management processes, focusing on data quality to prepare for these assessments. The sandbox has features such as rule matching, data categorisation, and data de-duplication to support compliance with FSCS regulations.

Cherie Lefever, FSCS Coordinator in Investec’s UK Central Compliance Team, has developed rules and exceptions to expedite data management and meet the PRA’s auditing requirements efficiently.

Cherie Lefever stated: “Streamlined data is crucial for complying with the PRA’s Depositor Protection Rules. The collaboration with �鶹���� has further simplified our data management processes, allowing us to ensure we maintain high standards of data quality and integrity.”

Stuart Harvey, CEO of �鶹����, commented: “Financial institutions like Investec manage vast amounts of client and internal data, necessitating robust data management, governance, and quality capabilities. With the FSCS requiring regular audits, access to streamlined data management tools enhances banks’ readiness for these assessments. Our digital sandbox solution empowers firms to evaluate their data systems in preparation for regulatory audits.”

The post Press Release: Digital Data Validation Sandbox appeared first on �鶹����.

�鶹���� has secured the ‘Most Innovative Technology for Regulatory Compliance’ award at this year’s A-Team Group RegTech Insight Awards USA.

The Belfast-based firm, which has made its name specializing in innovative solutions to challenging data problems, developed its Data Readiness solution in response to the prevalence of data-driven regulations across both sides of the Atlantic. 

Data Readiness grew out of �鶹����’ close work with banking customers in the UK, primarily around data used to identify and reporting on customer deposits in line with stringent regulation from the Financial Conduct Authority () and Prudential Regulatory Authority ().  

In 2024, �鶹���� developed Data Readiness as a specific defined solution to measure and prepare data to specific regulatory standards, offering the solution with a range of deployment options, including as-a-service.  

Behind the solution

Matt Flenley, Head of Marketing at �鶹����, noted, “This award is brilliant news for all our talented developers and engineers back home. We’ve a long record of working alongside customers to help them with specific problems they’re encountering, and for which the risk of bad data affecting their ability to demonstrate compliance is often significant.

“With our experience in developing rule-sets for customers seeking to comply with UK depositor protection regulations, alongside international standards such as BCBS 239, we felt the time was right to offer this solution as its own piece of regulatory compliance technology.

“We’d like to thank the A-Team panel and all those who saw fit to recognise the approach we’ve taken here – thank you all so much!” 

Angela Wilbraham, CEO at A-Team Group, and host of the 4th annual RegTech Insight Awards USA 2024, commented, “Congratulations to �鶹���� for winning the Most Innovative Technology for Regulatory Compliance award in this year’s A-Team Group Awards USA 2024.

“These awards celebrate providers of leading RegTech solutions, services and consultancy and are uniquely designed to recognise both start-up and established providers who are creatively finding solutions to help with regulatory challenges, and span a wide range of regulatory requirements.

“Our congratulations for their achievement in winning this award in a highly competitive contest.” 

For more on �鶹����, visit www.datactics/get-data-readiness 

For more on A-Team Group, visit   

The post �鶹���� Snaps Up Award For Regulator-Ready Data appeared first on �鶹����.

It’s October, the Northern Hemisphere nights are drawing in, and for many it’s time when things take a scarier turn. But for public sector leaders exploring AI, that fright need not apply to your data. It definitely shouldn’t be something that haunts your digital transformation dreams.

With a reported £800m budget unveiled by the previous government to address ‘digital and AI’, UK public sector departments are keen to be the first to explore the sizeable benefits that AI and automation offer. The change of government in July 2024 has done nothing to indicate that this drive has lessened in any way; in fact, the Labour manifesto included the commitment to a “single unique identifier” to “better support children and families”[1].

While we await the first Budget of this Labour government, it’s beyond doubt that there is an urgent need to tackle this task amid a cost-of-living crisis, with economies still trying to recover from the economic shock of COVID and deal with energy price hikes amid several sizeable international conflicts.

However, like Hollywood’s best Halloween villains, old systems, disconnected data, and a lack of standardisation are looming large in the background.

Acting First and Thinking Later

It’s completely understandable that the pressures would lead us to this point. Societal expectations from the emergence of ChatGPT, among others, have only fanned the flames, swelling the sense that technology should just ‘work’ and leading to an overinflated belief in what is possible.

Recently, LinkedIn attracted some consternation[i][2] by automatically including members’ data in its AI models without seeking express consent first. For whatever reason, the idea that people would just accept this change was overlooked. It took the UK’s Information Commissioner’s Office, the ICO, to intervene for the change to be withdrawn – in the UK, at least.

A dose of reality is the order of the day. Government systems are lacking integrated data, and clear consent frameworks of the type that LinkedIn actually possesses seldom exist in one consistent way. Already lacking funds, the public sector needs to act carefully, and mindfully, to prevent their AI experiments (which is, after all, what they are) from leading to inaccuracies and wider distrust from the general public.

Streamlining Data and Consent in One Single View

One solution is for Government departments to form one, holistic set of consents concerning use of data for AI, especially Large Language Models and Generative AI – similar to communication consents under the General Data Protection Regulation, GDPR.

The adoption of a flexible consent management policy, one which can be updated and maintained for future developments and tied to an interoperable, standardised single view of citizen (SCV), will serve to support the clear, safe development of AI models into the future. The risks of building models now, on shakier foundations, will only serve to erode public faith. The evidence of the COVID-era exam grades fiasco[3] demonstrates the risk that these models present to real human lives.

Of course, it’s not easy to do. Many legacy systems contain names, addresses and other citizen data in a variety of formats. This makes it difficult to be sure that when more than one dataset includes a particular name, that name actually refers to the same individual. Traditional solutions to this problem use anything from direct matching technology to the truly awful exercise of humans manually reviewing tens of thousands of records in spreadsheets. This is one recurring nightmare that society really does need to stop having.

Taking Refuge in Safer Models

Intelligent data matching uses a variety of matching algorithms and well-established machine learning techniques to reconcile data held in old systems, new ones, documents, even voice notes. Such approaches could help the public sector to streamline their SCV processes, managing consents more effectively. The ability to understand who has opted in, marrying opt-ins and opt-outs to demographic data is critical. This approach will help model creators to interpret the inherent bias in the models built on those consenting to take part, to understand how reflective of society the predictive models are likely to be – including whether or not it is actually safe to use the model at all.

It’s probable that this transparency in process could also lead to greater trust in the general public to take part in data sharing in this way. In the LinkedIn example, the news that data was being used without explicit consent, raced around like wildfire on the platform itself. This sort of outcome cannot be what LinkedIn anticipated, which in and of itself is a concern about the mindset of the model creators.

It Doesn’t Have to Be a Nightmare

It’s a spooky enough season without adding more fear to the bonfire; certainly, this article isn’t intended as a reprimand. The desire to save time and money to deliver better services to a country’s citizens is a major part of many a civil servant’s professional drive. And AI and automation offer so many opportunities for much better outcomes! For just one example, NHS England’s AI tool already uses image recognition to detect heart disease up to 30 times faster than a human[4] . Mid and South Essex (MSE) NHS Foundation used a predictive analytical machine learning model called Deep Medical to reduce the rate at which patients either didn’t attend appointments or cancelled with short notice (referred to as Did Not Attend, or DNA). Its pilot project identified which patients were more likely to fall into the DNA category, developed personalised reminder schedules, and through identifying frail patients who were less likely to attend an appointment, highlighted them to relevant clinical teams.[5]

The time for taking action is now. Public sector organisations, government departments and agencies should focus on the need to develop systems that will preserve and maintain trust in the AI-led future. This blog has shown that better is possible, through a dedicated desire to align citizen data and their consents to contact. In a society where people have trust and transparency in the ways that their data will be used to train AI, the risk of nightmare scenarios can be averted and we’ll all sleep better at night.

[1]

[2]

[3] .

[4]

[5]

[i]

The post Nightmare on LLM Street: How To Prevent Poor Data Haunting AI appeared first on �鶹����.

The oak-panelled setting of historic Toynbee Hall provided the venue for the 2024 �鶹���� Summit, which this year carried a theme of ‘Data-Driven Return on Investment.’

Attendees gathered for guest speaker slots covering:

• Danske Bank UK’s Lyndsay Shields presenting a ‘Data Management Playbook’ covering the experiences of beginning with a regulatory-driven change for FSCS compliance, through to broader internal evangelisation on the benefits of better data;

• �鶹����’ own data engineer, Eugene Coakley, in a lively discussion on the data driving sport, drawing from his past career as a professional athlete and Olympic rower with Team Ireland;

• and Renfrewshire HSCP’s Bobby McClung explaining how automation and the saving of person-hours or even days in data remediation was having a material impact on the level of care the organsation is now able to deliver to citizens making use of its critical services.

The �鶹���� Customer Awards in full

In recent months, the team at �鶹���� has worked to identify notable achievements in data in the past year. Matt Flenley, Head of Marketing at �鶹����, presented each with a specific citation, quoted below.

Data Culture Champion of the Year –

“We’re delighted to be presenting Lyndsay with this award. As one of our longest-standing customers, Lyndsay has worked tirelessly to embed a positive data culture at Danske Bank UK. Her work in driving the data team has helped inform and guide data policy at group level, bringing up the standard of data management across Danske Bank.

“Today’s launch of the Playbook serves to showcase the work Lyndsay and her team have put into driving the culture at Danske Bank UK, and the wider culture across Danske Bank.”

Data-Driven Social Impact Award –

“Through targeted use of automation, Renfrewshire Health and Social Care Partnership has been able to make a material difference to the operational costs of local government care provision.

“Joe Deary’s early vision and enthusiasm for the programme, and the drive of the team under and alongside Bobby, has effectively connected data automation to societally-beneficial outcomes.”

Data Strategy Leader of the Year –

“RBC Brewin Dolphin undertook a holistic data review towards the end of 2023, culminating in a set of proposals to create a rationalised data quality estate. The firm twinned data this strategy with technology innovation including being early adopters of ADQ from �鶹����. They overcame some sizeable hurdles, notably supporting �鶹���� in our early stages of deployment. Their commitment to being an ambitious, creative partner makes them stand out.

“At �鶹���� we’re delighted to be giving the team this award and would also like to thank them for being exemplars of patience in the way they have worked with us this year in particular.”

�鶹���� Award for Partner of the Year –

“Solidatus and �鶹���� have been partnered for the last two years but it’s really in 2023-2024 that this partnership took off.

“Ever since we jointly supported Maybank, in Malaysia, in their data quality and data lineage programme, we have worked together on joint bids and supported one another in helping customers choose the ‘best of breed’ option in procuring data management technology. We look forward to our next engagements!”

�鶹���� Data Champion of the Year –

“For all the efforts Tony, Nick and team have made to spread the word about doing more with data, we’d like to recognize NHS Business Services Authority with our �鶹���� Data Champion of the Year award.

“As well as their advocacy for our platform, applying it to identify opportunities for cost savings and efficiencies across the NHS, the team has regularly presented their work to other Government departments and acted as a reference client on multiple occasions. Their continued commitment to the centrality of data as a business resource is why they’re our final champions this year, the �鶹���� Data Champion 2024.”

Gallery of Winners

Toasting success at Potter & Reid

The event closed with its traditional visit to Shoreditch hot spot Potter & Reid. Over hand-picked canapés and sparkling drinks, attendees networked and mingled to share in the award winners’ achievements in demonstrating what data-driven culture and return on investment looks like in practice. Keep an eye out for a taster video from this year’s event!

The post �鶹���� Awards 2024: Celebrating Customer Innovation appeared first on �鶹����.

At �鶹����, maintaining the highest information security standards has always been at the core of our operations. This unwavering commitment has recently been formally recognised with our achievement of the ISO 27001:2022 Certification for our Information Security Management System (ISMS).

It is a major milestone in achieving ISO27001 and a powerful validation of our continuous efforts to protect client data and ensure the integrity, confidentiality, and availability of our information assets. In this blog, I’ll share our journey of achieving ISO 27001:2022.

The journey toward ISO 27001:2022 Certification

Our path to ISO certification began in Q4 2023, with invaluable support from industry experts at .

The process started with a thorough evaluation of our existing security posture, carefully measuring it against the rigorous requirements set by ISO 27001. This stage involved a deep dive into our policies, procedures, and infrastructure to identify potential vulnerabilities and areas for improvement.

Following the evaluation, the DevOps team implemented targeted improvements over several months to strengthen our security framework. These efforts came to a head in January 2024 with a successful Stage 1 audit conducted by . This initial audit was instrumental, providing us with crucial feedback and pinpointing specific areas for improvement.

The rigorous Stage 2 audit

By June 2024, we prepared for the critical Stage 2 audit and welcomed NQA back to the �鶹���� headquarters for an intensive review

This audit was exhaustive. For five days, the auditors scrutinized every facet of our ISMS. The audit team delved into our operations, from software development processes to client support systems and internal IT protocols. The auditors even spot-tested �鶹���� staff on their knowledge and understanding of Information Security Management within the company. This thorough examination ensured that no stone was left unturned.

Thanks to the hard work of our entire team, we successfully passed the audit! �鶹���� earned the ISO 27001:2022 certification, reinforcing our compliance with global information security standards and demonstrating our proactive approach to maintaining a secure operational environment.

Beyond certification: A commitment to excellence

For �鶹����, the ISO 27001 certification is more than just a formal recognition; it embodies our ongoing commitment to excellence in information security and sets the stage for future advancements.

Achieving ISO 27001 is a significant milestone for �鶹���� and is the result of hard work and dedication from the entire team as we aim to grow and improve our security posture, proving the team’s dedication to providing secure and reliable policies and procedures that protect ourselves and our clients.

This milestone is more than just an endpoint; it marks the beginning of an ongoing journey. As we continue to innovate and expand our platform, maintaining a robust information security practice will remain a top priority. Backed strongly by our senior management team, we are ready to build on this foundation, creating a more secure, process-driven, and impactful data quality platform that will positively influence the industry.

As we continue our journey, the ISO 27001:2022 Certification for Information Security Management System reinforces our dedication to building a more secure, process-driven, and impactful data quality platform that will positively influence the industry. As a team, we are excited to see what comes next.

About ISO 27001:2022

The ISO 27001 certification is recognised as the global benchmark for managing information security. �鶹���� accreditation has been issued by , a leading global independently accredited certification body that provides assessments (audits) of organisations to various management system standards since 1988. The process was supported by Vertical Structure, who conduct technical security training, helping companies to achieve certification to international standards such as ISO27001.

The post �鶹���� journey to ISO 27001:2022 certification appeared first on �鶹����.

�鶹����, a leader in data quality software has achieved ISO 27001:2022 Certification for Information Security Management System.

The ISO 27001 certification is recognised globally as a benchmark for managing information security. The rigorous certification process, conducted by and , involved an extensive evaluation of �鶹����’ security policies, procedures, people, and controls. Achieving this certification demonstrates �鶹����’ dedication to safeguarding client data and maintaining information assets’ integrity, confidentiality, and availability.

Victoria Wallace, Senior DevOps & Security Specialist, stated: “Security is at the heart of everything that �鶹���� does and achieving ISO 27001:2022 certification is a testament to the team’s unwavering commitment in this technical field. Showcasing the extensive work that went into this prestigious achievement proves that dedication and determination can lead to significant success, both within �鶹���� and across our client ecosystem. Achieving and maintaining this certification is a key part of �鶹����’ progress in enhancing our secure, process-driven, and powerful data quality platform.”

Tom Shields, Cyber & Information Security Consultant at Vertical Structure, said “It was a pleasure working with the team at �鶹����. Their enthusiastic approach to ISO 27001 Information Security and the associated business risk mitigation was evident in every interaction. Involvement from top to bottom was prioritised from day one, allowing us to integrate into their team from the very outset. The opportunity to guide such organisations in certifying to ISO 27001 is a privilege for us, and we look forward to continuing to work alongside their team in the future.

About ISO 27001:2022 Certification

�鶹����’ accreditation has been issued by , a leading global independently accredited certification body. NQA has provided assessments (audits) of organisations to various management system standards since 1988.

Founded in 2006, is an independent cyber security consultancy with a ‘people-first’ approach. Vertical Structure specialises in providing people-focused security and penetration testing services for web applications, cloud infrastructure and mobile applications.

Vertical Structure also conducts technical security training, helping companies to achieve certification to international standards such as ISO 27001, Cyber Essentials and CAIQ and are proud to be an Amazon Web Services® Select Consulting Partner.

The post ISO 27001:2022 Certification Success appeared first on �鶹����.

What is Data Quality and why does it matter?

��

Data Quality refers to how fit your data is for serving its intended purpose. Good quality data should be reliable, accurate and accessible.��

The post What is Data Quality and why does it matter? appeared first on �鶹����.

Kubernetes: An Introduction��

In the ever-evolving world of engineering, scalability isn’t just a feature—it’s a necessity. As businesses and data continue to grow, the ability to scale applications efficiently becomes critical. At �鶹����, we are at the forefront of integrating cutting-edge AI/ML functionality that enhances our Augmented Data Quality solutions. To align with current standards and ensure optimal AI/ML scalability with Kubernetes, our AI/ML team has integrated K8s into our infrastructure and deployment strategies.

What is Kubernetes? 

Kubernetes, also known as K8s, is an open-source platform designed to automate the deployment, scaling, and management of containerised applications. It adjusts the number of containerised applications to match incoming traffic, ensuring adequate resources to handle requests seamlessly.

Docker containers, managed through an API layer often using FastAPI, function like fully equipped packages of software, including all necessary dependencies. Kubernetes enables ‘horizontal scaling’—increasing or decreasing the number of container instances based on demand—using various load balancing and rollout strategies to make the process appear seamless. This method helps evenly spread traffic among containers, preventing overload and optimising resources. 

Kubernetes for Data Management

Every day, companies handle a lot of complicated data from different sources, at different velocities, and scales. This includes important tasks like cleaning, combining, matching, and resolving errors. It’s crucial to suggest and enforce Data Quality (DQ) rules in your data pipelines and efficiently identify DQ issues, ensuring these processes are automated, scalable, and responsive to fluctuating demands. 

Many organisations use Kubernetes (K8s) to automate deploying, scaling, and managing applications in containers across multiple machines. With features like service discovery, load balancing, self-healing, automated rollouts, and rollbacks, Kubernetes has become a standard for managing applications that are essential for handling complex data—both in the cloud and on-premise. Implementing AI/ML scalability with Kubernetes allows these organisations to process large volumes of data efficiently and respond quickly to changes in data flow and processing demands.

Real-World Scenario: The Power of Kubernetes 

It’s Friday at 5pm, and just as you’re about to leave the office, your boss informs you that transaction data for last month has been uploaded to the network share in a CSV document and it needs to be profiled immediately. The CSV file is massive—about a terabyte of data—and trying to open it in Excel would be disastrous. This is where �鶹���� and Kubernetes come to the rescue.  

You could run a Python application that might take all weekend to process, meaning you’d have to keep checking its progress and your weekend would be ruined. Instead, you could use Kubernetes to scale out �鶹����’ powerful Profiling tools and complete the profiling before you even leave the building. Company saved. Weekend saved. 

Application of Kubernetes 

The world has grown progressively faster, and speed in the digital realm is king: speed in service delivery, speed in recovery in the event of a failure, and speed to production. We believe that the AI/ML features offered by �鶹���� should adhere to the same high standards. No matter how much data your organisation handles or how many data sources��there are, it’s important to adjust resources to meet demand and reduce waste during the most critical moments.��

At �鶹����, AI/ML features are deployed as Docker containers and FastAPI. Depending on your particular environment, we might run these containers on a single machine like AWS EC2 and deploy a single instance of each AI/ML feature, which is suitable for experiments and proof of concepts. However, for a fully operational infrastructure capable of supporting a large organisation, Kubernetes is essential. 

Kubernetes helps deploy Docker containers by providing a blueprint with deployment details, necessary resources, and any dependencies like external storage. This blueprint facilitates horizontal scaling to support additional instances of each AI/ML feature. 

Conclusion 

Kubernetes proved to be a game-changer for scaling �鶹����’ AI/ML services, ultimately leading to a robust solution that ensures our AI/ML features can dynamically scale according to client needs. We tailor our deployment strategies to meet the diverse needs of our clients. Whether the requirement is a simple installation or a complex, scalable infrastructure, our commitment is to provide solutions that ensure our clients’ applications are efficient, reliable, and scalable. 

We aim to meet any specific requirements, always exploring various potential deployment setups preferred by our clients. If your organisation is looking to enhance its data processing capabilities, . Let us help you optimise your data management strategies with the power of Kubernetes and our innovative AI/ML solutions. 

The post AI/ML Scalability with Kubernetes���� appeared first on �鶹����.

Our quick ADQ in three minutes guide can be downloaded from the button below the graphic. Happy reading! As always, don’t hesitate to get in touch if you’re looking for an answer that you can’t find here. Simply hit ‘Contact us’ with your query and let us do the rest.

Wherever you are on your data journey, we have the expertise, the tooling and the guidance to help accelerate your data quality initiatives. From connecting to data sources, through rule building, measuring and into improving the quality of data your business relies on, let ADQ be your trusted partner.

If you would like to read some customer stories of how we’ve already achieved this, head on over to our Resources page where you’ll find a wide range of customer case studies, white papers, blogs and testimonials.

To get hold of this infographic, simply hit Download this! below.

The post Got three minutes? Get all you need to know on ADQ! appeared first on �鶹����.

Tia Cheang, Director of IT Data and Information Services at Gallagher, recently delivered an interview with Tech-Exec magazine drawing from her knowledge and experience in shaping the future of the insurance industry at one of the world’s largest insurance brokers. You can read the article .

Tia is also one of DataIQ’s (congratulations, Tia!). We took the opportunity to ask Tia a few questions of our own, building on some of the themes from the Tech-Exec interview.

In the article with Tech-Exec, you touched on your background, your drive and ambition, and what led you to your current role at Gallagher. What are you most passionate about in this new role?

In 2023, I started working at Gallagher after having an extensive career in data in both public and private sectors. This job was a logical next step for me, as it resonates with my longstanding interest in utilising data in creative ways to bring about beneficial outcomes. I was eager to manage a comprehensive data transformation at Gallagher to prepare for the future, aligning with my interests and expertise.

I am responsible for leading our data strategy and developing a strong data culture. We wish to capitalise on data as a route to innovation and strategic decision-making. Our organisation is therefore creating an environment where data plays a crucial role in our business operations, to allow us to acquire new clients and accomplish significant results rapidly. The role offers an exciting opportunity to combine my skills and lead positive changes in our thinking towards data and its role in the future of insurance.

The transition to making data an integral part of business operations is often challenging. How have you found the experience? 

At Gallagher, our current data infrastructure faces the typical challenges that arise when a firm is expanding. Our data warehouses collect data from many sources, which mirrors the diverse aspects of our brokerage activities. These encompass internal systems, such as customer relationship management (CRM), brokerage systems, and other business applications. We handle multiple data types in our data estate, ranging from structured numerical data to unstructured text. The vast majority of our estate is currently hosted on-premise using Microsoft SQL Server technology, however, we also manage various other departmental data platforms such as QlikView. 

“…we want data capabilities that provide flexibility and agility, to enable us to quickly react to new market opportunities.”

A key challenge we face is quickly incorporating new data sources obtained through our mergers and acquisitions activity. These problems affect our data management efforts in terms of migration, seamless data integration, maintaining data quality, and providing data accessibility.��
To overcome this, we want data capabilities that provide flexibility and agility, to enable us to quickly react��to new market opportunities. Consequently, we are implementing a worldwide data transformation to update our data technology, processes, and skills to provide support for this initiative. This transformation��will move Gallagher data to the cloud, using Snowflake��to leverage the scalability and elasticity of the platform for advanced analytics. Having this flexibility gives us��a major advantage, offering computational resources where and when they are required.

How does this technology strategy align with your data strategy, and how do you plan to ensure data governance and compliance while implementing these solutions, especially in a highly-regulated industry like insurance?

Gallagher’s data strategy aims to position us as the leader in the insurance sector. By integrating our chosen solutions within the Snowflake platform, we strive to establish a higher standard in data-driven decision-making. 

This strategy involves incorporating data management tools such as Collibra, CluedIn, and �鶹���� into our re-platforming efforts, with a focus on ensuring the compatibility and interoperability of each component. We are aligning each tool’s capabilities with Snowflake’s powerful data lake functionality with the support of our consulting partners to ensure that our set of tools function seamlessly within Snowflake’s environment.

“…we are contemplating upcoming AI and automation regulations and considering how to futureproof our products and approaches…”

We are meticulously navigating the waters of data governance and compliance. We carefully plan each stage to ensure that all components of our data governance comply with the industry regulations and legislation of the specific region. For example, we are contemplating upcoming AI and automation regulations and considering how to futureproof our products and approaches to comply with them.

The success of our programme requires cooperation across our different global regions, stakeholders, and partners. We are rethinking our data governance using a bottom-up approach tailored to the specific features of our global insurance industry. We review our documentation and test the methods we use to ensure they comply with regulations and maintain proper checks and balances. We seek to understand the operational aspects of a process in real-world scenarios and evaluate its feasibility and scalability.

Could you expand on your choice of multiple solutions for data management technology? What made you go this route over a one-stop shop for all technologies?

We have selected “best of breed”��solutions for data quality, data lineage, and Master Data Management (MDM), based on a requirement for specialised, high-performance tools. We concentrated on high-quality enterprise solutions for easy integration with our current technologies. Our main priorities were security, scalability, usability, and compatibility with our infrastructure.��

By adopting this approach, we achieve enhanced specialisation and capabilities��in each area, providing high-level performance. This strategy offers the necessary flexibility within the organisation to establish a unified data management ecosystem. This aligns with our strategic objectives, ensuring that our data management capability is scaleable, secure, and adaptable.

Regarding the technologies we have selected, Collibra increases data transparency through efficient cataloguing��and clear lineage; CluedIn ensures consistent and reliable data across systems; and �鶹���� is critical��for maintaining high-quality data.��

“As we venture into advanced analytics, the importance of our data quality increases.”

In �鶹����’ case, it provides data cleansing tools that ensure the reliability and accuracy of our data, underpinning effective decision-making and strategic planning. The benefits of this are immense, enhancing operating efficiency, reducing errors, and enabling well-informed decisions. As we venture into advanced analytics, the importance of our data quality increases. Therefore, �鶹���� was one of the first technologies we started using.

We anticipate gaining substantial competitive advantages from our strategic investment, such as improved decision-making capabilities, operational efficiency, and greater customer insights for personalisation. Our ability to swiftly adapt to market changes is also boosted. Gallagher’s adoption of automation and AI technologies will also strengthen our position, ensuring we remain at the forefront of technological progress.

On Master Data Management (MDM), you referred to the importance of having dedicated technology for this purpose. How do you see MDM making a difference at Gallagher, and what approach are you taking?

Gallagher is deploying Master Data Management to provide a single customer view. We expect substantial improvements in operational efficiency and customer service when it is completed. This will improve processing efficiency by removing duplicate data and offering more comprehensive, actionable customer insights. These improvements will benefit the insurance brokerage business and will enable improved data monetisation and stronger compliance, eventually enhancing client experience and increasing operational efficiency.

Implementing MDM at Gallagher is foundational to our ability to enable global analytics and automation. To facilitate it, we need to create a unified, accurate, and accessible data environment. We plan to integrate MDM seamlessly with our existing data systems, leveraging tools like CluedIn to manage reference data efficiently. This approach ensures that our MDM solution supports our broader data strategy, enhancing our overall data architecture.

“By including data quality activities in our approach, we anticipate significant benefits from the MDM initiative.”

Data quality is crucial in Gallagher’s journey��to achieve this, particularly in establishing��a unified consumer view via MDM. Accurate and consistent��data is essential for consolidating several client data sources into a master��profile; we see it as essential, as without good data quality the benefits of our transformation will be reduced. By including data quality activities in our approach, we anticipate significant benefits from the MDM initiative. We foresee a marked improvement��in data accuracy and consistency throughout all business units. We want to empower��users across the organisation to make more informed, data-driven decisions to facilitate growth. Furthermore, a single source of truth enables us to streamline��our operations, leading to greater efficiencies by removing manual processes. Essentially, this strategic MDM implementation transforms data into a valuable asset that drives innovation and growth for Gallagher.

Looking to the future of insurance, what challenges do you foresee in technology, data and the insurance market?

Keeping up with the fast speed of technology changes can be challenging. We are conducting horizon scanning on new technologies to detect emerging trends. We wish to include new tools and processes that will complement and improve our current systems as they become ready.

“We prioritise the security of our data assets and our clients’ privacy because it is essential for our reputation and confidence in the market.”

Next is ensuring robust data security and compliance,��particularly when considering legislation changes about AI and data protection. Our approach is to continuously strengthen our data policies as we grow and proactively manage our data. We prioritise the security of our data assets and our clients’ privacy because it is essential for our reputation and confidence in the market.

Finally, we work closely with our technology partners to leverage��their expertise. This collaborative approach ensures that we take advantage of new technologies to their maximum capacity while preserving the integrity and effectiveness of our current systems.��

Are there any other technologies or methodologies you are considering for improving data management in the future beyond what you have mentioned?

Beyond the technologies and strategies already mentioned, at Gallagher, we plan to align our data management practices with the principles outlined in /(Data Management Body of Knowledge). This framework will ensure that our data management capabilities are not just technologically advanced but also adhere to the best practices and standards in the industry.

In addition to this, we are always on the lookout for emerging technologies and methodologies that could further enhance our data management. Whether it’s advancements in AI, machine learning, or new data governance frameworks, we are committed to exploring and adopting methodologies that can add value to our data management practices.

For more from Tia, you can find her on .

The post Shaping the Future of Insurance: Insights from Tia Cheang appeared first on �鶹����.

Why does FSCS compliance matter?

HSBC Bank plc (HBEU) and HSBC UK Bank plc (HBUK)’s January 2024 fine, imposed by the for historic failures in deposit protection identification and notification, alongside the , jointly serve as stark reminders of the importance of depositor protection regulation.

Both events, emblematic of the broader challenges faced by the banking sector, underscore the necessity of rigorous data governance and quality for FSCS compliance and depositor protection.

HSBC’s penalty, the second largest imposed by the PRA, highlights the consequences of inadequate data management practices, while the 2023 US banking crisis, characterised by the failure of three small-to-midsize banks, reveals the systemic risks posed by liquidity concerns and market instability.

These incidents draw attention not only to the pressing issues of today, but also to the enduring mechanisms put in place to safeguard financial stability. The , established in the United Kingdom, embodies such a mechanism, created to instil consumer confidence and prevent the domino effect of bank runs.

What is Single Customer View (SCV)?

The FSCS’s role becomes especially pivotal in times of uncertainty: if a bank collapses, the FSCS’s compensation mechanism needs to activate almost instantaneously to maintain this confidence.

According to the (Section 12), firms are required to produce a Sin (SCV) — a comprehensive record of eligible guaranteed deposits — within 24 hours of a bank’s failure or whenever the PRA or FSCS requests it.

This response, underscored by the accuracy and rapidity of depositor information, is a bulwark designed to avert a banking crisis by ensuring timely compensation for affected customers. Over time, as the FSCS has amplified depositor protection to cover up to £85,000 per individual, the 24-hour SCV mandate has marked a significant stride towards a more secure and robust financial sector, solidifying the foundation where depositor trust is paramount.

What data challenges does SCV pose?

When it comes to implementing the SCV regulation, the devil lies in the details. The demand for accuracy and consistency in depositor records translates into specific, often arduous, data quality challenges. Financial institutions must ensure that each depositor’s record is not only accurate but also aligned with SCV’s granular requirements:

Below are 5 data challenges associated with SCV:

• Identification and rectification of duplicated records– Duplication can occur due to disparate data entry points or legacy systems not communicating effectively.
• Lack of consistency across records– Customer details may have slight variations across different systems, such as misspelt names or outdated addresses, which can impede the quick identification of accounts under SCV mandates.
• Data timeliness– SCV necessitates that data be updated within a 24-hour window, requiring real-time (or near-real-time) processing capabilities. Legacy systems, often built on batch processing, may struggle to adapt to this requirement.
• Discrepancies in account status — determining if an account is active, dormant, or closed — must be resolved to prevent compensation delays or errors.
• Aggregating siloed data– the comprehensive nature of depositor information mandated by SCV involves aggregating data across multiple product lines, account types, and even geographical locations for international banks, a task that can be formidable given the legacy data structures and the diversity of regulatory environments.

The HSBC fine, in particular, underscores the ramifications of inaccurate risk categorisation under the depositor protection rules and the insufficiency of stress testing scenarios tailored to depositor data. Without robust data quality controls, banks risk misclassifying depositor coverage, which could potentially lead to regulatory sanctions and reputational damage.

Why integrate SCV with wider data strategies?

By incorporating meticulous data standards and validation processes as part of an enterprise strategy, banks can transform data management from a regulatory burden into a strategic asset.

The crux of effective depositor protection lies not just in adhering to SCV requirements, but in embracing a broader perspective on data governance and quality. This means positioning SCV not in isolation but as a critical component of a comprehensive account and customer-level data strategy.

To overcome these challenges, financial institutions must not only deploy advanced data governance and quality tooling but also foster a culture of data stewardship where data quality is an enterprise-wide responsibility and not one that is siloed within IT departments. By incorporating meticulous data standards and validation processes as part of an enterprise strategy, banks can transform data management from a regulatory burden into a strategic asset.

An enterprise approach involves:

• Unified Data Governance Frameworks: Establishing unified data governance frameworks that ensure data accuracy, consistency, and accessibility across the enterprise.

• Advanced Data Quality Measures: Implementing advanced data quality measures that address inaccuracies and inconsistencies head-on, ensuring that all customer data is up-to-date and reliable.

• Integration with Broader Business Objectives: Aligning SCV and other regulatory data requirements with broader business objectives, including risk management, customer experience enhancement, and operational efficiency.

• Leveraging Technology and Analytics: Employing cutting-edge technology and analytics to streamline data management processes, from data collection and integration to analysis and reporting.

How does �鶹���� support FSCS compliance?

The recent HSBC fine and the 2023 US banking crisis serve as critical catalysts for reflection on the role of depositor protection regulation and the imperative of a holistic data strategy.

FSCS regulatory reporting compliance underscores the evolution of depositor protection in response to financial crises, whilst the challenges presented by these regulations highlight the need for advanced data governance and quality measures.

At �鶹����, we understand that the challenges posed by regulations like SCV indicate broader issues within data management and governance.

Our approach transcends the piecemeal addressing of regulatory requirements; instead, we advocate for and implement a comprehensive data strategy that integrates SCV within the wider context of account and customer-level data management.

Our solutions are designed to support regulatory compliance but also to bolster the overall data governance and quality framework of financial institutions.

We work closely with our clients to:

• Identify and Address Data Quality Issues: Through advanced analytics and machine learning, we pinpoint and rectify data quality issues, ensuring compliance and enhancing overall data integrity.

• Implement Robust Data Governance Practices: We help institutions establish and maintain robust data governance practices that align with both regulatory requirements and business goals.

• Foster a Culture of Data Excellence: Beyond technical solutions, we emphasise the importance of fostering a culture that values data accuracy, consistency, and transparency.

We are committed to helping our customers navigate FSCS compliance, not by addressing regulations in isolation but by integrating them into a broader, strategic and more sustainable framework of account and customer-level data management. By doing so, we ensure compliance and protection for depositors whilst paving the way for a more resilient, trustworthy, and efficient banking sector.

The post FSCS compliance: The Future of Depositor Protection appeared first on �鶹����.

What are Large Language Models (LLMs) and GPTs?

In today’s rapidly evolving digital landscape, two acronyms have been making waves across industries: LLMs and GPTs. But what do these terms really mean, and why are they becoming increasingly important?��

The post What are Large Language Models (LLM) and GPTs? appeared first on �鶹����.

A data quality firewall is a key component of data management. It is a form of data quality monitoring, using software to prevent the ingestion of messy or bad data.

It’s a set of measures or processes to ensure the integrity, accuracy, and reliability of data within an organisation, and helps support data governance strategies. This could involve controls and checks to prevent the entry of inaccurate or incomplete data from data sources into data stores, as well as mechanisms to identify and rectify any data quality issues that arise.��

In its simplest form, a data quality firewall could be data stewards manually checking the data. However, this isn’t recommended, as it’s considerably inefficient and could cause inaccuracies.�� Instead, a more effective approach is the use of automation.

An automated approach

(e.g. completeness, duplication, validity etc.) can be generated automatically and are useful for identifying a data quality issue. At �鶹����, with our expertise in AI-augmented data quality, we understand that the most value is derived from data quality rules that are highly specific to an organisation’s context. This includes rules focusing on Accuracy, Consistency, Duplication, and Validity. The ability to execute all the above rules should be a part of any data quality firewall.��

The above is perfectly suited to an API giving an on-demand view of the data’s health before ingestion into the warehouse. This real-time assessment ensures that only clean, high-quality data is stored, significantly reducing downstream errors and inefficiencies.

What Features are Required for a Data Quality Firewall?��

The ability to define Data Quality Requirements��

The ability to specify what data quality means for your organisation is key. For example, you may want to consider whether data should be processed in situ or passed through an API, depending on data volumes and other factors. Here are a couple of other questions worth considering when defining data quality requirements-��

• Which rules should be applied to the data?�� It goes without saying that not all data is the same. Rules which are highly applicable to the specific business context will be more useful than a generic completeness rule, for example. This may involve checking data types, ranges, and formats, or validation against sources of truth. Reject data that doesn’t meet the specified criteria.
• What should be done with broken data? Strategies for dealing with broken data should be flexible. Options might include quarantining the entire dataset, isolating only the problematic records, passing all data with flagged issues, or immediately correcting issues, like removing duplicates or standardising formats. All the above should be options for the user of the API.�� The point is, not every use case is the same and a one-size-fits-all solution won’t be sufficient.��

Key DQ Firewall Features:

Data Enrichment��

Data enrichment may involve adding identifiers and codes to the data entering the warehouse. This can help with usability and traceability.��

Logging and Auditing��

Robust logging and auditing mechanisms should be provided. Log all incoming and outgoing data, errors, and any data quality-related issues. This information can be valuable for troubleshooting and monitoring data quality over time.��

Error Handling��

A comprehensive error-handling strategy should be provided, with clearly defined error codes and messages to communicate issues with consumers of the API. Guidance on how to resolve or address data quality errors is provided.��

Reporting��

Regular reporting on data quality metrics and issues, including trend analysis, helps in keeping track of the data quality over time.

Documentation��

The API documentation should include information about data quality expectations, supported endpoints, request and response formats, and any specific data quality-related considerations.��

How �鶹���� can help��

You might have noticed that the concept of a Data Quality Firewall is not just limited to data entering an organisation. It’s equally valuable at any point in the data migration process, ensuring quality as data travels within an organisation. Wouldn’t it be nice to know the quality of your data is assured as it flows through your organisation?

�鶹���� can help with this. Our Augmented Data Quality (ADQ) solution uses AI and machine learning to streamline the process, providing advanced data profiling, outlier detection, and automated rule suggestions. Find out more about our ADQ platform here.

The post What is a Data Quality Firewall?�� appeared first on �鶹����.