Data. Consulting. Analytics. Technology.

Tag: tools

Data Quality Management

Dirty Data – Hygiene Etiquette

If you’ve ever analyzed data, you know the pain of digging into your data only to find that the data is poorly structured, full of inaccuracies, or just plain incomplete. But “dirty data” isn’t just a pain point for analysts; it can ultimately lead to missed opportunities and lost revenue to an organisation.  Gartner research shows that the “average financial impact of poor data quality on organizations is $9.7 million per year.”

The amount of time and energy it takes to go from disjointed data to actionable insights leads to inefficient ad-hoc analyses and declining trust in organizational data.

A recent Harvard Business Review study reports that people spend 80% of their time prepping data, and only 20% of their time analyzing it. And this statistic isn’t restricted to the role of the data stewards. Data prep tasks have bled into the work of analysts and even non-technical business users.

Enterprises are taking steps to overcome dirty data by establishing data hygiene etiquette:

  • Understand your data location, structure, and composition, along with granular details like field definitions.

Some people refer to this process as “data discovery” and it is a fundamental element of data    preparation. Confusion around data definitions, for example, can hinder analysis or worse, lead to inaccurate analyses across the company. For example, if someone wants to analyze customer data, they may find that a marketing team might have a different definition for the term“customer” than someone in finance.

  • Standardize data definitions across your company by creating a data dictionary.

This will help analysts understand how terms are used within each business application, showing the fields are relevant for analysis versus the ones that are strictly system-based. Developing a data dictionary is no small task. Data stewards and subject matter experts need to commit to ongoing iteration, checking in as requirements change. If a dictionary is out of date, it can actually do harm to your organization’s data strategy. Communication and ownership should be built into the process from the beginning to determine where the glossary should live and how often it should be updated and refined.

  • Data cleansing prior to imports

You need to prepare your data before even thinking of importing it in your system.  Every organization has specific needs and there is no ‘one size-fits-all’ approach to data preparation. A self-service data preparation tool allows people to see the full end-to-end process, seeing potential flags earlier on—like misspellings in the data, extra spaces, or incorrect join clauses. It also increases confidence in the final analysis.

  • Hands off!!

Keeping your hands out of the data in regular use increases the chances of it keeping clean. Introducing a little dirty data to a system will compromise an entire data set and your little bit of dirty data has suddenly created a lot of dirty data. Cleansing the mess is a far far bigger job than making sure the data is clean before importing it.

  • Invest in a self-service business intelligence tool

Adopting a self-service data prep across an organization requires users to learn the ins and outs of the data. Since this knowledge was historically reserved for IT and data engineering roles, it is crucial that analysts take time to learn about nuances within the data, including the granularity and any transformations that have been done to the data set. Scheduling regular check-ins or a standardized workflow for questions allows engineers to share the most up-to-date way to query and work with valid data, while empowering analysts to prepare data faster and with greater confidence.

Data hygiene should be a top concern in organisations. Devoting some resources to ensuring that the data you’re basing decisions on is complete and accurate is a smart investment, because dirty data is costly in so many ways. To get the most and best use out of your data, you need to take the time to ensure its quality is sufficient and that data used by different departments is integrated. This gives you the most complete and precise customer view, so you can make smarter decisions and maximize your return on investment.

Self-service Analytics

My World 2030 – Harness data to drive sustainability and corporate responsibility

Over the past couple of months if not years we have seen headline grabbing scandals in various sectors of the economy worldwide.  The public trust is being hugely impacted with regards to the competence of executive leadership, integrity and transparency.

Earning trust requires the utmost attention to demonstrating ethical leadership, responsible (and responsive) business practices, transparency, and a genuine commitment to an organisation’s mission.

Companies are being encouraged to put their increased profit into programs that give back to society in terms of environmental, social, and governance (ESG) aspects More than ever before, there’s growing expectations that organisations will continue to play a very active role in solving social problems such as poverty or discrimination. It’s important that organisations set standards of ethical behaviour for its peers, competition and industry.

So how can data drive sustainability and corporate responsibility? The writing is on the wall!  With the rapidly evolving technology and high velocity and volume of data flooding organisations, it becomes imperative to provide users with an ultimate analytics experience, one with zero discernible latency when interacting with data. By giving users the right tools, they will explore avenues to use data to solve real world problems.

As data becomes more available and analytic literacy more pervasive, it is crucial that companies continue to focus on how their business operations are impacting the value chain, from the farm to the factory to the boardroom.

With the advent of sensors and devices in mobile objects, companies can now leverage spatial data for powerful geospatial analysis for environmental risk assessments. The better the data sets available to assess these risks, the more informed the decisions about adaptation are likely to be.

Energy and Resources give modern society its high standard of living and produce vast quantities of data, from the energy used to the resources needed to make many of these things that help us in business and our everyday lives.

Without a deep understanding that energy is finite and that energy transformations impact not just individuals but also the environment., companies and society at large won’t be able to make informed decisions about the future. With an efficient platform for gaining insights across geographies, products, services, and sectors, companies can maximize downstream profits and minimize upstream costs.

It is the era of data-driven environmental policy-making. Governments can now harness data to effective policy making. Data analytics and visualisation give the opportunity to make the invisible visible, the intangible tangible, and the complex manageable. A data driven government calls for strong leadership and investment. This is highly feasible.  A data driven government would make it easier to identify problems, track trends, highlight policy successes and failures, identify best practices, and optimize the gains from investments in environmental protection. A responsive government would work in close collaboration with businesses, NGOs and the academic community for more conscientious environmental decision-making.

Data alone will not help us achieve the UN SDGs. What we need is strong leadership both from businesses and governments, transparency , integrity and a genuine commitment to the UN 17 SDGs. These combined with modern data analytics will provide collaborative, multilateral solutions to global challenges.

This is My World 2030!!

Data Governance, Data Regulations

GDPR and Data Governance: A hand in hand affair

The introduction of GDPR should not be seen as a burden for companies but rather as an opportunity to review all the data governance policies that are in place. Companies should be able to find the right balance between GDPR and their data governance structure.

Companies could create a competitive edge by not only addressing how they manage the personal data but for all the data they hold. If companies get it right, they could discover new business opportunities waiting to be exploited.

As we all know by now, the GDPR gives every EU citizen the right to know and decide how their personal data is being used, stored, protected, transferred and deleted.

Those companies that put data privacy at the forefront of their business strategy would be the ones who are clearly and efficiently managing their customer data in a fair and transparent way. Hence giving them the competitive edge based on privacy.

One of the requirements of GDPR is to document what personal data is held, where it came from and who is it shared with. By really understanding the data they hold, companies could be made aware of the data they can gather, as well as analyse and apply this data to boost sales or marketing efforts.

Companies should ensure that their data governance structure will support the GDPR requirements. Policies and procedures need to be created or re-assessed to help keep corporate data consistent and ensure that it meets the information needs of business users. It is also an opportunity to review data management practices.

The GDPR requirements combined with a robust data governance structure could give organisations the opportunity to become a data-driven company based on building tools, abilities, and a culture that acts on data hence really making an internal transformation around data.