DataKitchen

AUGUST 3, 2021

The data mesh design pattern breaks giant, monolithic enterprise data architectures into subsystems or domains, each managed by a dedicated team. But first, let’s define the data mesh design pattern. The past decades of enterprise data platform architectures can be summarized in 69 words.

Data Architecture 246

Data Architecture Data Lake Cost-Benefit Data Warehouse 246

DataKitchen

JANUARY 25, 2022

There’s no shortage of consultants who will promise to manage the end-to-end lifecycle of data from integration to transformation to visualization. . The challenge is that data engineering and analytics are incredibly complex. Ensuring that data is available, secure, correct, and fit for purpose is neither simple nor cheap.

Consulting 244

Consulting Testing Data Lake Data Quality 244

AWS Big Data

JUNE 6, 2023

We are excited to announce the General Availability of AWS Glue Data Quality. Our journey started by working backward from our customers who create, manage, and operate data lakes and data warehouses for analytics and machine learning. It takes days for data engineers to identify and implement data quality rules.

Data Quality 98

Data Quality Statistics Data Lake Visualization 98

AWS Big Data

DECEMBER 4, 2024

Domain ownership recognizes that the teams generating the data have the deepest understanding of it and are therefore best suited to manage, govern, and share it effectively. This principle makes sure data accountability remains close to the source, fostering higher data quality and relevance.

Metadata 105

Metadata Data Governance Data Quality Data-driven 105

Jet Global

JULY 2, 2019

The sheer scale of data being captured by the modern enterprise has necessitated a monumental shift in how that data is stored. What was at first a data stream has morphed into a data river as enterprise businesses are harvesting reams of data from every conceivable input across every conceivable business function.

Data Lake 73

Data Lake Data Warehouse Big Data Machine Learning 73

IBM Big Data Hub

MAY 9, 2023

We can use foundation models to quickly perform tasks with limited annotated data and minimal effort; in some cases, we need only to describe the task at hand to coax the model into solving it. But these powerful technologies also introduce new risks and challenges for enterprises.

Enterprise 128

Enterprise Technology Modeling Cost-Benefit 128

CIO Business Intelligence

AUGUST 14, 2024

Between building gen AI features into almost every enterprise tool it offers, adding the most popular gen AI developer tool to GitHub — GitHub Copilot is already bigger than GitHub when Microsoft bought it — and running the cloud powering OpenAI, Microsoft has taken a commanding lead in enterprise gen AI.

Enterprise 136

Enterprise Cost-Benefit Experimentation Modeling 136

CIO Business Intelligence

MARCH 9, 2022

Applying artificial intelligence (AI) to data analytics for deeper, better insights and automation is a growing enterprise IT priority. But the data repository options that have been around for a while tend to fall short in their ability to serve as the foundation for big data analytics powered by AI.

Data Lake 119

Data Lake Unstructured Data Data Warehouse Big Data 119

AWS Big Data

FEBRUARY 29, 2024

Data is your generative AI differentiator, and a successful generative AI implementation depends on a robust data strategy incorporating a comprehensive data governance approach. Data governance is a critical building block across all these approaches, and we see two emerging areas of focus.

Data Governance 124

Data Governance Unstructured Data Metadata Data Lake 124

CIO Business Intelligence

MAY 24, 2022

Just after launching a focused data management platform for retail customers in March, enterprise data management vendor Informatica has now released two more industry-specific versions of its Intelligent Data Management Cloud (IDMC) — one for financial services, and the other for health and life sciences.

Finance 140

Finance Management Metadata Machine Learning 140

CIO Business Intelligence

SEPTEMBER 20, 2023

On the agribusiness side we source, purchase, and process agricultural commodities and offer a diverse portfolio of products including grains, soybean meal, blended feed ingredients, and top-quality oils for the food industry to add value to the commodities our customers desire. The data can also help us enrich our commodity products.

Data Lake 135

Data Lake Data-driven Dashboards Risk 135

CIO Business Intelligence

JANUARY 20, 2023

Starting on a solid data foundation Before choosing a platform for sharing data, an organization needs to understand what data it already has and strip it of errors and duplicates. Data formats and data architectures are often inconsistent, and data might even be incomplete. They have data swamps,” he says.

Data Lake 135

Data Lake Data-driven Finance Data Architecture 135

IBM Big Data Hub

JANUARY 5, 2023

Poor data quality is one of the top barriers faced by organizations aspiring to be more data-driven. Ill-timed business decisions and misinformed business processes, missed revenue opportunities, failed business initiatives and complex data systems can all stem from data quality issues.

Data Architecture 52

Data Architecture Data Quality Strategy Data Lake 52

AWS Big Data

DECEMBER 13, 2023

For the past 5 years, BMS has used a custom framework called Enterprise Data Lake Services (EDLS) to create ETL jobs for business users. Data quality check – The data quality module enables you to perform quality checks on a huge amount of data and generate reports that describe and validate the data quality.

Metadata 111

Metadata Data Lake Visualization Data Quality 111

AWS Big Data

NOVEMBER 16, 2023

Amazon Redshift is a popular cloud data warehouse, offering a fully managed cloud-based service that seamlessly integrates with an organization’s Amazon Simple Storage Service (Amazon S3) data lake, real-time streams, machine learning (ML) workflows, transactional workflows, and much more—all while providing up to 7.9x

Enterprise 111

Enterprise Data Warehouse Snapshot Cost-Benefit 111

erwin

AUGUST 31, 2023

Q: Is data modeling cool again? In today’s fast-paced digital landscape, data reigns supreme. The data-driven enterprise relies on accurate, accessible, and actionable information to make strategic decisions and drive innovation. The continued federation of data in the enterprise resulted in data silos.

Data-driven 98

Data-driven Modeling Enterprise Structured Data 98

Timo Elliott

JANUARY 3, 2022

It’s stored in corporate data warehouses, data lakes, and a myriad of other locations – and while some of it is put to good use, it’s estimated that around 73% of this data remains unexplored. Improving data quality. Unexamined and unused data is often of poor quality. Data augmentation.

IT 143

IT Unstructured Data Data Quality Machine Learning 143

CIO Business Intelligence

APRIL 29, 2022

This would be straightforward task were it not for the fact that, during the digital-era, there has been an explosion of data – collected and stored everywhere – much of it poorly governed, ill-understood, and irrelevant. Further, data management activities don’t end once the AI model has been developed. Addressing the Challenge.

Data Governance 145

Data Governance IT Data Lake Risk 145

Alation

JUNE 17, 2021

Lastly, active data governance simplifies stewardship tasks of all kinds. Tehnical stewards have the tools to monitor data quality, access, and access control. A compliance steward is empowered to monitor sensitive data and usage sharing policies at scale. The Data Swamp Problem. The Governance Solution.

Data Governance 98

Data Governance Data Lake Insurance Enterprise 98

erwin

JULY 17, 2019

Part Two of the Digital Transformation Journey … In our last blog on driving digital transformation , we explored how enterprise architecture (EA) and business process (BP) modeling are pivotal factors in a viable digital transformation strategy. With automation, data quality is systemically assured.

Digital Transformation 98

Digital Transformation Strategy Metadata Data-driven 98

AWS Big Data

MARCH 1, 2024

One of the bank’s key challenges related to strict cybersecurity requirements is to implement field level encryption for personally identifiable information (PII), Payment Card Industry (PCI), and data that is classified as high privacy risk (HPR). Only users with required permissions are allowed to access data in clear text.

Data Lake 119

Data Lake Data Warehouse Management Risk 119

CIO Business Intelligence

OCTOBER 2, 2024

And Doug Shannon, automation and AI practitioner, and Gartner peer community ambassador, says the vast majority of enterprises are now focused on two categories of use cases that are most likely to deliver positive ROI. Classifiers are provided in the toolkits to allow enterprises to set thresholds. “We

Management 141

Management Data Governance Cost-Benefit Structured Data 141

Timo Elliott

DECEMBER 14, 2022

Gartner calls it the Composable Enterprise , for example – it’s about having a solid information foundation that enables fast and flexible creation of what they call composable applications that allow you to create new applications and workflows by just bringing together modular components. Business Process. Business Context.

Data Lake 105

Data Lake Recreation/Entertainment Data Warehouse Metadata 105

IBM Big Data Hub

JUNE 8, 2022

Implementing the right data strategy spurs innovation and outstanding business outcomes by recognizing data as a critical asset that provides insights for better and more informed decision-making. By taking advantage of data, enterprises can shape business decisions, minimize risk for stakeholders, and gain competitive advantage.

Data-driven 56

Data-driven Enterprise Data Governance Data Lake 56

AWS Big Data

MARCH 7, 2023

Analytics reference architecture for gaming organizations In this section, we discuss how gaming organizations can use a data hub architecture to address the analytical needs of an enterprise, which requires the same data at multiple levels of granularity and different formats, and is standardized for faster consumption.

Analytics 97

Analytics Data Warehouse Data Lake Metadata 97

Cloudera

JUNE 13, 2024

After countless open-source innovations ushered in the Big Data era, including the first commercial distribution of HDFS (Apache Hadoop Distributed File System), commonly referred to as Hadoop, the two companies joined forces, giving birth to an entire ecosystem of technology and tech companies.

Big Data 103

Big Data Machine Learning Contextual Data Data Lake 103

AWS Big Data

AUGUST 1, 2024

Migrating to Amazon Redshift offers organizations the potential for improved price-performance, enhanced data processing, faster query response times, and better integration with technologies such as machine learning (ML) and artificial intelligence (AI).

Data Warehouse 115

Data Warehouse KPI Optimization Cost-Benefit 115

Grooper

MARCH 5, 2020

These stewards monitor the input and output of data integrations and workflows to ensure data quality. Their focus is on master data management , data lakes / warehouses, and ensuring the trackability of data using audit trails and metadata. How to Get Started with Information Stewardship.

Data Lake 98

Data Lake Metadata Data Quality Software 98

CIO Business Intelligence

NOVEMBER 28, 2023

Making the most of enterprise data is a top concern for IT leaders today. With organizations seeking to become more data-driven with business decisions, IT leaders must devise data strategies gear toward creating value from data no matter where — or in what form — it resides. Quality is job one.

Unstructured Data 130

Unstructured Data Data-driven Visualization Data Quality 130

AWS Big Data

DECEMBER 19, 2024

Data has become an invaluable asset for businesses, offering critical insights to drive strategic decision-making and operational optimization. From establishing an enterprise-wide data inventory and improving data discoverability, to enabling decentralized data sharing and governance, Amazon DataZone has been a game changer for HEMA.

Data Governance 52

Data Governance Publishing Data-driven Metadata 52

datapine

APRIL 15, 2020

It’s necessary to say that these processes are recurrent and require continuous evolution of reports, online data visualization , dashboards, and new functionalities to adapt current processes and develop new ones. Testing will eliminate lots of data quality challenges and bring a test-first approach through your agile cycle.

Business Intelligence 276

Business Intelligence Analytics Testing Dashboards 276

Cloudera

AUGUST 30, 2022

For state and local agencies, data silos create compounding problems: Inaccessible or hard-to-access data creates barriers to data-driven decision making. Legacy data sharing involves proliferating copies of data, creating data management, and security challenges. Towards Data Science ). Forrester ).

Data Architecture 79

Data Architecture Data Lake Data Warehouse Metadata 79

AWS Big Data

NOVEMBER 9, 2023

It allows users to write data transformation code, run it, and test the output, all within the framework it provides. Use case The Enterprise Data Analytics group of a large jewelry retailer embarked on their cloud journey with AWS in 2021. AWS Glue – AWS Glue is used to load files into Amazon Redshift through the S3 data lake.

Data Warehouse 126

Data Warehouse Testing Data Quality Reporting 126

CIO Business Intelligence

JANUARY 30, 2024

Birgit Fridrich, who joined Allianz as sustainability manager responsible for ESG reporting in late 2022, spends many hours validating data in the company’s Microsoft Sustainability Manager tool. Data quality is key, but if we’re doing it manually there’s the potential for mistakes.

Reporting 115

Reporting Data Quality Strategy Data-driven 115

CIO Business Intelligence

FEBRUARY 16, 2023

Selling the value of data transformation Iyengar and his team are 18 months into a three- to five-year journey that started by building out the data layer — corralling data sources such as ERP, CRM, and legacy databases into data warehouses for structured data and data lakes for unstructured data.

Unstructured Data 105

Unstructured Data Data Lake Prescriptive Analytics Data Warehouse 105

AWS Big Data

APRIL 4, 2024

Many enterprises are migrating their on-premises data stores to the AWS Cloud. During data migration, a key requirement is to validate all the data that has been moved from source to target. This data validation is a critical step, and if not done correctly, may result in the failure of the entire project.

Data Quality 87

Data Quality Data Lake Data Warehouse Data-driven 87

Cloudera

MARCH 28, 2024

Among the most common challenges to achieving AI adoption at scale were data quality and availability (36%), scalability and deployment (36%), integration with existing systems and processes (35%), and change management and organizational culture (34%).

Data Architecture 94

Data Architecture Strategy Data Lake Data-driven 94