CIO Business Intelligence

APRIL 8, 2025

These strategies, such as investing in AI-powered cleansing tools and adopting federated governance models, not only address the current data quality challenges but also pave the way for improved decision-making, operational efficiency and customer satisfaction. When financial data is inconsistent, reporting becomes unreliable.

Data Quality 102

Data Quality Data-driven Key Performance Indicator Metadata 102

Wayne Yaddow

MARCH 14, 2025

How dbt Core aids data teams test, validate, and monitor complex data transformations and conversions Photo by NASA on Unsplash Introduction dbt Core, an open-source framework for developing, testing, and documenting SQL-based data transformations, has become a must-have tool for modern data teams as the complexity of data pipelines grows.

Data Transformation 52

Data Transformation Testing Unstructured Data Data Quality 52

datapine

SEPTEMBER 29, 2022

Business/Data Analyst: The business analyst is all about the “meat and potatoes” of the business. These needs are then quantified into data models for acquisition and delivery. This person (or group of individuals) ensures that the theory behind data quality is communicated to the development team. 2 – Data profiling.

Data Quality 241

Data Quality Metrics Data-driven Management 241

AWS Big Data

NOVEMBER 29, 2023

dbt is an open source, SQL-first templating engine that allows you to write repeatable and extensible data transforms in Python and SQL. dbt is predominantly used by data warehouses (such as Amazon Redshift ) customers who are looking to keep their data transform logic separate from storage and engine.

Data Lake 122

Data Lake Management Metrics Data Warehouse 122

Cloudera

JUNE 1, 2021

This means we can double down on our strategy – continuing to win the Hybrid Data Cloud battle in the IT department AND building new, easy-to-use cloud solutions for the line of business. It also means we can complete our business transformation with the systems, processes and people that support a new operating model. .

Uncertainty 144

Uncertainty Cost-Benefit Risk Strategy 144

IBM Big Data Hub

OCTOBER 16, 2023

It includes processes that trace and document the origin of data, models and associated metadata and pipelines for audits. Foundation models: The power of curated datasets Foundation models , also known as “transformers,” are modern, large-scale AI models trained on large amounts of raw, unlabeled data.

Risk 70

Risk Modeling Management Metadata 70

AWS Big Data

JULY 18, 2024

OpenSearch is an open source, distributed search engine suitable for a wide array of use-cases such as ecommerce search, enterprise search (content management search, document search, knowledge management search, and so on), site search, application search, and semantic search. OpenSearch also includes capabilities to ingest and analyze data.

Dashboards 114

Dashboards Testing Data-driven Visualization 114

CIO Business Intelligence

NOVEMBER 7, 2023

Given the importance of sharing information among diverse disciplines in the era of digital transformation, this concept is arguably as important as ever. The aim is to normalize, aggregate, and eventually make available to analysts across the organization data that originates in various pockets of the enterprise.

Data Warehouse 142

Data Warehouse Digital Transformation Data-driven Reporting 142

AWS Big Data

NOVEMBER 9, 2023

In this post, we delve into a case study for a retail use case, exploring how the Data Build Tool (dbt) was used effectively within an AWS environment to build a high-performing, efficient, and modern data platform. It does this by helping teams handle the T in ETL (extract, transform, and load) processes.

Data Warehouse 126

Data Warehouse Testing Data Quality Reporting 126

Cloudera

OCTOBER 7, 2022

dbt allows data teams to produce trusted data sets for reporting, ML modeling, and operational workflows using SQL, with a simple workflow that follows software engineering best practices like modularity, portability, and continuous integration/continuous development (CI/CD). The Open Data Lakehouse . Introduction.

Data Warehouse 81

Data Warehouse Data Transformation Machine Learning Data Lake 81

AWS Big Data

SEPTEMBER 13, 2024

In recent years, driven by the commoditization of data storage and processing solutions, the industry has seen a growing number of systematic investment management firms switch to alternative data sources to drive their investment decisions. The bulk of our data scientists are heavy users of Jupyter Notebook. or later.

Interactive 107

Interactive Strategy Cost-Benefit Data Governance 107

Sisense

AUGUST 6, 2020

Take Grammarly as an example: This popular program checks the grammar, tone, and style of documents. Getting this AI properly trained required a huge learning dataset with countless documents that were tagged according to specific criteria. Accurately prepared data is the base of AI. What will it take to build your MVP?

Management 95

Management Machine Learning Key Performance Indicator Cost-Benefit 95

Cloudera

JUNE 24, 2024

The complexities of modern data workflows often translate into countless hours spent coding, debugging, and optimizing models. Recognizing this pain point, we set out to redefine the data science experience with AI-driven innovation. This practical support speeds up project initiation and maintains consistent coding practices.

Machine Learning 72

Machine Learning Data Science Data-driven Testing 72

IBM Big Data Hub

JANUARY 9, 2024

Companies still often accept the risk of using internal data when exploring large language models (LLMs) because this contextual data is what enables LLMs to change from general-purpose to domain-specific knowledge. In the generative AI or traditional AI development cycle, data ingestion serves as the entry point.

Enterprise 71

Enterprise Data Integration Data Quality Contextual Data 71

Cloudera

OCTOBER 8, 2024

This service supports a range of optimized AI models, enabling seamless and scalable AI inference. Enterprise developers began exploring proof of concepts (POCs) for generative AI applications, leveraging API services and open models such as Llama 2 and Mistral. By 2023, the focus shifted towards experimentation.

Optimization 75

Optimization Experimentation Metrics Enterprise 75

Ontotext

DECEMBER 9, 2020

We all want to solve the interesting data challenges, build analytics, generate graph embeddings and train smart machine learning models over our knowledge graph data. This leads to lots of small data fetches to/from GraphDB over the network. Custom code also tends to over-fetch data that is not required.

Modeling 130

Modeling Management Optimization Machine Learning 130

Alation

OCTOBER 27, 2022

Few actors in the modern data stack have inspired the enthusiasm and fervent support as dbt. This data transformation tool enables data analysts and engineers to transform, test and document data in the cloud data warehouse. Jason: How do you use these models?

Metrics 52

Metrics Dashboards Sales Reporting 52

AWS Big Data

AUGUST 1, 2024

However, you might face significant challenges when planning for a large-scale data warehouse migration. As part of the success criteria for operational service levels, you need to document the expected service levels for the new Amazon Redshift data warehouse environment. Platform architects define a well-architected platform.

Data Warehouse 115

Data Warehouse KPI Optimization Cost-Benefit 115

Cloudera

AUGUST 3, 2022

The challenges of arbitrary code execution notwithstanding, there have been attempts to provide a stronger security model but with mixed results. By leveraging Hive to apply Ranger FGAC, Spark obtains secure access to the data in a protected staging area. Learn more on how to use the feature from our public documentation. .

Snapshot 90

Snapshot Cost-Benefit Machine Learning Data Science 90

AWS Big Data

AUGUST 19, 2024

Efficient data integration – AWS Glue simplifies the ETL process, providing a scalable and flexible solution for data integration between Snowflake and Amazon S3. Scalability and flexibility – The architecture supports scalable data transfers and can be extended to integrate additional data sources and destinations as needed.

Analytics 119

Analytics Data-driven Data Integration Data Lake 119

AWS Big Data

AUGUST 26, 2024

In this post, I’ll walk you through how to copy data from one Amazon Relational Database Service (Amazon RDS) for PostgreSQL database to another, while scrubbing PII along the way using AWS Glue. Built-in data transformations then scrub columns containing PII using pre-defined masking functions. PII detection and scrubbing.

Visualization 116

Visualization Metadata Data Transformation Testing 116

AWS Big Data

MARCH 13, 2024

You can modify the Lambda function to fetch additional vehicle information from a separate data store (for example, a DynamoDB table or a Customer Relationship Management system) to enrich the data, before storing the results in an S3 bucket. In this model, the Lambda function is invoked for each incoming event.

Analytics 122

Analytics IoT Metadata Internet of Things 122

IBM Big Data Hub

JULY 17, 2023

is our enterprise-ready next-generation studio for AI builders, bringing together traditional machine learning (ML) and new generative AI capabilities powered by foundation models. With watsonx.ai, businesses can effectively train, validate, tune and deploy AI models with confidence and at scale across their enterprise. IBM watsonx.ai

Machine Learning 62

Machine Learning Data Warehouse Modeling Cost-Benefit 62

erwin

JANUARY 11, 2019

Organizations have spent a lot of time and money trying to harmonize data across diverse platforms , including cleansing, uploading metadata, converting code, defining business glossaries, tracking data transformations and so on.

Data Governance 40

Data Governance Risk Metadata Management 40

AWS Big Data

APRIL 19, 2023

This allows for a new way of thinking and new organizational elements—namely, a modern data community. However, today’s data mesh platform contains largely independent data products. Even with well-documented data products, knowing how to connect or join data products is a time-consuming job.

Technology 105

Technology Data-driven Machine Learning Sales 105

DataRobot Blog

MARCH 10, 2022

By enabling data scientists to rapidly iterate through model development, validation, and deployment, DataRobot provides the tools to blitz through steps four and five of the machine learning lifecycle with AutoML and Auto Time-Series capabilities. Train, Compare, Rank, Validate, and Select Models for Production.

Manufacturing 52

Manufacturing IoT Machine Learning Forecasting 52

bridgei2i

SEPTEMBER 3, 2020

Ronobijay: Sure, I think it would, you know, what used to be anathema till a few months back, you know, data transformation is real now, right? We would have to visit a branch possibly, you know, multiple locations, submit multiple documents. So earlier customers would spend a week or two, trying to open a bank account.

Enterprise 94

Enterprise Insurance Digital Transformation Interactive 94

Alation

JUNE 28, 2022

A metadata management framework combines organizational structure and a set of tools to create a data asset taxonomy. Document type: describes creation, storage, and use during business processes. Collaborate more effectively: Break down data silos for better understanding of data assets across all business units.

Metadata 52

Metadata Management Data Governance Machine Learning 52

IBM Big Data Hub

FEBRUARY 9, 2023

In our last blog , we delved into the seven most prevalent data challenges that can be addressed with effective data governance. Today we will share our approach to developing a data governance program to drive data transformation and fuel a data-driven culture.

Data Governance 76

Data Governance Business Objectives Data Quality Measurement 76

Octopai

JUNE 9, 2024

They invested heavily in data infrastructure and hired a talented team of data scientists and analysts. The goal was to develop sophisticated data products, such as predictive analytics models to forecast patient needs, patient care optimization tools, and operational efficiency dashboards.

IT 52

IT Data-driven Predictive Analytics Data Strategy 52

Alation

OCTOBER 18, 2022

Data analysts and engineers use dbt to transform, test, and document data in the cloud data warehouse. Yet every dbt transformation contains vital metadata that is not captured – until now. Data Transformation in the Modern Data Stack. Lineage between dbt sources, models, and metrics.

Metadata 52

Metadata Metrics Recreation/Entertainment Data Quality 52

AWS Big Data

JULY 25, 2023

Redshift Serverless automatically provisions and intelligently scales data warehouse capacity to deliver fast performance for even the most demanding and unpredictable workloads, and you pay only for what you use. Solution overview The integration of Talend with Amazon Redshift adds new features and capabilities.

Data Analytics 52

Data Analytics Analytics Data Warehouse Data Processing 52

CDW Research Hub

OCTOBER 25, 2019

As we review data transformation and modernization strategies with our clients, we find many are investigating Snowflake as a data warehouse solution due to its ease of use, speed, and increased flexibility over a traditional data warehouse offering. Mapping a successful data migration can bring on rough weather.

Data Warehouse 40

Data Warehouse Testing Strategy Data-driven 40

Ontotext

APRIL 4, 2019

We translate their documents, presentations, tables, etc. Milena Yankova : We help the BBC and the Financial Times to model the knowledge available in various documents so they can manage it. Milena Yankova : We help the BBC and the Financial Times to model the knowledge available in various documents so they can manage it.

Recreation/Entertainment 76

Recreation/Entertainment Testing Enterprise Knowledge Discovery 76

Domino Data Lab

AUGUST 20, 2019

Feature engineering is useful for data scientists when assessing tradeoff decisions regarding the impact of their ML models. It is a framework for approaching ML as well as providing techniques for extracting features from raw data that can be used within the models. Feature Engineering Terminology and Motivation.

Testing 68

Testing Modeling Interactive Measurement 68

Octopai

DECEMBER 11, 2022

It’s for that reason that even as the first BCBS-239 implementation deadline came into effect a few years ago, McKinsey reported that one-third of Global Systemically Important Banks had focused on “documenting data lineage up to the level of provisioning data elements and including data transformation.”.

Insurance 52

Insurance Risk Metadata Reporting 52

Alation

JANUARY 17, 2023

As data science is growing in popularity and importance , if your organization uses data science, you’ll need to pay more attention to picking the right tools for this. An example of a data science tool is Dataiku. Business Intelligence Tools: Business intelligence (BI) tools are used to visualize your data.

Data Warehouse 52

Data Warehouse Cost-Benefit Data Science Data Transformation 52