AWS Big Data

JUNE 10, 2024

For detailed information on managing your Apache Hive metastore using Lake Formation permissions, refer to Query your Apache Hive metastore with AWS Lake Formation permissions. In this post, we present a methodology for deploying a data mesh consisting of multiple Hive data warehouses across EMR clusters.

Data Lake 96

Data Lake Metadata Data Warehouse Data Processing 96

CIO Business Intelligence

JUNE 29, 2022

But this glittering prize might cause some organizations to overlook something significantly more important: constructing the kind of event-driven data architecture that supports robust real-time analytics. We can, in the semantics of the software world, refer to digitally mediated business activities asreal-time events.

Interactive 98

Interactive Data-driven Data Architecture Software 98

datapine

AUGUST 20, 2019

Operations data: Data generated from a set of operations such as orders, online transactions, competitor analytics, sales data, point of sales data, pricing data, etc. The gigantic evolution of structured, unstructured, and semi-structured data is referred to as Big data. Videos, pictures etc.

Big Data 100

Big Data B2B Cost-Benefit Structured Data 100

Cloudera

OCTOBER 7, 2021

Those decentralization efforts appeared under different monikers through time, e.g., data marts versus data warehousing implementations (a popular architectural debate in the era of structured data) then enterprise-wide data lakes versus smaller, typically BU-Specific, “data ponds”.

Metadata 124

Metadata Cost-Benefit Enterprise Interactive 124

TDAN

MARCH 17, 2020

Earlier this month we hosted the second annual Data-Centric Architecture Forum (#DCAF2020) in Fort Collins, CO. Last year, (2019) we hosted the first Data-Centric Architecture conference. In 2019, the focus was on getting a sketch of a reference architecture (click here to see). Trip report).

Reporting 52

Reporting Data Processing Data Architecture Data Strategy 52

AWS Big Data

SEPTEMBER 5, 2024

The currently available choices include: The Amazon Redshift COPY command can load data from Amazon Simple Storage Service (Amazon S3), Amazon EMR , Amazon DynamoDB , or remote hosts over SSH. This native feature of Amazon Redshift uses massive parallel processing (MPP) to load objects directly from data sources into Redshift tables.

IoT 106

IoT Data Warehouse Cost-Benefit Reporting 106

AWS Big Data

OCTOBER 18, 2023

To create it, refer to Tutorial: Get started with Amazon EC2 Windows instances. To download and install AWS SCT on the EC2 instance that you created, refer to Installing, verifying, and updating AWS SCT. For more information about bucket names, refer to Bucket naming rules. Select Redshift data agent , then choose OK.

Analytics 107

Analytics Data Warehouse Dashboards Testing 107

Sisense

AUGUST 28, 2020

Uncomfortable truth incoming: Most people in your organization don’t think about the quality of their data from intake to production of insights. However, as a data team member, you know how important data integrity (and a whole host of other aspects of data management) is. Means of ensuring data integrity.

Data Integration 101

Data Integration Testing Data Quality Data-driven 101

Jen Stirrup

AUGUST 25, 2020

Just because technology is easy to use, it does not follow that the data is easy to understand. Don’t be fooled by easy-to-use technology; data can still be hard. Logic Apps are hosted in Azure and have a code view rather than a ‘business user coder’ view. Compliant or Complaint?

Advertising 68

Advertising Business Intelligence Data Processing Data Architecture 68

Cloudera

JULY 19, 2023

Cisco has multiple reference architectures for running Ozone. Relevance of Operations per Second to Scale Ozone Manager hosts the metadata for the Objects stored within Ozone and consists of a cluster of Ozone Manager instances replicated via Ratis (a raft implementation ).

Management 70

Management Metadata Metrics Optimization 70

AWS Big Data

JANUARY 22, 2024

Cost and resource efficiency – This is an area where Acast observed a reduction in data duplication, and therefore cost reduction (in some accounts, removing the copy of data 100%), by reading data across accounts while enabling scaling. In this approach, teams responsible for generating data are referred to as producers.

Data-driven 99

Data-driven Advertising Metadata Data Architecture 99

AWS Big Data

JULY 16, 2024

When data is moved to the Infrequent Access tier, costs are reduced by up to 40%. Similarly, when data is moved to the Archive Instant Access tier, storage costs are reduced by up to 68%. Refer to Amazon S3 pricing for current pricing, as well as for information by region.

Insurance 123

Insurance Management Cost-Benefit Optimization 123

AWS Big Data

MARCH 27, 2023

An essential capability needed in such a data lake architecture is the ability to continuously understand changes in the data lakes in various other domains and make those available to data consumers. The data mesh producer account hosts the encrypted S3 bucket, which is shared with the central governance account.

Data Lake 92

Data Lake Data-driven Management Data Architecture 92

AWS Big Data

MAY 30, 2024

Overall, the current architecture didn’t support workload prioritization, therefore a physical model of resources was reserved for this reason. The system had an integration with legacy backend services that were all hosted on premises. Solution overview Amazon Redshift is an industry-leading cloud data warehouse.

Data Warehouse 99

Data Warehouse Data Lake Cost-Benefit Structured Data 99

AWS Big Data

AUGUST 1, 2024

Success criteria alignment by all stakeholders (producers, consumers, operators, auditors) is key for successful transition to a new Amazon Redshift modern data architecture. The success criteria are the key performance indicators (KPIs) for each component of the data workflow.

Data Warehouse 99

Data Warehouse KPI Optimization Cost-Benefit 99

AWS Big Data

MARCH 3, 2023

The Delta tables created by the EMR Serverless application are exposed through the AWS Glue Data Catalog and can be queried through Amazon Athena. Solution overview The following diagram shows the overall architecture of the solution that we implement in this post. Let’s refer to this S3 bucket as the raw layer.

Data Lake 90

Data Lake Dashboards Metrics Metadata 90

Alation

FEBRUARY 20, 2020

Alation Connect synchronizes metadata, sample data, and query logs into the Alation Data Catalog. This rich usage context is what makes our Data Catalog a powerful point of reference for data consumers and data stewards. In the release of Alation 4.0, In the release of Alation 4.0,

Metadata 52

Metadata Enterprise Data Processing Data Architecture 52

erwin

APRIL 16, 2024

The data mesh framework In the dynamic landscape of data management, the search for agility, scalability, and efficiency has led organizations to explore new, innovative approaches. One such innovation gaining traction is the data mesh framework. This empowers individual teams to own and manage their data.

Metadata 52

Metadata Data Quality Data Governance Modeling 52

BizAcuity

MAY 24, 2022

IaaS provides a platform for compute, data storage and networking capabilities. IaaS is mainly used for developing softwares (testing and development, batch processing), hosting web applications and data analysis. Companies develop data ecosystems on the cloud, the data architecture is now independent of on-premise systems.

Data-driven 52

Data-driven Cost-Benefit Digital Transformation Strategy 52

AWS Big Data

APRIL 28, 2023

When building a scalable data architecture on AWS, giving autonomy and ownership to the data domains are crucial for the success of the platform. Solution overview In the first post of this series, we explained how Novo Nordisk and AWS Professional Services built a modern data architecture based on data mesh tenets.

Data Governance 72

Data Governance Management Data-driven Analytics 72

AWS Big Data

MARCH 26, 2024

It is prudent to consolidate this data into a single customer view, serving as a primary reference for downstream applications, ranging from ecommerce platforms to CRM systems. This consolidated view acts as a liaison between the data platform and customer-centric applications.

Data Strategy 78

Data Strategy Strategy Data Warehouse Prescriptive Analytics 78

AWS Big Data

JUNE 7, 2023

Metadata exporter This section provides details on the AWS Glue job that exports the AWS Glue Data Catalog into an S3 location. The source code for the application is hosted the AWS Glue GitHub. He advises clients on architecting and adopting Data Architectures that best serve their Data Analytics and Machine Learning needs.

Metadata 59

Metadata Data Lake Machine Learning Big Data 59

Andrew White

JANUARY 9, 2022

On Thursday January 6th I hosted Gartner’s 2022 Leadership Vision for Data and Analytics webinar. Most of D&A concerns and activities are done within EA in the Info/Data architecture domain/phases. Here is a suggested note: Use Gartner’s Reference Model to Deliver Intelligent Composable Business Applications.

Analytics 64

Analytics Measurement Data-driven Modeling 64

Jet Global

MAY 1, 2023

that gathers data from many sources. Data Environment First off, the solutions you consider should be compatible with your current data architecture. We have outlined the requirements that most providers ask for: Data Sources Strategic Objective Use native connectivity optimized for the data source.

Analytics 52

Analytics Cost-Benefit Visualization Dashboards 52

AWS Big Data

JANUARY 21, 2025

The Lambda function will invoke the Amazon Titan Text Embeddings Model hosted in Amazon Bedrock , allowing for efficient and scalable embedding creation. This architecture simplifies various use cases, including recommendation engines, personalized chatbots, and fraud detection systems.

Data Processing 96

Data Processing Metrics Data-driven Publishing 96

AWS Big Data

FEBRUARY 28, 2025

In modern data architectures, the need to manage and query vast datasets efficiently, consistently, and accurately is paramount. For organizations that deal with big data processing, managing metadata becomes a critical concern. Any reference to HMS refers to a Standalone Hive Metastore.

Metadata 61

Metadata Data Lake Data Processing Data Architecture 61

AWS Big Data

OCTOBER 30, 2024

This is the final part of a three-part series where we show how to build a data lake on AWS using a modern data architecture. This post shows how to process data with Amazon Redshift Spectrum and create the gold (consumption) layer. His focus areas are MLOps, feature stores, data lakes, model hosting, and generative AI.

Data Lake 72

Data Lake Machine Learning Data Architecture Data-driven 72