2008, Data Collection and Metadata

Preprocess and fine-tune LLMs quickly and cost-effectively using Amazon EMR Serverless and Amazon SageMaker

AWS Big Data

FEBRUARY 1, 2024

The Common Crawl corpus contains petabytes of data, regularly collected since 2008, and contains raw webpage data, metadata extracts, and text extracts. In addition to determining which dataset should be used, cleansing and processing the data to the fine-tuning’s specific need is required.

Metadata

Metadata Modeling Data Processing Unstructured Data

Data Science, Past & Future

Domino Data Lab

JULY 22, 2019

In 2008, there was a JIRA ticket, and as an engineering manager, I wrote a $3,000 check to a young engineer in London named Tom White who pushed a fix. We had Julia Lane talking about Coleridge Initiative and the work on Project Jupyter to support metadata and data governance and lineage. It was called Hadoop.

Data Science

Data Science Machine Learning Data Governance Modeling

Data Leaders Brief

Preprocess and fine-tune LLMs quickly and cost-effectively using Amazon EMR Serverless and Amazon SageMaker

Data Science, Past & Future

Webinars

Stay Connected