Sign In Start Free Trial
Account

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Data Engineering Best Practices
  • Table Of Contents Toc
Data Engineering Best Practices

Data Engineering Best Practices

By : Richard J. Schiller, David Larochelle
5 (2)
close
close
Data Engineering Best Practices

Data Engineering Best Practices

5 (2)
By: Richard J. Schiller, David Larochelle

Overview of this book

Revolutionize your approach to data processing in the fast-paced business landscape with this essential guide to data engineering. Discover the power of scalable, efficient, and secure data solutions through expert guidance on data engineering principles and techniques. Written by two industry experts with over 60 years of combined experience, it offers deep insights into best practices, architecture, agile processes, and cloud-based pipelines. You’ll start by defining the challenges data engineers face and understand how this agile and future-proof comprehensive data solution architecture addresses them. As you explore the extensive toolkit, mastering the capabilities of various instruments, you’ll gain the knowledge needed for independent research. Covering everything you need, right from data engineering fundamentals, the guide uses real-world examples to illustrate potential solutions. It elevates your skills to architect scalable data systems, implement agile development processes, and design cloud-based data pipelines. The book further equips you with the knowledge to harness serverless computing and microservices to build resilient data applications. By the end, you'll be armed with the expertise to design and deliver high-performance data engineering solutions that are not only robust, efficient, and secure but also future-ready.
Table of Contents (21 chapters)
close
close

Data annotation

So, you want to know how you, as a data engineer, can help a data scientist? You can begin by curating your data into indices that are semantically aligned with the object in your knowledge base. These, in turn, are correctly modeled for your business domains with the current known truths that are relevant for your enterprise. We bet you thought we would say, just build a vector store and make it available to your LLM, using the cloud provider of choice’s tool. However, you will fail to have that model propagate to production, since it would suffer from many of the failings of untuned GenAI models (such as hallucinations, quality errors, and the inappropriate exposure of training source text in model output). If you did as some of the cloud providers propose and built your RAG with an embedding, without semantic structure, you would be creating several iterations in your factory. Your vision should be to implement a knowledge-aware semantic form for embeddings...

CONTINUE READING
83
Tech Concepts
36
Programming languages
73
Tech Tools
Icon Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.
Icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Icon 50+ new titles added per month and exclusive early access to books as they are being written.
Data Engineering Best Practices
notes
bookmark Notes and Bookmarks search Search in title playlist Add to playlist download Download options font-size Font size

Change the font size

margin-width Margin width

Change margin width

day-mode Day/Sepia/Night Modes

Change background colour

Close icon Search
Country selected

Close icon Your notes and bookmarks

Confirmation

Modal Close icon
claim successful

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY

Submit Your Feedback

Modal Close icon
Modal Close icon
Modal Close icon