Book Image

Architecting Data-Intensive Applications

By : Anuj Kumar
Book Image

Architecting Data-Intensive Applications

By: Anuj Kumar

Overview of this book

<p>Are you an architect or a developer who looks at your own applications gingerly while browsing through Facebook and applauding it silently for its data-intensive, yet ?uent and efficient, behaviour? This book is your gateway to build smart data-intensive systems by incorporating the core data-intensive architectural principles, patterns, and techniques directly into your application architecture.</p> <p>This book starts by taking you through the primary design challenges involved with architecting data-intensive applications. You will learn how to implement data curation and data dissemination, depending on the volume of your data. You will then implement your application architecture one step at a time. You will get to grips with implementing the correct message delivery protocols and creating a data layer that doesn’t fail when running high traffic. This book will show you how you can divide your application into layers, each of which adheres to the single responsibility principle. By the end of this book, you will learn to streamline your thoughts and make the right choice in terms of technologies and architectural principles based on the problem at hand.</p>
Table of Contents (18 chapters)
Title Page
Packt Upsell
Contributors
Preface
Index

HDFS file formats


There are multiple formats in which you can store a file in HDFS. For example, HDFS can be used to store images in GIF format, text data in plain-text CSV format, or spreadsheets in XLS format. This is similar to how you store files on your computer's local filesystem. A storage format gives indications to the underlying storage system as to how the file should be stored. This is usually indicated by the extension of the file.

Although one may not realize, choosing the right file format for your data is important to get the desirable performance from your big data storage. [5]—https://community.hitachivantara.com/community/products-and-solutions/pentaho/blog/2017/11/07/hadoop-file-formats-its-not-just-csv-anymore.

Some of us may confuse file formatting with flexible formatting. You may ask that the basic idea of Hadoop was to be able to store format-free data, also called unstructured data. Then, what is the need to have a file format in the first place? The question is valid...