DevOps and big data are most sought-after technology trends bringing multifold benefits to business, industries, and R and D centers. Their widespread adoption is associated with the popularity and acceptance of Open source tools along with other technology advancements, such as cloud computing, containers, and the Internet of Things.
In this book, we will discuss the business drivers and market trends contributing to the evolution of technology. We will explore the key technological concepts, such as NoSQL databases, microservices, data science, and data visualization. We will build big data enterprise applications, using core architectural principles, based on the Spark in-memory engine. We will discuss DevOps practices and frameworks, and their application with popular tools, to prepare for continuous integration, continuous delivery, and continuous deployment. We will dive deeper into the popular open source tools, such as Chef, Ansible, Jenkins, Git, and Stack, which assist in the automation of the development to deployment cycle.
At every opportunity, this book introduces different flavors of popular vendor offerings with the key components and differences. This will provide the reader with information to aid their decision making. The technology choices discussed are for Hadoop platforms, cloud offerings, container and orchestration services, Internet of Things platforms, and NoSQL databases. The choice of popular tools for continuous integration, delivery, deployment, and monitoring is also thoroughly discussed. There are topics on capacity planning for Hadoop cluster, and security models for data and cloud based, discussed to provide reader with comprehensive knowledge on these subjects.
The migration strategy to DevOps for different systems, including big data systems, cloud platforms, microservices, data sciences, security authentications, and Internet of Things platforms, is discussed. ERP systems adoption to DevOps is also shared with detailed testing strategy.
Every effort is made to benefit the reader to learn the wide range of technology, tools and platform evolutions, frameworks, and methodologies from practical usage perspective. I feel confident that reader will find it very interesting and useful from multiple perspectives.
Chapter 1, Introduction to DevOps, focuses on business trends, drivers, market propellers of evolution, and adoption of DevOps. The key technological concepts are introduced for big data, cloud, data sciences, and in-memory computing. DevOps application scenarios and the benefits of adoption for an organization are presented.
Chapter 2, Introduction to Big Data and Data Sciences, starts with the introduction of big data concepts, discuss the features of in-memory databases for real-time systems, and presents different NoSQL database types for different uses as well as concepts relating to data visualization, data science, and data scientist roles.
Chapter 3, DevOps Framework, discusses for source code management, build, repository, release managements, and test automation were discussed. Continuous integration, delivery, deployment were covered along with infrastructure automation for configuration (Infrastructure as Code), application monitoring, and so on. DevOps maturity models, progress frameworks, and the Agile methodology are also discussed.
Chapter 4, Big Data Hadoop Ecosystems, focuses on big data clusters concepts such as data lake, processing frameworks, node types, roles, capacity planning, security, and authentications. Different flavors of Hadoop frameworks (Cloudera, Hortonworks, MapR, Pivotal, IBM Open platform, Amazon Elastic Mapreduce, and Micosoft Azure HDInsight) were discussed.
Chapter 5, Cloud Computing, discusses technology concepts for cloud, such as virtualization and hypervisor are discussed along with WEB protocols such as REST APIs. Cloud architecture layers and components are discussed, and different cloud models such as private, public, and hybrid cloud are presented. Offerings such as IaaS, PaaS, SaaS, and their variants are detailed. Features included in cloud vendor-based offerings, such as security, encryption, and business continuity models, are discussed.
Chapter 6, Building Big Data Applications, discusses life cycle stages, such as data discovery, quality, ingestion, analytics, visualization, governance, as well as their best practices. Data ingestion models for batch and real-time, streaming data, and respective tools are also discussed. Data storage concepts such as data lake, data analytics, Spark in-memory engine are also discussed. Core architecture components, such as workflow, security, data catalog, client services, and APIs. Data science algorithms; their business usage/application, different models, such as decision trees, clusters, and supervised and unsupervised learning, are amply covered.
Chapter 7, DevOps - Continuous Integration and Delivery, discusses the CI/CD methodology with open source popular tools such as Git, Maven, Gerrit, Nexus, Selenium, and Jenkins. Different tools and plugins compatible with Jenkins were discussed.
Chapter 8, DevOps Continuous Deployment, shows the popular continuous deployment tools Ansible and Chef; their advanced features such as Habitat, Automate, and Compliance for security are discussed. Core components, architecture and terminology (Chef, cookbooks knife, Playbooks, towers) are elaborated. Continuous monitoring tools Splunk and Nagios are covered in depth.
Chapter 9, Containers, IoT, and Microservices, covers virtualization, containers, orchestration, Internet of Things (IoT), microservices, hypervisors, and emulation. Containers offered by multiple vendors (AWS, Google, Pivotal, and Azure), including open source technologies such as J2EE and Docker, are covered at the component level. Orchestration tools (Mesos, Kubernetics, and Docker Swarm) are discussed. Mutiple IoT platforms, from both commercial vendors (ThingWorx, Electric Imp, and Predix) and open source (Eclipse and Contiki), are presented. Microservice design patterns, architecture models, client-side, server-side service models, and best practices are detailed.
Chapter 10, DevOps for Digital Transformation, covers the details of the digital journey for enterprise systems such as big data, cloud, data science, Internet of Things, and security authentication systems for progressive adoption with challenges and best practices. Mircoservice models are elaborated with different models and patterns.
Appendix A, DevOps Adoption by ERP Systems, discusses ERP systems like SAP have adopted DevOps process along with open source tools like Jenkins and Git to improve efficiency.
Appendix B, DevOps Periodic Table, depicts the entire list presented in the form of periodic table describing many open source DevOps tools for each stage of development, deployment, and monitoring.
Appendix C, Business Intelligence Trends, depicts shift in priorities, as we can see that data preparation and discovery are rated as high value added to master data management.
Appendix D, Testing Types and Levels, presents us end to end testing cycle at each of the layers DB, APP, and UI. Elaborate testing types are discussed.
Appendix E, Java Platform SE 8, shows Java enterprise edition layered architecture components are shared for reference.
Prerequisite knowledge for fundamentals of software development life cycle and systems. It will help readers connect with basic topics and appreciate the application of various technologies. For advanced concepts of cloud offerings, Hadoop clusters, big data applications, microservices, containers, security, and digital migration, familiarity with architecture is required.
If you are a big data architects, solutions provider, or any stakeholder working in big data environment and wants to implement the strategy of DevOps, then this book is for you.
In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning. Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: "For example--sqoop import -connect jdbc:mysql://:/ -username -password --table --target-dir"
A block of code is set as follows:
- hosts: webservers
serial: 6 # update 6 machines at a time
roles:
- common
- webapp
- hosts: content_serversAny command-line input or output is written as follows:
C:>Java -jar Jenkins.warNew terms and important words are shown in bold. Words that you see on the screen, for example, in menus or dialog boxes, appear in the text like this: "TheManage Jenkinsoption in the dashboard will provide various options to configure various parameters."
Feedback from our readers is always welcome. Let us know what you think about this book-what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of. To send us general feedback, simply email [email protected], and mention the book's title in the subject of your message. If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.
Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.
We also provide you with a PDF file that has color images of the screenshots/diagrams used in this book. The color images will help you better understand the changes in the output. You can download this file from https://www.packtpub.com/sites/default/files/downloads/HandsonDevOps_ColorImages.pdf.
Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books-maybe a mistake in the text or the code-we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website or added to any list of existing errata under the Errata section of that title. To view the previously submitted errata, go to https://www.packtpub.com/books/content/support and enter the name of the book in the search field. The required information will appear under the Errata section.
Piracy of copyrighted material on the internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works in any form on the internet, please provide us with the location address or website name immediately so that we can pursue a remedy. Please contact us at [email protected] with a link to the suspected pirated material. We appreciate your help in protecting our authors and our ability to bring you valuable content.
If you have a problem with any aspect of this book, you can contact us at [email protected], and we will do our best to address the problem.