Hadoop Application Architectures - Helion

ebook

Autor: Mark Grover, Ted Malaska, Jonathan Seidman
ISBN: 978-14-919-0005-5
stron: 400, Format: ebook
Data wydania: 2015-06-30
Księgarnia: Helion

Cena książki: 152,15 zł (poprzednio: 176,92 zł)
Oszczędzasz: 14% (-24,77 zł)

Osoby, które kupiły tę książkę, wybierały także »

Get expert guidance on architecting end-to-end data management solutions with Apache Hadoop. While many sources explain how to use various components in the Hadoop ecosystem, this practical book takes you through architectural considerations necessary to tie those components together into a complete tailored application, based on your particular use case.

To reinforce those lessons, the book’s second section provides detailed examples of architectures used in some of the most commonly found Hadoop applications. Whether you’re designing a new Hadoop application, or planning to integrate Hadoop into your existing data infrastructure, Hadoop Application Architectures will skillfully guide you through the process.

This book covers:

Factors to consider when using Hadoop to store and model data
Best practices for moving data in and out of the system
Data processing frameworks, including MapReduce, Spark, and Hive
Common Hadoop processing patterns, such as removing duplicate records and using windowing analytics
Giraph, GraphX, and other tools for large graph processing on Hadoop
Using workflow orchestration and scheduling tools such as Apache Oozie
Near-real-time stream processing with Apache Storm, Apache Spark Streaming, and Apache Flume
Architecture examples for clickstream analysis, fraud detection, and data warehousing

Osoby które kupowały "Hadoop Application Architectures", wybierały także:

Windows Media Center. Domowe centrum rozrywki 66,67 zł, (8,00 zł -88%)
Ruby on Rails. Ćwiczenia 18,75 zł, (3,00 zł -84%)
DevOps w praktyce. Kurs video. Jenkins, Ansible, Terraform i Docker 190,00 zł, (39,90 zł -79%)
Przywództwo w świecie VUCA. Jak być skutecznym liderem w niepewnym środowisku 58,64 zł, (12,90 zł -78%)
Scrum. O zwinnym zarządzaniu projektami. Wydanie II rozszerzone 58,64 zł, (12,90 zł -78%)

Spis treści

Hadoop Application Architectures eBook -- spis treści

Foreword
Preface
- A Note About the Code Examples
- Who Should Read This Book
- Why We Wrote This Book
- Navigating This Book
- Conventions Used in This Book
- Using Code Examples
- Safari Books Online
- How to Contact Us
- Acknowledgments
  - Mark Grovers Acknowledgements
  - Ted Malaskas Acknowledgements
  - Jonathan Seidmans Acknowledgements
  - Gwen Shapiras Acknowledgements
I. Architectural Considerations for Hadoop Applications
1. Data Modeling in Hadoop
- Data Storage Options
  - Standard File Formats
    - Text data
    - Structured text data
    - Binary data
  - Hadoop File Types
    - File-based data structures
  - Serialization Formats
    - Thrift
    - Protocol Buffers
    - Avro
  - Columnar Formats
    - RCFile
    - ORC
    - Parquet
      - Avro and Parquet
  - Compression
    - Snappy
    - LZO
    - Gzip
    - bzip2
    - Compression recommendations
- HDFS Schema Design
  - Location of HDFS Files
  - Advanced HDFS Schema Design
    - Partitioning
    - Bucketing
    - Denormalizing
  - HDFS Schema Design Summary
- HBase Schema Design
  - Row Key
    - Record retrieval
    - Distribution
    - Block cache
    - Ability to scan
    - Size
    - Readability
    - Uniqueness
  - Timestamp
  - Hops
  - Tables and Regions
    - Put performance
    - Compaction time
  - Using Columns
  - Using Column Families
  - Time-to-Live
- Managing Metadata
  - What Is Metadata?
  - Why Care About Metadata?
  - Where to Store Metadata?
  - Examples of Managing Metadata
  - Limitations of the Hive Metastore and HCatalog
  - Other Ways of Storing Metadata
    - Embedding metadata in file paths and names
    - Storing the metadata in HDFS
- Conclusion
2. Data Movement
- Data Ingestion Considerations
  - Timeliness of Data Ingestion
  - Incremental Updates
  - Access Patterns
  - Original Source System and Data Structure
    - Read speed of the devices on source systems
    - Original file type
    - Compression
    - Relational database management systems
    - Streaming data
    - Logfiles
  - Transformations
    - Interceptors
    - Selectors
  - Network Bottlenecks
  - Network Security
  - Push or Pull
    - Sqoop
    - Flume
  - Failure Handling
  - Level of Complexity
- Data Ingestion Options
  - File Transfers
    - HDFS client commands
    - Mountable HDFS
  - Considerations for File Transfers versus Other Ingest Methods
  - Sqoop: Batch Transfer Between Hadoop and Relational Databases
    - Choosing a split-by column
    - Using database-specific connectors whenever available
    - Using the Goldilocks method of Sqoop performance tuning
    - Loading many tables in parallel with fair scheduler throttling
    - Diagnosing bottlenecks
    - Keeping Hadoop updated
  - Flume: Event-Based Data Collection and Processing
    - Flume architecture
    - Flume patterns
    - File formats
    - Recommendations
      - Flume sources
      - Flume sinks
      - Flume interceptors
      - Flume memory channels
      - Flume file channels
      - Sizing Channels
    - Finding Flume bottlenecks
  - Kafka
    - Kafka fault tolerance
    - Kafka and Hadoop
- Data Extraction
- Conclusion
3. Processing Data in Hadoop
- MapReduce
  - MapReduce Overview
    - Map phase
      - InputFormat
      - RecordReader
      - Mapper.setup()
      - Mapper.map
      - Partitioner
      - Mapper.cleanup()
      - Combiner
    - Reducer
      - Shuffle
      - Reducer.setup()
      - Reducer.reduce()
      - Reducer.cleanup()
      - OutputFormat
  - Example for MapReduce
  - When to Use MapReduce
- Spark
  - Spark Overview
    - DAG Model
  - Overview of Spark Components
  - Basic Spark Concepts
    - Resilient Distributed Datasets
    - Shared variables
    - SparkContext
    - Transformations
    - Action
  - Benefits of Using Spark
    - Simplicity
    - Versatility
    - Reduced disk I/O
    - Storage
    - Multilanguage
    - Resource manager independence
    - Interactive shell (REPL)
  - Spark Example
  - When to Use Spark
- Abstractions
  - Pig
  - Pig Example
  - When to Use Pig
- Crunch
  - Crunch Example
  - When to Use Crunch
- Cascading
  - Cascading Example
  - When to Use Cascading
- Hive
  - Hive Overview
  - Example of Hive Code
  - When to Use Hive
- Impala
  - Impala Overview
  - Speed-Oriented Design
    - Efficient use of memory
    - Long running daemons
    - Efficient execution engine
    - Use of LLVM
  - Impala Example
  - When to Use Impala
- Conclusion
4. Common Hadoop Processing Patterns
- Pattern: Removing Duplicate Records by Primary Key
  - Data Generation for Deduplication Example
  - Code Example: Spark Deduplication in Scala
  - Code Example: Deduplication in SQL
- Pattern: Windowing Analysis
  - Data Generation for Windowing Analysis Example
  - Code Example: Peaks and Valleys in Spark
  - Code Example: Peaks and Valleys in SQL
- Pattern: Time Series Modifications
  - Use HBase and Versioning
  - Use HBase with a RowKey of RecordKey and StartTime
  - Use HDFS and Rewrite the Whole Table
  - Use Partitions on HDFS for Current and Historical Records
  - Data Generation for Time Series Example
  - Code Example: Time Series in Spark
  - Code Example: Time Series in SQL
- Conclusion
5. Graph Processing on Hadoop
- What Is a Graph?
- What Is Graph Processing?
- How Do You Process a Graph in a Distributed System?
  - The Bulk Synchronous Parallel Model
  - BSP by Example
- Giraph
  - Read and Partition the Data
  - Batch Process the Graph with BSP
  - Write the Graph Back to Disk
  - Putting It All Together
  - When Should You Use Giraph?
- GraphX
  - Just Another RDD
  - GraphX Pregel Interface
  - vprog()
  - sendMessage()
  - mergeMessage()
- Which Tool to Use?
- Conclusion
6. Orchestration
- Why We Need Workflow Orchestration
- The Limits of Scripting
- The Enterprise Job Scheduler and Hadoop
- Orchestration Frameworks in the Hadoop Ecosystem
- Oozie Terminology
- Oozie Overview
- Oozie Workflow
- Workflow Patterns
  - Point-to-Point Workflow
  - Fan-Out Workflow
  - Capture-and-Decide Workflow
- Parameterizing Workflows
- Classpath Definition
- Scheduling Patterns
  - Frequency Scheduling
  - Time and Data Triggers
- Executing Workflows
- Conclusion
7. Near-Real-Time Processing with Hadoop
- Stream Processing
- Apache Storm
  - Storm High-Level Architecture
  - Storm Topologies
  - Tuples and Streams
  - Spouts and Bolts
  - Stream Groupings
  - Reliability of Storm Applications
  - Exactly-Once Processing
  - Fault Tolerance
  - Integrating Storm with HDFS
  - Integrating Storm with HBase
  - Storm Example: Simple Moving Average
  - Evaluating Storm
    - Support for aggregation and windowing
    - Enrichment and alerting
    - Lamdba Architecture
- Trident
  - Trident Example: Simple Moving Average
  - Evaluating Trident
    - Support for counting and windowing
    - Enrichment and alerting
    - Lamdba Architecture
- Spark Streaming
  - Overview of Spark Streaming
  - Spark Streaming Example: Simple Count
  - Spark Streaming Example: Multiple Inputs
  - Spark Streaming Example: Maintaining State
  - Spark Streaming Example: Windowing
  - Spark Streaming Example: Streaming versus ETL Code
  - Evaluating Spark Streaming
    - Support for counting and windowing
    - Enrichment and alerting
    - Lambda Architecture
- Flume Interceptors
- Which Tool to Use?
  - Low-Latency Enrichment, Validation, Alerting, and Ingestion
    - Solution One: Flume
    - Solution Two: Kafka and Storm
  - NRT Counting, Rolling Averages, and Iterative Processing
  - Complex Data Pipelines
- Conclusion
II. Case Studies
8. Clickstream Analysis
- Defining the Use Case
- Using Hadoop for Clickstream Analysis
- Design Overview
- Storage
- Ingestion
  - The Client Tier
  - The Collector Tier
- Processing
  - Data Deduplication
    - Deduplication in Hive
    - Deduplication in Pig
  - Sessionization
    - Sessionization in Spark
    - Sessionization in MapReduce
    - Sessionization in Pig
    - Sessionization in Hive
- Analyzing
- Orchestration
- Conclusion
9. Fraud Detection
- Continuous Improvement
- Taking Action
- Architectural Requirements of Fraud Detection Systems
- Introducing Our Use Case
- High-Level Design
- Client Architecture
- Profile Storage and Retrieval
  - Caching
    - Distributed memory caching
    - HBase with BlockCache
  - HBase Data Definition
    - Columns (combined or atomic)
    - Event counting using HBase increment or put
    - Event history using HBase put
  - Delivering Transaction Status: Approved or Denied?
- Ingest
  - Path Between the Client and Flume
    - Client push
    - Logfile pull
    - Message queue or Kafka in the middle
- Near-Real-Time and Exploratory Analytics
- Near-Real-Time Processing
- Exploratory Analytics
- What About Other Architectures?
  - Flume Interceptors
  - Kafka to Storm or Spark Streaming
  - External Business Rules Engine
- Conclusion
10. Data Warehouse
- Using Hadoop for Data Warehousing
- Defining the Use Case
- OLTP Schema
- Data Warehouse: Introduction and Terminology
- Data Warehousing with Hadoop
- High-Level Design
  - Data Modeling and Storage
    - Choosing a storage engine
    - Denormalizing
    - Tracking updates in Hadoop
    - Selecting storage format and compression
    - Partitioning
  - Ingestion
  - Data Processing and Access
    - Partitioning
    - Merge/update
  - Aggregations
  - Data Export
  - Orchestration
- Conclusion
A. Joins in Impala
- Broadcast Joins
- Partitioned Hash Join
Index