Bad Data Handbook. Cleaning Up The Data So You Can Get Back To Work - Helion

ebook

Autor: Q. Ethan McCallum
ISBN: 978-14-493-2497-1
stron: 264, Format: ebook
Data wydania: 2012-11-07
Księgarnia: Helion

Cena książki: 126,65 zł (poprzednio: 147,27 zł)
Oszczędzasz: 14% (-20,62 zł)

Osoby, które kupiły tę książkę, wybierały także »

Tagi: Bazy danych

What is bad data? Some people consider it a technical phenomenon, like missing values or malformed records, but bad data includes a lot more. In this handbook, data expert Q. Ethan McCallum has gathered 19 colleagues from every corner of the data arena to reveal how they’ve recovered from nasty data problems.

From cranky storage to poor representation to misguided policy, there are many paths to bad data. Bottom line? Bad data is data that gets in the way. This book explains effective ways to get around it.

Among the many topics covered, you’ll discover how to:

Test drive your data to see if it’s ready for analysis
Work spreadsheet data into a usable form
Handle encoding problems that lurk in text data
Develop a successful web-scraping effort
Use NLP tools to reveal the real sentiment of online reviews
Address cloud computing issues that can impact your analysis effort
Avoid policies that create data analysis roadblocks
Take a systematic approach to data quality analysis

Osoby które kupowały "Bad Data Handbook. Cleaning Up The Data So You Can Get Back To Work", wybierały także:

Oracle Database 12c. Programowanie w języku PL/SQL 149,00 zł, (74,50 zł -50%)
Bazy danych. Podstawy projektowania i języka SQL 34,89 zł, (26,17 zł -25%)
Head First PHP & MySQL. Edycja polska 99,00 zł, (74,25 zł -25%)
MySQL. Mechanizmy wewnętrzne bazy danych 37,00 zł, (27,75 zł -25%)
Metody i techniki odkrywania wiedzy. Narzędzia CAQDAS w procesie analizy danych jakościowych 28,99 zł, (22,90 zł -21%)

Spis treści

Bad Data Handbook. Cleaning Up The Data So You Can Get Back To Work eBook -- spis treści

Bad Data Handbook
About the Authors
Preface
- Conventions Used in This Book
- Using Code Examples
- Safari Books Online
- How to Contact Us
- Acknowledgments
1. Setting the Pace: What Is Bad Data?
2. Is It Just Me, or Does This Data Smell Funny?
- Understand the Data Structure
- Field Validation
- Value Validation
- Physical Interpretation of Simple Statistics
- Visualization
- Keyword PPC Example
- Search Referral Example
- Recommendation Analysis
- Time Series Data
- Conclusion
3. Data Intended for Human Consumption, Not Machine Consumption
- The Data
- The Problem: Data Formatted for Human Consumption
  - The Arrangement of Data
  - Data Spread Across Multiple Files
- The Solution: Writing Code
  - Reading Data from an Awkward Format
  - Reading Data Spread Across Several Files
- Postscript
- Other Formats
- Summary
4. Bad Data Lurking in Plain Text
- Which Plain Text Encoding?
- Guessing Text Encoding
- Normalizing Text
- Problem: Application-Specific Characters Leaking into Plain Text
- Text Processing with Python
- Exercises
5. (Re)Organizing the Webs Data
- Can You Get That?
- General Workflow Example
  - robots.txt
  - Identifying the Data Organization Pattern
  - Store Offline Version for Parsing
  - Scrape the Information Off the Page
- The Real Difficulties
  - Download the Raw Content If Possible
  - Forms, Dialog Boxes, and New Windows
  - Flash
- The Dark Side
- Conclusion
6. Detecting Liars and the Confused in Contradictory Online Reviews
- Weotta
- Getting Reviews
- Sentiment Classification
- Polarized Language
- Corpus Creation
- Training a Classifier
- Validating the Classifier
- Designing with Data
- Lessons Learned
- Summary
- Resources
7. Will the Bad Data Please Stand Up?
- Example 1: Defect Reduction in Manufacturing
- Example 2: Whos Calling?
- Example 3: When Typical Does Not Mean Average
- Lessons Learned
- Will This Be on the Test?
8. Blood, Sweat, and Urine
- A Very Nerdy Body Swap Comedy
- How Chemists Make Up Numbers
- All Your Database Are Belong to Us
- Check, Please
- Live Fast, Die Young, and Leave a Good-Looking Corpse Code Repository
- Rehab for Chemists (and Other Spreadsheet Abusers)
- tl;dr
9. When Data and Reality Dont Match
- Whose Ticker Is It Anyway?
- Splits, Dividends, and Rescaling
- Bad Reality
- Conclusion
10. Subtle Sources of Bias and Error
- Imputation Bias: General Issues
- Reporting Errors: General Issues
- Other Sources of Bias
  - Topcoding/Bottomcoding
  - Seam Bias
  - Proxy Reporting
  - Sample Selection
- Conclusions
- References
11. Dont Let the Perfect Be the Enemy of the Good: Is Bad Data Really Bad?
- But First, Lets Reflect on Graduate School
- Moving On to the Professional World
- Moving into Government Work
- Government Data Is Very Real
- Service Call Data as an Applied Example
- Moving Forward
- Lessons Learned and Looking Ahead
12. When Databases Attack: A Guide for When to Stick to Files
- History
  - Building My Toolset
  - The Roadblock: My Datastore
- Consider Files as Your Datastore
  - Files Are Simple!
  - Files Work with Everything
  - Files Can Contain Any Data Type
  - Data Corruption Is Local
  - They Have Great Tooling
  - Theres No Install Tax
- File Concepts
  - Encoding
  - Text Files
  - Binary Data
  - Memory-Mapped Files
  - File Formats
  - Delimiters
- A Web Framework Backed by Files
  - Motivation
  - Implementation
- Reflections
13. Crouching Table, Hidden Network
- A Relational Cost Allocations Model
- The Delicate Sound of a Combinatorial Explosion
- The Hidden Network Emerges
- Storing the Graph
- Navigating the Graph with Gremlin
- Finding Value in Network Properties
- Think in Terms of Multiple Data Models and Use the Right Tool for the Job
- Acknowledgments
14. Myths of Cloud Computing
- Introduction to the Cloud
- What Is The Cloud?
- The Cloud and Big Data
- Introducing Fred
- At First Everything Is Great
- They Put 100% of Their Infrastructure in the Cloud
- As Things Grow, They Scale Easily at First
- Then Things Start Having Trouble
- They Need to Improve Performance
- Higher IO Becomes Critical
- A Major Regional Outage Causes Massive Downtime
- Higher IO Comes with a Cost
- Data Sizes Increase
- Geo Redundancy Becomes a Priority
- Horizontal Scale Isnt as Easy as They Hoped
- Costs Increase Dramatically
- Freds Follies
- Myth 1: Cloud Is a Great Solution for All Infrastructure Components
  - How This Myth Relates to Freds Story
- Myth 2: Cloud Will Save Us Money
  - How This Myth Relates to Freds Story
- Myth 3: Cloud IO Performance Can Be Improved to Acceptable Levels Through Software RAID
  - How This Myth Relates to Freds Story
- Myth 4: Cloud Computing Makes Horizontal Scaling Easy
  - How This Myth Relates to Freds Story
- Conclusion and Recommendations
15. The Dark Side of Data Science
- Avoid These Pitfalls
- Know Nothing About Thy Data
  - Be Inconsistent in Cleaning and Organizing the Data
  - Assume Data Is Correct and Complete
  - Spillover of Time-Bound Data
- Thou Shalt Provide Your Data Scientists with a Single Tool for All Tasks
  - Using a Production Environment for Ad-Hoc Analysis
  - The Ideal Data Science Environment
- Thou Shalt Analyze for Analysis Sake Only
- Thou Shalt Compartmentalize Learnings
- Thou Shalt Expect Omnipotence from Data Scientists
  - Where Do Data Scientists Live Within the Organization?
- Final Thoughts
16. How to Feed and Care for Your Machine-Learning Experts
- Define the Problem
- Fake It Before You Make It
- Create a Training Set
- Pick the Features
- Encode the Data
- Split Into Training, Test, and Solution Sets
- Describe the Problem
- Respond to Questions
- Integrate the Solutions
- Conclusion
17. Data Traceability
- Why?
- Personal Experience
  - Snapshotting
  - Saving the Source
  - Weighting Sources
  - Backing Out Data
  - Separating Phases (and Keeping them Pure)
  - Identifying the Root Cause
  - Finding Areas for Improvement
- Immutability: Borrowing an Idea from Functional Programming
- An Example
  - Crawlers
  - Change
  - Clustering
  - Popularity
- Conclusion
18. Social Media: Erasable Ink?
- Social Media: Whose Data Is This Anyway?
- Control
- Commercial Resyndication
- Expectations Around Communication and Expression
- Technical Implications of New End User Expectations
- What Does the Industry Do?
  - Validation API
  - Update Notification API
- What Should End Users Do?
- How Do We Work Together?
19. Data Quality Analysis Demystified: Knowing When Your Data Is Good Enough
- Framework Introduction: The Four Cs of Data Quality Analysis
- Complete
- Coherent
- Correct
- aCcountable
- Conclusion
Index
About the Author
Colophon
Copyright