目錄
1. Introduction 引言… ………………………………………………………………………………1
1.1 Towards Intelligent Machines 走向智能機器 ………………………………………………1
1.2 Well-Posed Machine Learning Problems 適定的機器學習問題 ……………………………5
1.3 Examples of Applications in Diverse Fields 不同領域的應用實例 …………………………7
1.4 Data Representation 數(shù)據(jù)表示 ………………………………………………………………12
1.4.1 Time Series Forecasting 時間序列預測 ………………………………………………15
1.4.2 Datasets for Toy (Unreastically Simple) and Realistic Problems
初級問題和現(xiàn)實問題數(shù)據(jù)集 …………………………………………………………17
1.5 Domain Knowledge for Productive use of Machine Learning
使機器學習有效應用的領域知識 ……………………………………………………………18
1.6 Diversity of Data: Structured/Unstructured 數(shù)據(jù)多樣性：結構化/非結構化 ……………20
1.7 Forms of Learning 學習形式 …………………………………………………………………21
1.7.1 Supervised/Directed Learning 監(jiān)督/指導學習 ……………………………………21
1.7.2 Unsupervised/Undirected Learning 非監(jiān)督/無指導學習 …………………………22
1.7.3 Reinforcement Learning 強化學習 …………………………………………………22
1.7.4 Learning Based on Natural Processes: Evolution, Swarming, and Immune Systems
基于自然處理的學習：進化、集群和免疫系統(tǒng)………………………………………23
1.8 Machine Learning and Data Mining 機器學習和數(shù)據(jù)挖掘 …………………………………25
1.9 Basic Linear Algebra in Machine Learning Techniques
機器學習技術中的基礎線性代數(shù) ……………………………………………………………26
1.10 Relevant Resources for Machine Learning 機器學習的相關資源 …………………………34
2. Supervised Learning: Rationale and Basics 監(jiān)督學習：基本原理和基礎… ………………36
2.1 Learning from Observations 從觀測中學習 …………………………………………………36
2.2 Bias and Variance 偏差和方差 ………………………………………………………………42
2.3 Why Learning Works: Computational Learning Theory
學習為什么有效：計算學習理論 ……………………………………………………………46
2.4 Occam’s Razor Principle and Overfitting Avoidance
奧卡姆剃刀原理和防止過擬合 ………………………………………………………………49
2.5 Heuristic Search in Inductive Learning 歸納學習中的啟發(fā)式搜索 …………………………51
2.5.1 Search through Hypothesis Space 假設空間搜索 ……………………………………52
2.5.2 Ensemble Learning 集成學習 ………………………………………………………53
2.5.3 Evaluation of a Learning System 學習系統(tǒng)的評價 …………………………………55
2.6 Estimating Generalization Errors 估計泛化誤差 ……………………………………………56
2.6.1 Holdout Method and Random Subsampling 留出法和隨機下采樣 …………………56
2.6.2 Cross-validation 交叉驗證 ……………………………………………………………57
2.6.3 Bootstrapping 自助法 …………………………………………………………………58
2.7 Metrics for Assessing Regression (Numeric Prediction) Accuracy
評價回歸（數(shù)值預測）精度的指標 …………………………………………………………59
2.7.1 Mean Square Error 均方誤差 …………………………………………………………60
2.7.2 Mean Absolute Error 平均絕對誤差 …………………………………………………60
2.8 Metrics for Assessing Classification (Pattern Recognition) Accuracy
評價分類（模式識別）精度的指標 …………………………………………………………61
2.8.1 Misclassification Error 誤分類誤差 …………………………………………………61
2.8.2 Confusion Matrix 混淆矩陣 …………………………………………………………62
2.8.3 Comparing Classifiers Based on ROC Curves 基于ROC曲線的分類器比較 ……66
2.9 An Overview of the Design Cycle and Issues in Machine Learning
機器學習中的設計周期和問題概述 …………………………………………………………68
3. Statistical Learning 統(tǒng)計學習… …………………………………………………………………73
3.1 Machine Learning and Inferential Statistical Analysis 機器學習與推斷統(tǒng)計分析 …………73
3.2 Descriptive Statistics in Learning Techniques 學習技術中的描述統(tǒng)計學 …………………74
3.2.1 Representing Uncertainties in Data: Probability Distributions
表示數(shù)據(jù)中的不確定性：概率分布 …………………………………………………75
3.2.2 Descriptive Measures of Probability Distributions 概率分布的描述方法 …………80
3.2.3 Descriptive Measures from Data Sample 數(shù)據(jù)樣本的描述方法 ……………………83
3.2.4 Normal Distributions 正態(tài)分布 ………………………………………………………84
3.2.5 Data Similarity 數(shù)據(jù)相似性 …………………………………………………………85
3.3 Bayesian Reasoning: A Probabilistic Approach to Inference
貝葉斯推理：推斷的概率方法 ………………………………………………………………87
3.3.1 Bayes Theorem 貝葉斯定理 …………………………………………………………88
3.3.2 Naive Bayes Classifier 樸素貝葉斯分類器 …………………………………………93
3.3.3 Bayesian Belief Networks 貝葉斯信念網絡 …………………………………………98
3.4 k-Nearest Neighbor (k-NN) Classifier k近鄰（k-NN）分類器 …………………………102
3.5 Discriminant Functions and Regression Functions 判別函數(shù)和回歸函數(shù) ………………106
3.5.1 Classification and Discriminant Functions 分類和判別函數(shù) ……………………107
3.5.2 Numeric Prediction and Regression Functions 數(shù)值預測和回歸函數(shù) ……………108
3.5.3 Practical Hypothesis Functions 實踐應用中的假設函數(shù) …………………………109
3.6 Linear Regression with Least Square Error Criterion
基于最小二乘誤差準則的線性回歸法 ……………………………………………………112
3.6.1 Minimal Sum-of-Error-Squares and the Pseudoinverse 最小誤差平方和與偽逆 113
3.6.2 Gradient Descent Optimization Schemes 梯度下降法優(yōu)化方案 …………………115
3.6.3 Least Mean Square (LMS) Algorithm 最小均方（LMS）算法 …………………115
3.7 Logistic Regression for Classification Tasks 分類任務的邏輯回歸法 ……………………116
3.8 Fisher’s Linear Discriminant and Thresholding for Classification
Fisher線性判別和分類閾值…………………………………………………………………120
3.8.1 Fisher’s Linear Discriminant Fisher線性判別式 …………………………………120
3.8.2 Thresholding 閾值 …………………………………………………………………125
3.9 Minimum Description Length Principle 最小描述長度原理 ……………………………126
3.9.1 Bayesian Perspective 貝葉斯角度 …………………………………………………127
3.9.2 Entropy and Information 熵和信息 ………………………………………………128
4. Learning With Support Vector Machines (SVM) 利用支持向量機（SVM）學習…………130
4.1 Introduction 簡介 …………………………………………………………………………130
4.2 Linear Discriminant Functions for Binary Classification
二元分類的線性判別函數(shù) …………………………………………………………………132
4.3 Perceptron Algorithm 感知機算法 …………………………………………………………136
4.4 Linear Maximal Margin Classifier for Linearly Separable Data
線性可分數(shù)據(jù)的最大邊緣線性分類器 ……………………………………………………141
4.5 Linear Soft Margin Classifier for Overlapping Classes
重疊類的軟邊緣線性分類器 ………………………………………………………………152
4.6 Kernel-Induced Feature Spaces 核函數(shù)引導的特征空間 …………………………………158
4.7 Nonlinear Classifier 非線性分類器 ………………………………………………………162
4.8 Regression by Support Vector Machines 支持向量機回歸 ………………………………167
4.8.1 Linear Regression 線性回歸 ………………………………………………………169
4.8.2 Nonlinear Regression 非線性回歸 …………………………………………………172
4.9 Decomposing Multiclass Classification Problem Into Binary Classification Tasks
將多類分類問題分解為二元分類任務 ……………………………………………………174
4.9.1 One-Against-All (OAA) 一對所有（OAA） ……………………………………175
4.9.2 One-Against-One (OAO) 一對一（OAO） ………………………………………176
4.10 Variants of Basic SVM Techniques 基礎支持向量機技術的變體 ………………………177
5. Learning With Neural Networks (NN) 利用神經網絡（NN）學習…………………………181
5.1 Towards Cognitive Machine 走向認知機器 ………………………………………………181
5.1.1 From Perceptrons to Deep Networks 從感知機到深度網絡 ………………………182
5.2 Neuron Models 神經元模型 ………………………………………………………………184
5.2.1 Biological Neuron 生物神經元 ……………………………………………………184
5.2.2 Artificial Neuron 人工神經元 ……………………………………………………186
5.2.3 Mathmatical Model 數(shù)學模型 ……………………………………………………190
5.3 Network Architectures 網絡架構 …………………………………………………………193
5.3.1 Feedforward Networks 前饋網絡 …………………………………………………194
5.3.2 Recurrent Networks 循環(huán)網絡 ……………………………………………………199
5.4 Perceptrons 感知機 …………………………………………………………………………200
5.4.1 Limitations of Perceptron Algorithm for Linear Classification Tasks
線性分類任務采用感知機算法的局限性 …………………………………………201
5.4.2 Linear Classification using Regression Techniques
使用回歸技術的線性分類 …………………………………………………………201
5.4.3 Standard Gradient Descent Optimization Scheme: Steepest Descent
標準梯度下降法優(yōu)化方案：最速下降法 …………………………………………203
5.5 Linear Neuron and the Widrow-Hoff Learning Rule
線性神經元與Widrow-Hoff學習規(guī)則 ……………………………………………………206
5.5.1 Stochastic Gradient Descent 隨機梯度下降法 ……………………………………207
5.6 The Error-Correction Delta Rule 糾錯Delta法則 …………………………………………208
5.6.1 Sigmoid Unit: Soft-Limiting Perceptron Sigmoid單元：軟限制感知機 …………211
5.7 Multi-Layer Perceptron (MLP) Networks and the Error-Backpropagation Algorithm
多層感知機網絡及誤差反向傳播算法 ……………………………………………………213
5.7.1 The Generalized Delta Rule 廣義Delta法則 ……………………………………216
5.7.2 Convergence and Local Minima 收斂性和局部極小值 …………………………226
5.7.3 Adding Momentum to Gradient Descent 為梯度下降法加入動量 ………………227
5.7.4 Heuristic Aspects of the Error-backpropagation Algorithm
誤差反向傳播算法的啟發(fā)性 ………………………………………………………228
5.8 Multi-Class Discrimination with MLP Networks 使用MLP網絡的多類判別 …………232
5.9 Radial Basis Functions (RBF) Networks 徑向基函數(shù)（RBF）網絡 ……………………235
5.9.1 Training the RBF Network 訓練RBF網絡 ………………………………………239
5.10 Genetic-Neural Systems 遺傳-神經系統(tǒng) …………………………………………………241
6. Fuzzy Inference Systems 模糊推理系統(tǒng)… ……………………………………………………245
6.1 Introduction 簡介 …………………………………………………………………………245
6.2 Cognitive Uncertainty and Fuzzy Rule-Base 認知不確定性和模糊規(guī)則庫 ………………248
6.3 Fuzzy Quantification of Knowledge 知識的模糊量化 ……………………………………253
6.3.1 Fuzzy Logic 模糊邏輯 ……………………………………………………………253
6.3.2 Fuzzy Sets 模糊集 …………………………………………………………………257
6.3.3 Fuzzy Set Operations 模糊集運算 …………………………………………………267
6.3.4 Fuzzy Relations 模糊關系 …………………………………………………………268
6.4 Fuzzy Rule-Base and Approximate Reasoning 模糊規(guī)則庫與近似推理 …………………277
6.4.1 Quantification of Rules via Fuzzy Relations 通過模糊關系量化規(guī)則 ……………281
6.4.2 Fuzzification of Input 輸入的模糊化 ………………………………………………283
6.4.3 Inference Mechanism 推理機制 ……………………………………………………284
6.4.4 Defuzzification of Inferred Fuzzy Set 推理模糊集的解模糊 ……………………298
6.5 Mamdani Model for Fuzzy Inference Systems 模糊推理系統(tǒng)的Mamdani模型 …………301
6.5.1 Mobile Robot Navigation Among Moving Obstacles
移動障礙物環(huán)境下的移動機器人導航 ……………………………………………303
6.5.2 Mortgage Loan Assessment 抵押貸款評估 ………………………………………308
6.6 Takagi-Sugeno Fuzzy Model Takagi-Sugeno模糊模型 …………………………………311
6.7 Neuro-Fuzzy Inference Systems 神經模糊推理系統(tǒng) ……………………………………317
6.7.1 ANFIS Architecture ANFIS結構 …………………………………………………318
6.7.2 How Does an ANFIS Learn? ANFIS如何學習 ……………………………………320
6.8 Gentic-Fuzzy Systems 遺傳模糊系統(tǒng) ……………………………………………………324
7. Data Clustering and Data Transformations 數(shù)據(jù)聚類和數(shù)據(jù)變換… ………………………328
7.1 Unsupervised Learning 非監(jiān)督學習 ………………………………………………………328
7.1.1 Clustering 聚類 ……………………………………………………………………329
7.2 Engineering the Data 數(shù)據(jù)工程 ……………………………………………………………331
7.2.1 Exploratory Data Analysis: Learning about What is in the Data
探索數(shù)據(jù)分析：了解數(shù)據(jù)中的內容 ………………………………………………333
7.2.2 Cluster Analysis: Finding Similarities in the Data
聚類分析：發(fā)現(xiàn)數(shù)據(jù)間的相似性 …………………………………………………334
7.2.3 Data Transformations: Enhancing the Information Content of the Data
數(shù)據(jù)變換：提高數(shù)據(jù)的信息量 ……………………………………………………339
7.3 Overview of Basic Clustering Methods 基本聚類方法概述 ………………………………341
7.3.1 Partitional Clustering 劃分聚類 ……………………………………………………341
7.3.2 Hierarchical Clustering 層次聚類 …………………………………………………344
7.3.3 Spectral Clustering 譜聚類 …………………………………………………………345
7.3.4 Clustering using Self-Organizing Maps 自組織映射聚類 ………………………349
7.4 K-Means Clustering K均值聚類 …………………………………………………………352
7.5 Fuzzy K-Means Clustering 模糊K均值聚類 ……………………………………………356
7.6 Expectation-Maximization (EM) Algorithm and Gaussian Mixtures Clustering
期望最大化（EM）算法和高斯混合聚類 …………………………………………………362
7.6.1 EM Algorithm EM算法 ……………………………………………………………362
7.6.2 Gaussian Mixture Models 高斯混合模型 …………………………………………365
7.7 Some Useful Data Transformations 一些有用的數(shù)據(jù)變換 ………………………………372
7.7.1 Data Cleansing 數(shù)據(jù)清洗 …………………………………………………………372
7.7.2 Derived Attributes 衍生屬性 ………………………………………………………374
7.7.3 Discretizing Numeric Attributes 離散化數(shù)值屬性 ………………………………375
7.7.4 Attribute Reduction Techniques 屬性約簡技術 ……………………………………377
7.8 Entropy-Based Method for Attribute Discretization 基于熵的屬性離散化方法 …………377
7.9 Principal Components Analysis (PCA) for Attribute Reduction
用于屬性約簡的主成分分析（PCA） ……………………………………………………382
7.10 Rough Sets-Based Methods for Attribute Reduction 基于粗糙集的屬性約簡方法 ……390
7.10.1 Rough Set Preliminaries 粗糙集準備 ……………………………………………392
7.10.2 Analysis of Relevance of Attributes 屬性相關性分析 ……………………………397
7.10.3 Reduction of Attributes 約簡屬性 …………………………………………………399
8. Decision Tree Learning 決策樹學習……………………………………………………………404
8.1 Introduction 簡介 …………………………………………………………………………404
8.2 Example of a Classification Decision Tree 分類決策樹示例 ……………………………406
8.3 Measures of Impurity for Evaluating Splits in Decision Trees
評價決策樹分裂的不純度度量 ……………………………………………………………411
8.3.1 Information Gain/Entropy reduction 信息增益/熵減少 …………………………411
8.3.2 Gain Ratio 增益率 …………………………………………………………………416
8.3.3 Gini Index 基尼指數(shù) ………………………………………………………………417
8.4 ID3, C4.5, and CART Decision Trees ID3, C4.5與CART決策樹 ………………………418
8.5 Pruning the Tree 剪枝 ………………………………………………………………………427
8.6 Strengths and Weaknesses of Decision-Tree Approach 決策樹方法的優(yōu)缺點 …………429
8.7 Fuzzy Decision Trees 模糊決策樹 …………………………………………………………433
9. Business Intelligence and Data Mining: Techniques and Applications
商業(yè)智能和數(shù)據(jù)挖掘：技術與應用… …………………………………………………………445
9.1 An Introduction to Analytics 分析簡介 ……………………………………………………445
9.1.1 Machine Learning, Data Mining, and Predictive Analytics
機器學習、數(shù)據(jù)挖掘和預測分析 …………………………………………………448
9.1.2 Basic Analytics Techniques 基本分析技術 ………………………………………449
9.2 The CRISP-DM (Cross Industry Standard Process for Data Mining) Model
CRISP-DM（數(shù)據(jù)挖掘跨行業(yè)標準流程）模型 ……………………………………………451
9.3 Data Warehousing and Online Analytical Processing 數(shù)據(jù)倉庫和聯(lián)機分析處理 ………456
9.3.1 Basic Concepts 基本概念 …………………………………………………………456
9.3.2 Databases 數(shù)據(jù)庫 …………………………………………………………………458
9.3.3 Data Warehousing: A General Architecture, and OLAP Operations
數(shù)據(jù)倉庫：通用架構和OLAP操作 ………………………………………………461
9.3.4 Data Mining in the Data Warehouse Environment
數(shù)據(jù)倉庫環(huán)境中的數(shù)據(jù)挖掘 ………………………………………………………466
9.4 Mining Frequent Patterns and Association Rules 挖掘頻繁模式和關聯(lián)規(guī)則 ……………467
9.4.1 Basic Concepts 基本概念 …………………………………………………………469
9.4.2 Measures of Strength of Frequent Patterns and Association Rules
頻繁模式和關聯(lián)規(guī)則的強度的度量 ………………………………………………471
9.4.3 Frequent Item Set Mining Methods 頻繁項集挖掘方法 …………………………473
9.4.4 Generating Association Rules from Frequent Itemsets
從頻繁項集生成關聯(lián)規(guī)則 …………………………………………………………477
9.5 Intelligent Information Retrieval Systems 智能信息檢索系統(tǒng) ……………………………479
9.5.1 Text Retrieval 文本檢索 ……………………………………………………………483
9.5.2 Image Retrieval 圖像檢索 …………………………………………………………486
9.5.3 Audio Retrieval 音頻檢索 …………………………………………………………488
9.6 Applications and Trends 應用和趨勢 ………………………………………………………490
9.6.1 Data Mining Applications 數(shù)據(jù)挖掘應用 …………………………………………490
9.6.2 Data Mining Trends 數(shù)據(jù)挖掘趨勢 ………………………………………………495
9.7 Technologies for Big Data 大數(shù)據(jù)技術 ……………………………………………………498
9.7.1 Emerging Analytic Methods 新興分析方法 ………………………………………500
9.7.2 Emerging Technologies for Higher Levels of Scalability
用于更高層次可擴展性的新興技術 ………………………………………………503
Appendix A Genetic Algorithm (GA) For Search Optimization
搜索優(yōu)化的遺傳算法（GA）… …………………………………………………… 508
A.1 A Simple Overview of Genetics 遺傳學的簡單概述 ……………………………………510
A.2 Genetics on Computers 計算機遺傳學 ……………………………………………………511
A.3 The Basic Genetic Algorithm 基本遺傳算法 ……………………………………………515
A.4 Beyond the Basic Genetic Algorithm 超越基本遺傳算法 ………………………………524
Appendix B Reinforcement Learning (RL) 強化學習（RL）… ……………………………527
B.1 Introduction 簡介 …………………………………………………………………………527
B.2 Elements of Reinforcement Learning 強化學習的要素 …………………………………530
B.3 Basics of Dynamic Programming 動態(tài)編程基礎 …………………………………………535
B.3.1 Finding Optimal Policies 尋找最優(yōu)策略 …………………………………………538
B.3.2 Value Iteration 值迭代 ……………………………………………………………539
B.3.3 Policy Iteration 策略迭代 …………………………………………………………540
B.4 Temporal Difference Learning 時間差分學習 ……………………………………………542
B.4.1 Q-learning Q-學習 …………………………………………………………………544
B.4.2 Generalization 歸納 ………………………………………………………………546
B.4.3 Sarsa-learning SARSA學習 ………………………………………………………548
Datasets from Real-Life Applications for Machine Learning Experiments
機器學習實驗的實際應用數(shù)據(jù)集………………………………………………………………… 549
Problems 習題… …………………………………………………………………………………567
References 參考文獻………………………………………………………………………………613
Index 索引… ………………………………………………………………………………………623