-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathatom.xml
More file actions
202 lines (118 loc) · 138 KB
/
Copy pathatom.xml
File metadata and controls
202 lines (118 loc) · 138 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>欢迎光临</title>
<subtitle>https://nghqqa.cn/</subtitle>
<link href="/atom.xml" rel="self"/>
<link href="www.nghqqa.cn/"/>
<updated>2019-08-22T04:31:47.698Z</updated>
<id>www.nghqqa.cn/</id>
<author>
<name>逆光海</name>
</author>
<generator uri="http://hexo.io/">Hexo</generator>
<entry>
<title>numpy库的使用</title>
<link href="www.nghqqa.cn/2019/08/16/numpy%E5%BA%93%E7%9A%84%E4%BD%BF%E7%94%A8/"/>
<id>www.nghqqa.cn/2019/08/16/numpy库的使用/</id>
<published>2019-08-16T05:49:03.000Z</published>
<updated>2019-08-22T04:31:47.698Z</updated>
<content type="html"><![CDATA[<h1 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h1><p>numpy库是python中一个非常重要的库,它提供了一个多维数组（ndarray）数据类型，以及关于多维数组的操作，NumPy 已经成为其他大数据和机器学习模块的基础。 </p><a id="more"></a><p>如果不想安装numpy库的,而你的电脑里没有安装python语言的,我们可以使用Anaconda,Anaconda里包含了很多我们要用到的科学包,包括了panda,numpy等,Anaconda可以在它的官网下载,速度可能会较慢.</p><p>官网地址为:<a href="https://www.anaconda.com/" target="_blank" rel="noopener">https://www.anaconda.com/</a></p><p>安装Anaconda的教程可以看这篇文章:<a href="https://blog.csdn.net/qq_39610888/article/details/80805356" target="_blank" rel="noopener">https://blog.csdn.net/qq_39610888/article/details/80805356</a></p><p>如果想看视频的可以去b站看这个视频:<a href="https://www.bilibili.com/video/av23124018?from=search&seid=17113017816135374394" target="_blank" rel="noopener">https://www.bilibili.com/video/av23124018?from=search&seid=17113017816135374394</a></p><p>安装完,我们就能在anaconda自带的juptyer里写代码了.</p><h1 id="numpy的数组属性"><a href="#numpy的数组属性" class="headerlink" title="numpy的数组属性"></a>numpy的数组属性</h1><p>在使用numpy我们需要像java一样,先进行导包</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">import numpy as np</span><br></pre></td></tr></table></figure><p>numpy我们数组创建的常规创建方法</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">array=np.array([[<span class="number">1</span>,<span class="number">2</span>,<span class="number">3</span>],</span><br><span class="line"> [<span class="number">4</span>,<span class="number">5</span>,<span class="number">6</span>],</span><br><span class="line"> [<span class="number">7</span>,<span class="number">8</span>,<span class="number">9</span>]])</span><br><span class="line"></span><br><span class="line">print(array)</span><br></pre></td></tr></table></figure><p>输出结果</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[[1 2 3]</span><br><span class="line"> [4 5 6]</span><br><span class="line"> [7 8 9]]</span><br></pre></td></tr></table></figure><p>这里我们的第一个array数组就创建好了.</p><p>我们可以使用numpy里的基本属性来获取我们新建数组的一些信息</p><h3 id="ndarray-ndim"><a href="#ndarray-ndim" class="headerlink" title="ndarray.ndim"></a>ndarray.ndim</h3><p>NumPy 数组的维数称为秩（rank），一维数组的秩为 1，二维数组的秩为 2，以此类推。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">print(array.ndim)</span><br></pre></td></tr></table></figure><p>ndarray.ndim 用于返回数组的维数，等于秩。</p><p>输出结果为：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">2</span><br></pre></td></tr></table></figure><h3 id="ndarray-shape"><a href="#ndarray-shape" class="headerlink" title="ndarray.shape"></a>ndarray.shape</h3><p>表示数组的维度，返回一个元组，这个元组的长度就是维度的数目，即 ndim 属性(秩)。比如，一个二维数组，其维度表示”行数”和”列数”。</p><p>ndarray.shape 也可以用于调整数组大小。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">print(array.shape)<span class="comment">#代表矩阵的维度</span></span><br></pre></td></tr></table></figure><p>输出结果为：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">(3, 3)</span><br></pre></td></tr></table></figure><h3 id="ndarray-size"><a href="#ndarray-size" class="headerlink" title="ndarray.size"></a>ndarray.size</h3><p>数组元素的总个数，相当于 .shape 中 n*m 的值</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">print(array.size)<span class="comment">#代表数组元素的总个数</span></span><br></pre></td></tr></table></figure><p>输出结果为：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">9</span><br></pre></td></tr></table></figure><h3 id="ndarray-dtype"><a href="#ndarray-dtype" class="headerlink" title="ndarray.dtype"></a>ndarray.dtype</h3><p>ndarray 对象的元素类型</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">print(array.dtype)<span class="comment">#代表矩阵中元素的属性</span></span><br></pre></td></tr></table></figure><p>输出结果为：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">int32</span><br></pre></td></tr></table></figure><h3 id="numpy的数据类型"><a href="#numpy的数据类型" class="headerlink" title="numpy的数据类型"></a>numpy的数据类型</h3><p>numpy 支持的数据类型比 Python 内置的类型要多很多，基本上可以和 C 语言的数据类型对应上，其中部分类型对应为 Python 内置的类型。下表列举了常用 NumPy 基本类型。</p><p><img src="http://qqa.nghqqa.cn/blog/20190816/iW0gMI5m7R1u.png?imageslim" alt="mark"></p><h1 id="创建array的方法"><a href="#创建array的方法" class="headerlink" title="创建array的方法"></a>创建array的方法</h1><h3 id="numpy-zeros"><a href="#numpy-zeros" class="headerlink" title="numpy.zeros"></a>numpy.zeros</h3><p>创建指定大小的数组，数组元素以 0 来填充：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line">zero=np.zeros((<span class="number">2</span>,<span class="number">3</span>))<span class="comment">#生成2行3列全为0的矩阵</span></span><br><span class="line">print(zero)</span><br></pre></td></tr></table></figure><p>输出结果为：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[[0. 0. 0.]</span><br><span class="line"> [0. 0. 0.]]</span><br></pre></td></tr></table></figure><h3 id="numpy-ones"><a href="#numpy-ones" class="headerlink" title="numpy.ones"></a>numpy.ones</h3><p>创建指定形状的数组，数组元素以 1 来填充：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line">one=np.ones((<span class="number">3</span>,<span class="number">4</span>))<span class="comment">#生成3行4列全为1的矩阵</span></span><br><span class="line">print(one)</span><br></pre></td></tr></table></figure><p>输出结果为：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[[1. 1. 1. 1.]</span><br><span class="line"> [1. 1. 1. 1.]</span><br><span class="line"> [1. 1. 1. 1.]]</span><br></pre></td></tr></table></figure><h3 id="numpy-empty"><a href="#numpy-empty" class="headerlink" title="numpy.empty"></a>numpy.empty</h3><p>numpy.empty 方法用来创建一个指定形状（shape）、数据类型（dtype）且未初始化的数组：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line">empty=np.empty((<span class="number">3</span>,<span class="number">2</span>))<span class="comment">#生成3行2列全都接近0（不等于0）的矩阵</span></span><br><span class="line">print(empty)</span><br></pre></td></tr></table></figure><p>输出结果为：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[[0. 0.]</span><br><span class="line"> [0. 0.]</span><br><span class="line"> [0. 0.]]</span><br></pre></td></tr></table></figure><h3 id="numpy-arange"><a href="#numpy-arange" class="headerlink" title="numpy.arange"></a>numpy.arange</h3><p>numpy 包中的使用 arange 函数创建数值范围并返回 ndarray 对象，函数格式如下：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">numpy.arange(start, stop, step, dtype)</span><br></pre></td></tr></table></figure><p>根据 start 与 stop 指定的范围以及 step 设定的步长，生成一个 ndarray。</p><p>参数说明：</p><table><thead><tr><th>参数</th><th align="center"><strong>描述</strong></th></tr></thead><tbody><tr><td>start</td><td align="center">起始值，默认为<code>0</code></td></tr><tr><td>stop</td><td align="center">终止值（不包含）</td></tr><tr><td>step</td><td align="center">步长，默认为<code>1</code></td></tr><tr><td>dtype</td><td align="center">返回<code>ndarray</code>的数据类型，如果没有提供，则会使用输入数据的类型。</td></tr></tbody></table><p>实例</p><p>生成 0 到 9 的数组:</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line">e=np.arange(<span class="number">10</span>)</span><br><span class="line">print(e)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[0 1 2 3 4 5 6 7 8 9]</span><br></pre></td></tr></table></figure><p>生成 4到 11 的数组:</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line">f=np.arange(<span class="number">4</span>,<span class="number">12</span>)</span><br><span class="line">print(f)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[ 4 5 6 7 8 9 10 11]</span><br></pre></td></tr></table></figure><p>生成 1到 19 (每生成一个数字间隔3个数字)的数组:</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line">g=np.arange(<span class="number">1</span>,<span class="number">20</span>,<span class="number">3</span>)</span><br><span class="line">print(g)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[ 1 4 7 10 13 16 19]</span><br></pre></td></tr></table></figure><h1 id="Numpy-数组操作"><a href="#Numpy-数组操作" class="headerlink" title="Numpy 数组操作"></a>Numpy 数组操作</h1><h2 id="修改数组形状"><a href="#修改数组形状" class="headerlink" title="修改数组形状"></a>修改数组形状</h2><h3 id="numpy-reshape"><a href="#numpy-reshape" class="headerlink" title="numpy.reshape"></a>numpy.reshape</h3><p>numpy.reshape 函数可以在不改变数据的条件下修改形状，格式如下：</p><p> numpy.reshape(arr, newshape, order=’C’)</p><ul><li>arr：要修改形状的数组</li><li>newshape：整数或者整数数组，新的形状应当兼容原有形状</li><li>order：’C’ – 按行，’F’ – 按列，’A’ – 原顺序，’k’ – 元素在内存中的出现顺序。</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line">h=np.arange(<span class="number">8</span>).reshape(<span class="number">4</span>,<span class="number">2</span>)<span class="comment">#重新定义矩阵的形状</span></span><br><span class="line">print(h)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[[0 1]</span><br><span class="line"> [2 3]</span><br><span class="line"> [4 5]</span><br><span class="line"> [6 7]]</span><br></pre></td></tr></table></figure><h2 id="翻转数组"><a href="#翻转数组" class="headerlink" title="翻转数组"></a>翻转数组</h2><h3 id="numpy-transpose"><a href="#numpy-transpose" class="headerlink" title="numpy.transpose"></a>numpy.transpose</h3><p>numpy.transpose 函数用于对换数组的维度，格式如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">numpy.transpose(arr, axes)</span><br></pre></td></tr></table></figure><ul><li>arr：要操作的数组</li><li>axes：整数列表，对应维度，通常所有维度都会对换。</li></ul><p>实例</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"> </span><br><span class="line">a = np.arange(<span class="number">12</span>).reshape(<span class="number">3</span>,<span class="number">4</span>)</span><br><span class="line"> </span><br><span class="line"><span class="keyword">print</span> (<span class="string">'原数组：'</span>)</span><br><span class="line"><span class="keyword">print</span> (a )</span><br><span class="line"><span class="keyword">print</span> (<span class="string">'\n'</span>)</span><br><span class="line"> </span><br><span class="line"><span class="keyword">print</span> (<span class="string">'对换数组：'</span>)</span><br><span class="line"><span class="keyword">print</span> (np.transpose(a))</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">原数组：</span><br><span class="line">[[ 0 1 2 3]</span><br><span class="line"> [ 4 5 6 7]</span><br><span class="line"> [ 8 9 10 11]]</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">对换数组：</span><br><span class="line">[[ 0 4 8]</span><br><span class="line"> [ 1 5 9]</span><br><span class="line"> [ 2 6 10]</span><br><span class="line"> [ 3 7 11]]</span><br></pre></td></tr></table></figure><h3 id="ndarray-T"><a href="#ndarray-T" class="headerlink" title="ndarray.T"></a>ndarray.T</h3><p>一维数组不能转置</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line">arr1=np.array([<span class="number">1</span>,<span class="number">2</span>,<span class="number">3</span>])</span><br><span class="line">arr1.T</span><br><span class="line">print(arr1)<span class="comment">#一维的array不能转置</span></span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[1 2 3]</span><br></pre></td></tr></table></figure><p>numpy.ndarray.T 类似 numpy.transpose</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">import numpy as np</span><br><span class="line"> </span><br><span class="line">a = np.arange(12).reshape(3,4)</span><br><span class="line"> </span><br><span class="line">print ('原数组：')</span><br><span class="line">print (a)</span><br><span class="line">print ('\n')</span><br><span class="line"> </span><br><span class="line">print ('转置数组：')</span><br><span class="line">print (a.T)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">原数组：</span><br><span class="line">[[ 0 1 2 3]</span><br><span class="line"> [ 4 5 6 7]</span><br><span class="line"> [ 8 9 10 11]]</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">转置数组：</span><br><span class="line">[[ 0 4 8]</span><br><span class="line"> [ 1 5 9]</span><br><span class="line"> [ 2 6 10]</span><br><span class="line"> [ 3 7 11]]</span><br></pre></td></tr></table></figure><h2 id="改变维度"><a href="#改变维度" class="headerlink" title="改变维度"></a>改变维度</h2><p>使用np.newaxis()函数</p><p>例子:</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">import numpy as np</span><br><span class="line">arr1=np.array([1,2,3])</span><br><span class="line">arr_1=arr1[np.newaxis,:]#改变维度,在行这里添加一个维度，使其变成1行3列</span><br><span class="line">print(arr_1)</span><br><span class="line">print(arr_1.shape)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[[1 2 3]]</span><br><span class="line">(1, 3)</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">import numpy as np</span><br><span class="line">arr1=np.array([1,2,3])</span><br><span class="line">arr_2=arr1[:,np.newaxis]#在列这里添加一个维度，使其变成3行1列</span><br><span class="line">print(arr_2)</span><br><span class="line">print(arr_2.shape)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[[1]</span><br><span class="line"> [2]</span><br><span class="line"> [3]]</span><br><span class="line">(3, 1)</span><br></pre></td></tr></table></figure><p>使用</p><p>例子：np.atleast_2d()和np.atleast_3d()函数</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">print(arr1)#原数据，arr1是一维数据</span><br><span class="line">arr_31=np.atleast_2d(arr1)#把arr1变成二维数据</span><br><span class="line">arr_32=np.atleast_3d(arr1)#把arr1变成三维数据</span><br><span class="line"></span><br><span class="line">print(arr_31)</span><br><span class="line">print(arr_32)</span><br><span class="line"></span><br><span class="line">print(arr_31.T)#二维数据转置</span><br><span class="line">print(arr_32.T)#三维数据转置</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">[1 2 3]</span><br><span class="line"></span><br><span class="line">[[1 2 3]]</span><br><span class="line"></span><br><span class="line">[[[1]</span><br><span class="line"> [2]</span><br><span class="line"> [3]]]</span><br><span class="line"> </span><br><span class="line">[[1]</span><br><span class="line"> [2]</span><br><span class="line"> [3]]</span><br><span class="line"> </span><br><span class="line">[[[1]</span><br><span class="line"> [2]</span><br><span class="line"> [3]]]</span><br></pre></td></tr></table></figure><h2 id="numpy的基本运算"><a href="#numpy的基本运算" class="headerlink" title="numpy的基本运算"></a>numpy的基本运算</h2><p>先创建数组</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line">arr1=np.array([[<span class="number">1</span>,<span class="number">2</span>,<span class="number">3</span>],</span><br><span class="line"> [<span class="number">4</span>,<span class="number">5</span>,<span class="number">6</span>]])</span><br><span class="line">arr2=np.array([[<span class="number">1</span>,<span class="number">1</span>,<span class="number">2</span>],</span><br><span class="line"> [<span class="number">2</span>,<span class="number">3</span>,<span class="number">3</span>]])</span><br><span class="line">print(arr1)</span><br><span class="line">print(arr2)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[[1 2 3]</span><br><span class="line"> [4 5 6]]</span><br><span class="line">[[1 1 2]</span><br><span class="line"> [2 3 3]]</span><br></pre></td></tr></table></figure><h3 id="加法"><a href="#加法" class="headerlink" title="加法"></a>加法</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">print(arr1+arr2)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[[2 3 5]</span><br><span class="line"> [6 8 9]]</span><br></pre></td></tr></table></figure><h3 id="减法"><a href="#减法" class="headerlink" title="减法"></a>减法</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">print(arr1-arr2)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[[0 1 1]</span><br><span class="line"> [2 2 3]]</span><br></pre></td></tr></table></figure><h3 id="乘法"><a href="#乘法" class="headerlink" title="乘法"></a>乘法</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">print(arr1*arr2)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[[ 1 2 6]</span><br><span class="line"> [ 8 15 18]]</span><br></pre></td></tr></table></figure><h3 id="次方"><a href="#次方" class="headerlink" title="次方"></a>次方</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">print(arr1**arr2)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[[ 1 2 9]</span><br><span class="line"> [ 16 125 216]]</span><br></pre></td></tr></table></figure><h3 id="相除"><a href="#相除" class="headerlink" title="相除"></a>相除</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">print(arr1/arr2)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[[1. 2. 1.5 ]</span><br><span class="line"> [2. 1.66666667 2. ]]</span><br></pre></td></tr></table></figure><h3 id="求余"><a href="#求余" class="headerlink" title="求余"></a>求余</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">print(arr1%arr2)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[[0 0 1]</span><br><span class="line"> [0 2 0]]</span><br></pre></td></tr></table></figure><h3 id="整除"><a href="#整除" class="headerlink" title="整除"></a>整除</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">print(arr1//arr2)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[[1 2 1]</span><br><span class="line"> [2 1 2]]</span><br></pre></td></tr></table></figure><h3 id="整体加"><a href="#整体加" class="headerlink" title="整体加"></a>整体加</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">print(arr1+<span class="number">2</span>)<span class="comment">#所有的元素都加2</span></span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[[3 4 5]</span><br><span class="line"> [6 7 8]]</span><br></pre></td></tr></table></figure><h3 id="整体乘"><a href="#整体乘" class="headerlink" title="整体乘"></a>整体乘</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">print(arr1*<span class="number">10</span>)<span class="comment">#所有的元素都乘以10</span></span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[[10 20 30]</span><br><span class="line"> [40 50 60]]</span><br></pre></td></tr></table></figure><p>判断大小</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">arr3=arr1><span class="number">3</span><span class="comment">#判断那个元素大于3</span></span><br><span class="line">print(arr3)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[[False False False]</span><br><span class="line"> [ True True True]]</span><br></pre></td></tr></table></figure><h3 id="numpy-dot"><a href="#numpy-dot" class="headerlink" title="numpy.dot()"></a>numpy.dot()</h3><p>numpy.dot() 对于两个一维的数组，计算的是这两个数组对应下标元素的乘积和(数学上称之为内积)；对于二维数组，计算的是两个数组的矩阵乘积；对于多维数组，它的通用计算公式如下，即结果数组中的每个元素都是：数组a的最后一维上的所有元素与数组b的倒数第二位上的所有元素的乘积和： <strong>dot(a, b)[i,j,k,m] = sum(a[i,j,:] * b[k,:,m])</strong>。</p><p>先定义一个arr4</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">import numpy as np</span><br><span class="line">arr4=np.ones((3,5))</span><br><span class="line">print(arr4)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[[1. 1. 1. 1. 1.]</span><br><span class="line"> [1. 1. 1. 1. 1.]</span><br><span class="line"> [1. 1. 1. 1. 1.]]</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">np.dot(arr1,arr4)#矩阵乘法</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">array([[ 6., 6., 6., 6., 6.],</span><br><span class="line"> [15., 15., 15., 15., 15.]])</span><br></pre></td></tr></table></figure><p>对于矩阵乘法忘记了的或者了解不深的可以看这篇文章：<a href="https://www.cnblogs.com/alantu2018/p/8528299.html" target="_blank" rel="noopener">https://www.cnblogs.com/alantu2018/p/8528299.html</a></p><h3 id="随机数生成及矩阵的运算"><a href="#随机数生成及矩阵的运算" class="headerlink" title="随机数生成及矩阵的运算"></a>随机数生成及矩阵的运算</h3><h4 id="np-random-random-函数"><a href="#np-random-random-函数" class="headerlink" title="np.random.random()函数"></a>np.random.random()函数</h4><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line">samolel=np.random.random((<span class="number">3</span>,<span class="number">2</span>))<span class="comment">#生成3行2列的随机数，范围（0-1）</span></span><br><span class="line">print(samolel)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[[0.6150121 0.59427834]</span><br><span class="line"> [0.58806337 0.892053 ]</span><br><span class="line"> [0.59315305 0.92415359]]</span><br></pre></td></tr></table></figure><h4 id="numpy-random-standard-normal-函数用法"><a href="#numpy-random-standard-normal-函数用法" class="headerlink" title="numpy.random.standard_normal()函数用法"></a>numpy.random.standard_normal()函数用法</h4><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line">samole2=np.random.normal(size=(<span class="number">3</span>,<span class="number">2</span>))<span class="comment">#生成3行2列的随机数，符合标准正态分布</span></span><br><span class="line">print(samole2)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[[ 0.36192863 -0.30079782]</span><br><span class="line"> [ 0.21587959 -0.15799396]</span><br><span class="line"> [-0.87615964 1.23683354]]</span><br></pre></td></tr></table></figure><h2 id="矩阵的运算"><a href="#矩阵的运算" class="headerlink" title="矩阵的运算"></a>矩阵的运算</h2><h3 id="对于矩阵进行求和"><a href="#对于矩阵进行求和" class="headerlink" title="对于矩阵进行求和"></a>对于矩阵进行求和</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">np.sum(samolel)#对于samolel中的元素进行求和</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">4.206713449491107</span><br></pre></td></tr></table></figure><p>对矩阵中的列进行求和</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">np.sum(samolel,axis=<span class="number">0</span>)<span class="comment">#当axis=0时是对于列，axis=1对于行，在这里是对列求和</span></span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">array([1.79622852, 2.41048493])</span><br></pre></td></tr></table></figure><p>对矩阵中的列进行行求和</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">np.sum(samolel,axis=1)#在这里是对行求和</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">array([1.20929044, 1.48011637, 1.51730664])</span><br></pre></td></tr></table></figure><h3 id="提取矩阵中的最小值"><a href="#提取矩阵中的最小值" class="headerlink" title="提取矩阵中的最小值"></a>提取矩阵中的最小值</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">np.min(samolel)#提取samolel中的最小值</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">0.5880633706006644</span><br></pre></td></tr></table></figure><h3 id="提取矩阵中的最大值"><a href="#提取矩阵中的最大值" class="headerlink" title="提取矩阵中的最大值"></a>提取矩阵中的最大值</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">np.max(samolel)#提取samolel中的最大值</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">0.9241535872093841</span><br></pre></td></tr></table></figure><h3 id="求矩阵中的平均值"><a href="#求矩阵中的平均值" class="headerlink" title="求矩阵中的平均值"></a>求矩阵中的平均值</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">print(np.mean(samolel))#求所有元素的平均值</span><br><span class="line">print(samolel.mean())</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">0.7011189082485179</span><br><span class="line">0.7011189082485179</span><br></pre></td></tr></table></figure><h3 id="求矩阵中的中位数"><a href="#求矩阵中的中位数" class="headerlink" title="求矩阵中的中位数"></a>求矩阵中的中位数</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">np.median(samolel)#求所有元素的中位数</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">0.6046452201028809</span><br></pre></td></tr></table></figure><h3 id="对于矩阵的数组就行开方"><a href="#对于矩阵的数组就行开方" class="headerlink" title="对于矩阵的数组就行开方"></a>对于矩阵的数组就行开方</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">np.sqrt(samolel)#开方</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">array([[0.78422707, 0.77089451],</span><br><span class="line"> [0.7668529 , 0.94448558],</span><br><span class="line"> [0.7701643 , 0.96132907]])</span><br></pre></td></tr></table></figure><h3 id="对于矩阵的进行排序"><a href="#对于矩阵的进行排序" class="headerlink" title="对于矩阵的进行排序"></a>对于矩阵的进行排序</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">np.sort(samolel)#排序</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">array([[0.39407608, 0.72871605],</span><br><span class="line"> [0.48484466, 0.73652244],</span><br><span class="line"> [0.19709554, 0.44424115]])</span><br></pre></td></tr></table></figure><h3 id="生成1行10列从1到10的随机数"><a href="#生成1行10列从1到10的随机数" class="headerlink" title="生成1行10列从1到10的随机数"></a>生成1行10列从1到10的随机数</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">import numpy as np</span><br><span class="line">samole4=np.random.randint(0,10,size=(1,10))</span><br><span class="line">print(samole4)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[[4 1 3 1 3 9 8 2 3 0]]</span><br></pre></td></tr></table></figure><p>对于samole4使用np.clip()函数</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">np.clip(samole4,2,7)#小于2的元素变成2，大于7的元素变成7</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">array([[7, 7, 2, 6, 2, 5, 2, 5, 7, 6]])</span><br></pre></td></tr></table></figure><h4 id="np-clip-函数"><a href="#np-clip-函数" class="headerlink" title="np.clip()函数"></a>np.clip()函数</h4><p>numpy.clip(a, a_min, a_max, out=None)</p><ul><li>参数说明 </li><li>a : 输入的数组</li><li>a_min: 限定的最小值 也可以是数组 如果为数组时 shape必须和a一样</li><li>a_max:限定的最大值 也可以是数组 shape和a一样</li><li>out：剪裁后的数组存入的数组</li></ul><h2 id="numpy的索引和切片，合并"><a href="#numpy的索引和切片，合并" class="headerlink" title="numpy的索引和切片，合并"></a>numpy的索引和切片，合并</h2><h3 id="索引"><a href="#索引" class="headerlink" title="索引"></a>索引</h3><h4 id="一维数组的索引"><a href="#一维数组的索引" class="headerlink" title="一维数组的索引"></a>一维数组的索引</h4><p>先新建一个array数组</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line">arr1=np.arange(<span class="number">2</span>,<span class="number">14</span>)</span><br><span class="line">print(arr1)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[ 2 3 4 5 6 7 8 9 10 11 12 13]</span><br></pre></td></tr></table></figure><h4 id="获取位置在2的数据"><a href="#获取位置在2的数据" class="headerlink" title="获取位置在2的数据"></a>获取位置在2的数据</h4><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">print(arr1[<span class="number">2</span>])<span class="comment">#第二个位置的数据</span></span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">4</span><br></pre></td></tr></table></figure><h4 id="获取第一到第三个位置的数据"><a href="#获取第一到第三个位置的数据" class="headerlink" title="获取第一到第三个位置的数据"></a>获取第一到第三个位置的数据</h4><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">print(arr1[<span class="number">1</span>:<span class="number">4</span>])<span class="comment">#第一到第三个位置的数据，【1:4】不会包括4</span></span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[3 4 5]</span><br></pre></td></tr></table></figure><h4 id="获取第二个到倒数第一个位置的数据"><a href="#获取第二个到倒数第一个位置的数据" class="headerlink" title="获取第二个到倒数第一个位置的数据"></a>获取第二个到倒数第一个位置的数据</h4><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">print(arr1[<span class="number">2</span>:<span class="number">-1</span>])<span class="comment">#第二到倒数第一个位置的数据</span></span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[ 4 5 6 7 8 9 10 11 12]</span><br></pre></td></tr></table></figure><p>获取前5个数据</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">print(arr1[:<span class="number">5</span>])<span class="comment">#前5个数据</span></span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[2 3 4 5 6]</span><br></pre></td></tr></table></figure><h4 id="获取最后两个数据"><a href="#获取最后两个数据" class="headerlink" title="获取最后两个数据"></a>获取最后两个数据</h4><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">print(arr1[<span class="number">-2</span>:])<span class="comment">#取最后两个数据</span></span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[12 13]</span><br></pre></td></tr></table></figure><h3 id="二维数组的索引"><a href="#二维数组的索引" class="headerlink" title="二维数组的索引"></a>二维数组的索引</h3><p>把之前的一维数组变为二维数组</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">arr2=arr1.reshape(3,4)</span><br><span class="line">print(arr2)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[[ 2 3 4 5]</span><br><span class="line"> [ 6 7 8 9]</span><br><span class="line"> [10 11 12 13]]</span><br></pre></td></tr></table></figure><h4 id="获取第一行数据"><a href="#获取第一行数据" class="headerlink" title="获取第一行数据"></a>获取第一行数据</h4><p>对于一维数组而言,索引[1]是获取第一位的数据,而对于二维数组索引[1]是获取第一行的数据</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">print(arr2[1])#第一行</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[6 7 8 9]</span><br></pre></td></tr></table></figure><p>想要获取一个数据的话就要写清在二维数组的某一行某一列</p><p>例如:</p><p>获取第一行第一列的数据</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">print(arr2[[<span class="number">1</span>][<span class="number">1</span>])<span class="comment">#第一行的第一列</span></span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">7</span><br></pre></td></tr></table></figure><p>获取第一行第二列的数据</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">print(arr2[<span class="number">1</span>,<span class="number">2</span>])<span class="comment">#第一行的第二列</span></span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">8</span><br></pre></td></tr></table></figure><p>获取所有行的第二列</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">print(arr2[:,<span class="number">2</span>])<span class="comment">#所有行的第二列</span></span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[ 4 8 12]</span><br></pre></td></tr></table></figure><h3 id="切片"><a href="#切片" class="headerlink" title="切片"></a>切片</h3><p>新建一个数组</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line">arr1=np.arange(<span class="number">12</span>).reshape(<span class="number">3</span>,<span class="number">4</span>)</span><br><span class="line">print(arr1)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[[ 0 1 2 3]</span><br><span class="line"> [ 4 5 6 7]</span><br><span class="line"> [ 8 9 10 11]]</span><br></pre></td></tr></table></figure><h4 id="按列切片"><a href="#按列切片" class="headerlink" title="按列切片"></a>按列切片</h4><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">arr2,arr3=np.split(arr1,<span class="number">2</span>,axis=<span class="number">1</span>)<span class="comment">#按列方向分割,分成2份</span></span><br><span class="line">print(arr2)</span><br><span class="line">print(arr3)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">[[0 1]</span><br><span class="line"> [4 5]</span><br><span class="line"> [8 9]]</span><br><span class="line">[[ 2 3]</span><br><span class="line"> [ 6 7]</span><br><span class="line"> [10 11]]</span><br></pre></td></tr></table></figure><h4 id="按行切片"><a href="#按行切片" class="headerlink" title="按行切片"></a>按行切片</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">arr4,arr5,arr6=np.split(arr1,3,axis=0)#按行方向分割,分成3份</span><br><span class="line">print(arr4)</span><br><span class="line">print(arr5)</span><br><span class="line">print(arr6)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[[0 1 2 3]]</span><br><span class="line">[[4 5 6 7]]</span><br><span class="line">[[ 8 9 10 11]]</span><br></pre></td></tr></table></figure><h4 id="可能报错的地方"><a href="#可能报错的地方" class="headerlink" title="可能报错的地方"></a>可能报错的地方</h4><p>但是如果我们本身的数组,无法在我们的要求下进行切片会报错</p><p>例如：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">arr2,arr3,arr4=np.split(arr1,<span class="number">3</span>,axis=<span class="number">1</span>)<span class="comment">#按列方向分割,分成3份</span></span><br><span class="line">print(arr2)</span><br><span class="line">print(arr3)</span><br><span class="line">print(arr4)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line">---------------------------------------------------------------------------</span><br><span class="line">TypeError Traceback (most recent call last)</span><br><span class="line">C:\ProgramData\Anaconda3\lib\site-packages\numpy\lib\shape_base.py in split(ary, indices_or_sections, axis)</span><br><span class="line"> 842 try:</span><br><span class="line">--> 843 len(indices_or_sections)</span><br><span class="line"> 844 except TypeError:</span><br><span class="line"></span><br><span class="line">TypeError: object of type 'int' has no len()</span><br><span class="line"></span><br><span class="line">During handling of the above exception, another exception occurred:</span><br><span class="line"></span><br><span class="line">ValueError Traceback (most recent call last)</span><br><span class="line"><ipython-input-6-62c15d9e9149> in <module></span><br><span class="line">----> 1 arr2,arr3,arr4=np.split(arr1,3,axis=1)#按列方向分割,分成2份</span><br><span class="line"> 2 print(arr2)</span><br><span class="line"> 3 print(arr3)</span><br><span class="line"> 4 print(arr4)</span><br><span class="line"></span><br><span class="line">C:\ProgramData\Anaconda3\lib\site-packages\numpy\lib\shape_base.py in split(ary, indices_or_sections, axis)</span><br><span class="line"> 847 if N % sections:</span><br><span class="line"> 848 raise ValueError(</span><br><span class="line">--> 849 'array split does not result in an equal division')</span><br><span class="line"> 850 res = array_split(ary, indices_or_sections, axis)</span><br><span class="line"> 851 return res</span><br><span class="line"></span><br><span class="line">ValueError: array split does not result in an equal division</span><br></pre></td></tr></table></figure><p>这就是本身数组无法被切分为3份，所以报错了</p><p>而我们一定要把arr1切分为3份的话，我们可以使用np.array_split()函数</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">arr7,arr8,arr9=np.array_split(arr1,3,axis=1)#按列分割,分成3份,不等分</span><br><span class="line">print(arr7)</span><br><span class="line">print(arr8)</span><br><span class="line">print(arr9)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">[[0 1]</span><br><span class="line"> [4 5]</span><br><span class="line"> [8 9]]</span><br><span class="line">[[ 2]</span><br><span class="line"> [ 6]</span><br><span class="line"> [10]]</span><br><span class="line">[[ 3]</span><br><span class="line"> [ 7]</span><br><span class="line"> [11]]</span><br></pre></td></tr></table></figure><p>这样arr1会被强制切分为3份</p><h4 id="其他切分方法"><a href="#其他切分方法" class="headerlink" title="其他切分方法"></a>其他切分方法</h4><h4 id="vsplit和hsplit"><a href="#vsplit和hsplit" class="headerlink" title="vsplit和hsplit"></a>vsplit和hsplit</h4><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">arrv1,arrv2,arrv3=np.vsplit(arr1,<span class="number">3</span>)<span class="comment">#按行分割</span></span><br><span class="line">print(arrv1)</span><br><span class="line">print(arrv2)</span><br><span class="line">print(arrv3)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[[0 1 2 3]]</span><br><span class="line">[[4 5 6 7]]</span><br><span class="line">[[ 8 9 10 11]]</span><br></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">arrh1,arrh2=np.hsplit(arr1,<span class="number">2</span>)<span class="comment">#按列分割</span></span><br><span class="line">print(arrh1)</span><br><span class="line">print(arrh2)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">[[0 1]</span><br><span class="line"> [4 5]</span><br><span class="line"> [8 9]]</span><br><span class="line">[[ 2 3]</span><br><span class="line"> [ 6 7]</span><br><span class="line"> [10 11]]</span><br></pre></td></tr></table></figure><h3 id="合并"><a href="#合并" class="headerlink" title="合并"></a>合并</h3><p>先新建数组</p><h4 id="行合并"><a href="#行合并" class="headerlink" title="行合并"></a>行合并</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">import numpy as np</span><br><span class="line">arr1=np.array([1,2,3])</span><br><span class="line">arr2=np.array([4,5,6])</span><br><span class="line">arr3=np.vstack((arr1,arr2))#行合并</span><br><span class="line">print(arr3)</span><br><span class="line">print(arr3.shape)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[[1 2 3]</span><br><span class="line"> [4 5 6]]</span><br><span class="line">(2, 3)</span><br></pre></td></tr></table></figure><h4 id="列合并"><a href="#列合并" class="headerlink" title="列合并"></a>列合并</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">arr4=np.hstack((arr1,arr2))#列合并</span><br><span class="line">print(arr4)</span><br><span class="line">print(arr4.shape)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[1 2 3 4 5 6]</span><br><span class="line">(6,)</span><br></pre></td></tr></table></figure><p>也可以使用concatenate()函数合并</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">arr=np.concatenate((arr1,arr2,arr1))</span><br><span class="line">print(arr)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[1 2 3 4 5 6 1 2 3]</span><br></pre></td></tr></table></figure><p>但是注意使用concatenate()函数合并，array数组的维度要相同，array的形状要匹配</p><h2 id="numpy数组的迭代"><a href="#numpy数组的迭代" class="headerlink" title="numpy数组的迭代"></a>numpy数组的迭代</h2><p>新建一个数组</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">import numpy as np</span><br><span class="line">arr=np.random.randint(0,20,size=(3,4))</span><br><span class="line">print(arr)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[[ 2 3 4 5]</span><br><span class="line"> [ 6 7 8 9]</span><br><span class="line"> [10 11 12 13]]</span><br></pre></td></tr></table></figure><p>对于arr数组进行迭代行</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">for i in arr:#迭代行</span><br><span class="line"> print(i)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[2 3 4 5]</span><br><span class="line">[6 7 8 9]</span><br><span class="line">[10 11 12 13]</span><br></pre></td></tr></table></figure><p>对于arr数组进行迭代列</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">for i in arr.T:#使用转置，来迭代列</span><br><span class="line"> print(i)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[ 2 6 10]</span><br><span class="line">[ 3 7 11]</span><br><span class="line">[ 4 8 12]</span><br><span class="line">[ 5 9 13]</span><br></pre></td></tr></table></figure><p>对于arr数组一个一个元素进行迭代</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">for i in arr2.flat:#一个一个元素的迭代</span><br><span class="line"> print(i)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td></tr></table></figure><h2 id="numpy的浅拷贝和深拷贝"><a href="#numpy的浅拷贝和深拷贝" class="headerlink" title="numpy的浅拷贝和深拷贝"></a>numpy的浅拷贝和深拷贝</h2><p>直接上例子</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">import numpy as np</span><br><span class="line">a=np.array([1,2,3])</span><br><span class="line">b=arr1#arr1和arr2共享一块内存,浅拷贝</span><br><span class="line">b[0]=5</span><br><span class="line">print(a)</span><br><span class="line">print(b)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[5 2 3]</span><br><span class="line">[5 2 3]</span><br></pre></td></tr></table></figure><p>可以看到，改变a后，b的值也跟着变了，这是为什么呢？</p><p>实际上，变量a中并没有存储任何的值，它只是指向了一个内存地址，而这个地址里存储着array具体的内容，当把a赋值给b的时候，实际上是把a指向内存中某对象的链接赋给了b，也就是说，现在a和b都指向了同一个对象。</p><p>因此，在改变了内存中array的值后，而a与b都引用了该array对象，所以都一起发生了变化</p><p>这种将内存引用赋值给另一个变量的操作叫做浅拷贝</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">c=a.copy()#深拷贝</span><br><span class="line">c[0]=10</span><br><span class="line"></span><br><span class="line">print(a)</span><br><span class="line">print(c)</span><br></pre></td></tr></table></figure><p>输出结果如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[5 2 3]</span><br><span class="line">[10 2 3]</span><br><span class="line">a引用的对象的地址： 2130290221504</span><br><span class="line">c引用的对象的地址： 2130290057088</span><br></pre></td></tr></table></figure><p>深拷贝呢，其实就是在赋值的时候，不把同一个内存对象的引用赋值给另一个变量，令两个变量所指向的对象不一样，更改值的时候不相互影响，这种操作就是<strong>深拷贝</strong></p><p>copy()会创建a的一个副本，也就是创建一个一模一样的array对象，存储到内存的另一个地址中，然后将这个副本的地址赋值给c</p><p>目前我对于numpy库的学习到这里也就结束了,当后面学的更深入的时候,可能还会更新,本文写的可能不是很好,但还是感谢大家的阅读</p>]]></content>
<summary type="html">
<h1 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h1><p>numpy库是python中一个非常重要的库,它提供了一个多维数组（ndarray）数据类型，以及关于多维数组的操作，NumPy 已经成为其他大数据和机器学习模块的基础。 </p>
</summary>
<category term="python学习记录" scheme="www.nghqqa.cn/categories/python%E5%AD%A6%E4%B9%A0%E8%AE%B0%E5%BD%95/"/>
<category term="python" scheme="www.nghqqa.cn/tags/python/"/>
</entry>
<entry>
<title>使用的hadoop相关安装包</title>
<link href="www.nghqqa.cn/2019/08/14/%E4%BD%BF%E7%94%A8%E7%9A%84hadoop%E7%9B%B8%E5%85%B3%E5%AE%89%E8%A3%85%E5%8C%85/"/>
<id>www.nghqqa.cn/2019/08/14/使用的hadoop相关安装包/</id>
<published>2019-08-14T05:29:40.000Z</published>
<updated>2019-08-14T12:16:32.357Z</updated>
<content type="html"><![CDATA[<h1 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h1><p>这里放一些之前安装用到的安装包</p><p>链接：<a href="https://pan.baidu.com/s/1_qp_AKhRJNvzr6P63nJ9Rw" target="_blank" rel="noopener">https://pan.baidu.com/s/1_qp_AKhRJNvzr6P63nJ9Rw</a><br>提取码：7s9n </p><table><thead><tr><th>hadoop</th><th>spark</th><th>hive</th><th>scala</th></tr></thead><tbody><tr><td>2.7.2</td><td>spark-2.4.0</td><td>hive-1.1.0</td><td>scala-2.11.4</td></tr></tbody></table>]]></content>
<summary type="html">
<h1 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h1><p>这里放一些之前安装用到的安装包</p>
<p>链接：<a href="https://pan.baidu.com/s/1_qp_AKhRJN
</summary>
<category term="相关安装包" scheme="www.nghqqa.cn/categories/%E7%9B%B8%E5%85%B3%E5%AE%89%E8%A3%85%E5%8C%85/"/>
<category term="hadoop" scheme="www.nghqqa.cn/tags/hadoop/"/>
</entry>
<entry>
<title>scala的安装</title>
<link href="www.nghqqa.cn/2019/08/13/scala%E7%9A%84%E5%AE%89%E8%A3%85/"/>
<id>www.nghqqa.cn/2019/08/13/scala的安装/</id>
<published>2019-08-13T05:51:20.000Z</published>
<updated>2019-08-22T04:32:00.815Z</updated>
<content type="html"><![CDATA[<h1 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h1><p>之前写的spark的安装中忘记说明安装spark要先安装scala，因为spark的底层是使用scala脚本语言开发 <a id="more"></a></p><p>使用版本</p><table><thead><tr><th>scala-2.11.4</th></tr></thead><tbody><tr><td></td></tr></tbody></table><h2 id="安装spark"><a href="#安装spark" class="headerlink" title="安装spark"></a>安装spark</h2><h3 id="上传解压spark"><a href="#上传解压spark" class="headerlink" title="上传解压spark"></a>上传解压spark</h3><p>spark的压缩包上传到我们的/opt/software上，解压到/opt/module/</p><p>代码如下</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">tar -zxf scala-2.11.4.tgz -C /opt/module/</span><br></pre></td></tr></table></figure><h3 id="配置环境变量"><a href="#配置环境变量" class="headerlink" title="配置环境变量"></a>配置环境变量</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">vim /etc/profile</span><br></pre></td></tr></table></figure><p>在末尾添加</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">#scala</span><br><span class="line">export SCALA_HOME=/opt/module/scala-2.11.4</span><br><span class="line">export PATH=$PATH:$SCALA_HOME/bin</span><br></pre></td></tr></table></figure><p>记得保存使其生效</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">source /etc/profile</span><br></pre></td></tr></table></figure><h3 id="验证"><a href="#验证" class="headerlink" title="验证"></a>验证</h3><p>输入</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">scala -version</span><br></pre></td></tr></table></figure><p>出现</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">Scala code runner version 2.11.4 -- Copyright 2002-2013, LAMP/EPFL</span><br></pre></td></tr></table></figure><p>则代表安装完成</p><p>之后在其他两个节点中重复上述步骤！</p><p>安装scala到这里也就完成，一定要安装完scala后去安装spark，谢谢大家的阅读。</p>]]></content>
<summary type="html">
<h1 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h1><p>之前写的spark的安装中忘记说明安装spark要先安装scala，因为spark的底层是使用scala脚本语言开发</p>
</summary>
<category term="搭建自己的hadoop学习集群" scheme="www.nghqqa.cn/categories/%E6%90%AD%E5%BB%BA%E8%87%AA%E5%B7%B1%E7%9A%84hadoop%E5%AD%A6%E4%B9%A0%E9%9B%86%E7%BE%A4/"/>
<category term="hadoop" scheme="www.nghqqa.cn/tags/hadoop/"/>
</entry>
<entry>
<title>spark安装</title>
<link href="www.nghqqa.cn/2019/08/13/spark%E7%9A%84%E5%AE%89%E8%A3%85/"/>
<id>www.nghqqa.cn/2019/08/13/spark的安装/</id>
<published>2019-08-13T03:40:20.000Z</published>
<updated>2019-08-22T04:32:05.864Z</updated>
<content type="html"><![CDATA[<h1 id="前言——spark介绍"><a href="#前言——spark介绍" class="headerlink" title="前言——spark介绍"></a>前言——spark介绍</h1><p>Apache Spark 是专为大规模数据处理而设计的快速通用的计算引擎是类似于Hadoop MapReduce的通用并行框架。</p> <a id="more"></a><p>Spark拥有Hadoop MapReduce所具有的优点，但不同于MapReduce的是——Job中间输出结果可以保存在内存中，从而不再需要读写HDFS，因此Spark能更好地适用于数据挖掘与机器学习等需要迭代的MapReduce的算法。Spark实际上是对Hadoop的一种补充，可以很好的在Hadoop 文件系统中并行运行。</p><h2 id="安装spark"><a href="#安装spark" class="headerlink" title="安装spark"></a>安装spark</h2><table><thead><tr><th>安装版本</th></tr></thead><tbody><tr><td>spark-2.4.0</td></tr></tbody></table><h3 id="上传解压spark"><a href="#上传解压spark" class="headerlink" title="上传解压spark"></a>上传解压spark</h3><p>我们将spark上传到/opt/software下，之后将其解压到/opt/module/</p><p>解压代码如下</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">tar -zxf spark-2.4.0-bin-hadoop2.7.tgz -C /opt/module/</span><br></pre></td></tr></table></figure><p>修改名称</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">mv spark-2.4.0-bin-hadoop2.7 spark-2.4.0</span><br></pre></td></tr></table></figure><h3 id="配置环境变量"><a href="#配置环境变量" class="headerlink" title="配置环境变量"></a>配置环境变量</h3><p>在/etc/profile文件的最后添加</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">export SPARK_HOME=/opt/module/spark-2.4.0</span><br><span class="line">export PATH=$PATH:$SPARK_HOME/bin</span><br></pre></td></tr></table></figure><p>记得保存</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">source /etc/profile</span><br></pre></td></tr></table></figure><h2 id="修改spark-env-sh文件"><a href="#修改spark-env-sh文件" class="headerlink" title="修改spark-env.sh文件"></a>修改spark-env.sh<strong>文件</strong></h2><p>进入spark文件夹下的conf文件夹，修改文件名</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">mv spark-env.sh.template spark-env.sh</span><br></pre></td></tr></table></figure><p>改完之后</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">vim spark-env.sh</span><br></pre></td></tr></table></figure><p>在spark-env.sh文件的末尾添加</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">export JAVA_HOME=/opt/module/jdk1.8.0_191</span><br><span class="line">export SCALA_HOME=/opt/module/scala-2.11.4</span><br><span class="line">export HADOOP_HOME=/opt/module/hadoop-2.7.2</span><br><span class="line">export HADOOP_CONF_DIR=/opt/module/hadoop-2.7.2/etc/hadoop</span><br></pre></td></tr></table></figure><h2 id="修改-slaves"><a href="#修改-slaves" class="headerlink" title="修改 slaves"></a>修改 slaves</h2><p> 修改 slaves 文件：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">mv slaves.template slaves</span><br></pre></td></tr></table></figure><p> 打开 slaves 文件：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">vim slaves</span><br></pre></td></tr></table></figure><p>添加以下内容：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">hadoop102</span><br><span class="line">hadoop103</span><br><span class="line">hadoop104</span><br></pre></td></tr></table></figure><p>完成后，我们将spark文件夹传给slave1和slave2</p><p>输入（传输的时候要在spark-2.4.0的上一目录下传输）</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">scp -r spark-2.4.0 root@salve1:/opt/module/</span><br><span class="line">scp -r spark-2.4.0 root@salve2:/opt/module/</span><br></pre></td></tr></table></figure><p>传输完成，在slave1和slave2中配置环境变量</p><h2 id="启动spark"><a href="#启动spark" class="headerlink" title="启动spark"></a>启动spark</h2><p>在spark-2.4.0目录下输入</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sbin/start-all.sh</span><br></pre></td></tr></table></figure><p>启动完成后，如果可以访问</p><p><a href="http://192.168.128.172:8080/" target="_blank" rel="noopener">http://192.168.128.172:8080/</a></p><p>如图</p><p><img src="http://qqa.nghqqa.cn/blog/20190813/XVXaymkIosSi.png?imageslim" alt="mark"></p><p>则为成功</p><p>到这里spark的安装也就完成了，谢谢大家的阅读。</p>]]></content>
<summary type="html">
<h1 id="前言——spark介绍"><a href="#前言——spark介绍" class="headerlink" title="前言——spark介绍"></a>前言——spark介绍</h1><p>Apache Spark 是专为大规模数据处理而设计的快速通用的计算引擎是类似于Hadoop MapReduce的通用并行框架。</p>
</summary>
<category term="搭建自己的hadoop学习集群" scheme="www.nghqqa.cn/categories/%E6%90%AD%E5%BB%BA%E8%87%AA%E5%B7%B1%E7%9A%84hadoop%E5%AD%A6%E4%B9%A0%E9%9B%86%E7%BE%A4/"/>
<category term="hadoop" scheme="www.nghqqa.cn/tags/hadoop/"/>
</entry>
<entry>
<title>hive的安装</title>
<link href="www.nghqqa.cn/2019/08/13/hive%E7%9A%84%E5%AE%89%E8%A3%85/"/>
<id>www.nghqqa.cn/2019/08/13/hive的安装/</id>
<published>2019-08-13T00:17:40.000Z</published>
<updated>2019-08-22T04:31:34.668Z</updated>
<content type="html"><![CDATA[<h1 id="前言——hive介绍"><a href="#前言——hive介绍" class="headerlink" title="前言——hive介绍"></a>前言——hive介绍</h1><p>Hive是基于Hadoop的一个数据仓库工具，可以将结构化的数据文件映射为一张数据库表，并提供简单的sql查询功能，可以将sql语句转换为MapReduce任务进行运行。 其优点是学习成本低，可以通过和SQL类似的HiveQL语言快速实现简单的MapReduce统计,不必开发专门的MapReduce应用，十分适合数据仓库的统计分析。同时，这个语言也允许熟悉 MapReduce 开发者的开发自定义的 mapper 和 reducer 来处理内建的 mapper 和 reducer 无法完成的复杂的分析工作。 </p><a id="more"></a><p>Hive 没有专门的数据格式。所有Hive 的数据都存储在Hadoop兼容的文件系统（例如HDFS）中。Hive 在加载数据过程中不会对数据进行任何的修改，只是将数据移动到HDFS中Hive 设定的目录下，因此，Hive 不支持对数据的改写和添加，所有的数据都是在加载的时候确定的。</p><p>附一张hadoop的生态圈图</p><p><img src="http://qqa.nghqqa.cn/blog/20190813/7gwkKD6NYb3T.png?imageslim" alt="mark"></p><h2 id="安装hive"><a href="#安装hive" class="headerlink" title="安装hive"></a>安装hive</h2><p>hive的安装基于之前的搭建的hadoop完全分布式集群，只需要安装在hadoop102上就行</p><table><thead><tr><th>安装hive版本</th></tr></thead><tbody><tr><td>hive-1.1.0</td></tr></tbody></table><h3 id="上传解压hive"><a href="#上传解压hive" class="headerlink" title="上传解压hive"></a>上传解压hive</h3><p>我们将hive上传到/opt/software下，之后将其解压到/opt/module/</p><p>解压代码如下</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">tar -zxf apache-hive-1.1.0-bin.tar.gz -C /opt/module/</span><br></pre></td></tr></table></figure><p>解压完成后我们修改hive的文件名</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">mv apache-hive-1.1.0-bin.tar.gz hive-1.1.0</span><br></pre></td></tr></table></figure><p>修改文件名是为了让我们在配置环境变量时更加的方便</p><h3 id="配置环境变量"><a href="#配置环境变量" class="headerlink" title="配置环境变量"></a>配置环境变量</h3><p>在/etc/profile文件的最后添加</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">export HIVE_HOME=/opt/module/hive-1.1.0</span><br><span class="line">export PATH=$PATH:$HIVE_HOME/sbin</span><br></pre></td></tr></table></figure><p>配置完成后，记得保存使其生效</p><p>输入</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">source /etc/profile</span><br></pre></td></tr></table></figure><h3 id="安装mysql"><a href="#安装mysql" class="headerlink" title="安装mysql"></a>安装mysql</h3><p>hive它有自己的内置数据库derby，但是hive 使用derby 数据库存在不支持多个连接的问题，所以我们一般会使用mysql来代替hive的元数据库</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop102 ~]# cd /opt/module/src/</span><br><span class="line">[root@hadoop102 src]# wget http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm</span><br><span class="line">[root@hadoop102 src]# rpm -ivh mysql-community-release-el7-5.noarch.rpm</span><br><span class="line">[root@hadoop102 src]# yum install mysql-community-server</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"># 这里时间较长，耐心等待...</span><br><span class="line"></span><br><span class="line"># 安装完成后，重启服务</span><br><span class="line">[root@hadoop102 src]# service mysqld restart</span><br><span class="line">[root@hadoop102 src]# mysql</span><br><span class="line">Welcome to the MySQL monitor. Commands end with ; or \g.</span><br><span class="line">Your MySQL connection id is 3</span><br><span class="line">Server version: 5.6.42 MySQL Community Server (GPL)</span><br><span class="line">Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.</span><br><span class="line">Oracle is a registered trademark of Oracle Corporation and/or its</span><br><span class="line">affiliates. Other names may be trademarks of their respective owners.</span><br><span class="line">Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.</span><br><span class="line">mysql></span><br><span class="line"></span><br><span class="line"># mysql安装成功</span><br></pre></td></tr></table></figure><p>同时我们还需要去网上下载mysql的驱动包（mysql-connector-java.jar）把这个驱动包放置在hive目录下的lib目录下。</p><h3 id="修改hive-site-xml"><a href="#修改hive-site-xml" class="headerlink" title="修改hive-site.xml"></a>修改hive-site.xml</h3><p>hive的配置文件放置在/opt/module/hive-1.1.0/conf下</p><p>配置hive-site.xml(conf中可能没有这个文件，我们使用vim打开时，没有的话，vim会帮我们自动创建)</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">vim hive-site.xml</span><br></pre></td></tr></table></figure><p>在文件中添加</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br></pre></td><td class="code"><pre><span class="line"><?xml version="1.0"?></span><br><span class="line"><?xml-stylesheet type="text/xsl" href="configuration.xsl"?></span><br><span class="line"></span><br><span class="line"><configuration></span><br><span class="line"></span><br><span class="line"><property></span><br><span class="line"><name>hive.metastore.local</name></span><br><span class="line"><value>true</value></span><br><span class="line"></property></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><property></span><br><span class="line"><name>javax.jdo.option.ConnectionURL</name></span><br><span class="line"><value>jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true</value></span><br><span class="line"><description>JDBC connect string for a JDBC metastore</description></span><br><span class="line"></property></span><br><span class="line"></span><br><span class="line"><property></span><br><span class="line"><name>javax.jdo.option.ConnectionDriverName</name></span><br><span class="line"><value>com.mysql.jdbc.Driver</value></span><br><span class="line"><description>Driver class name for a JDBC metastore</description></span><br><span class="line"></property></span><br><span class="line"></span><br><span class="line"><property></span><br><span class="line"><name>javax.jdo.option.ConnectionUserName</name></span><br><span class="line"><value>root</value></span><br><span class="line"><description>username to use against metastore database</description></span><br><span class="line"></property></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><property></span><br><span class="line"><name>javax.jdo.option.ConnectionPassword</name></span><br><span class="line"><value>hivepwd</value></span><br><span class="line"></property></span><br><span class="line"></span><br><span class="line"></configuration></span><br></pre></td></tr></table></figure><h3 id="修改hive-env-sh"><a href="#修改hive-env-sh" class="headerlink" title="修改hive-env.sh"></a>修改hive-env.sh</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop102 conf]# mv hive-env.sh.template hive-env.sh</span><br></pre></td></tr></table></figure><p>在文件最后添加</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">export JAVA_HOME=/opt/module/jdk1.8.0_191</span><br><span class="line">export HADOOP_HOME=/opt/module/hadoop-2.7.2</span><br></pre></td></tr></table></figure><h3 id="配置mysql"><a href="#配置mysql" class="headerlink" title="配置mysql"></a>配置mysql</h3><p> 创建数据库 hive ，用来保存 Hive 元数据：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">create database hive;</span><br></pre></td></tr></table></figure><p>同时使 root 用户可以操作数据库 hive 中的所有表：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">GRANT all ON hive.* TO root@'Hadoop102' IDENTIFIED BY 'hivepwd';</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">flush privileges;</span><br></pre></td></tr></table></figure><p>这样<br>Hive 的元数据库就安装完成。</p><h2 id="测试hive安装是否成功"><a href="#测试hive安装是否成功" class="headerlink" title="测试hive安装是否成功"></a>测试hive安装是否成功</h2><p>启动Hadoop与mysql</p><p>输入hive</p><p>进入hive，出现命令行就说明之前搭建是成功的</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@Hadoop102 ]# hive</span><br><span class="line">hive></span><br></pre></td></tr></table></figure><h3 id="测试"><a href="#测试" class="headerlink" title="测试"></a>测试</h3><p>进入<br>Hive 命令行 执行命令 创建一个名为 test 的表 查询该表的记录数：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">create table test(id int);</span><br><span class="line">select count(*) from test;</span><br></pre></td></tr></table></figure><p>如果查询结果为0，则成功</p><p><img src="http://qqa.nghqqa.cn/blog/20190813/8jcn3tWNMSgl.png?imageslim" alt="mark"></p><p>到这里，hive的安装也就完成了，谢谢大家的阅读。</p>]]></content>
<summary type="html">
<h1 id="前言——hive介绍"><a href="#前言——hive介绍" class="headerlink" title="前言——hive介绍"></a>前言——hive介绍</h1><p>Hive是基于Hadoop的一个数据仓库工具，可以将结构化的数据文件映射为一张数据库表，并提供简单的sql查询功能，可以将sql语句转换为MapReduce任务进行运行。 其优点是学习成本低，可以通过和SQL类似的HiveQL语言快速实现简单的MapReduce统计,不必开发专门的MapReduce应用，十分适合数据仓库的统计分析。同时，这个语言也允许熟悉 MapReduce 开发者的开发自定义的 mapper 和 reducer 来处理内建的 mapper 和 reducer 无法完成的复杂的分析工作。 </p>
</summary>
<category term="搭建自己的hadoop学习集群" scheme="www.nghqqa.cn/categories/%E6%90%AD%E5%BB%BA%E8%87%AA%E5%B7%B1%E7%9A%84hadoop%E5%AD%A6%E4%B9%A0%E9%9B%86%E7%BE%A4/"/>
<category term="hadoop" scheme="www.nghqqa.cn/tags/hadoop/"/>
</entry>
<entry>
<title>hadoop完全分布式安装</title>
<link href="www.nghqqa.cn/2019/08/10/hadoop%E5%AE%8C%E5%85%A8%E5%88%86%E5%B8%83%E5%BC%8F%E5%AE%89%E8%A3%85/"/>
<id>www.nghqqa.cn/2019/08/10/hadoop完全分布式安装/</id>
<published>2019-08-10T13:43:42.000Z</published>
<updated>2019-08-22T04:31:17.940Z</updated>
<content type="html"><![CDATA[<h1 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h1><p>我们之前搭建了hadoop的伪分布式的集群，伪分布式的集群对于学习一些基础的hadoop操作是足够的，但是当你涉及到更复杂的操作时，伪分布式就不够了，完全分布式的性能比伪分布的要强，而且完全分布式可操作性也更高，与此同时完全分布式对于电脑的配置也要求更高，完全分布式一般需要3台虚拟机来完成,下面我们开始搭建自己的完全分布式。 </p><a id="more"></a><h2 id="使用环境"><a href="#使用环境" class="headerlink" title="使用环境"></a>使用环境</h2><p>下面是本次搭建使用的环境</p><table><thead><tr><th align="left">操作环境</th><th>主机名</th><th>IP地址</th><th>jdk</th><th align="center">hadoop版本</th><th></th></tr></thead><tbody><tr><td align="left">centos6.8</td><td>hadoop102</td><td>192.168.128.172</td><td>jdk1.8.0_191</td><td align="center">hadoop-2.7.2</td><td></td></tr><tr><td align="left"></td><td>hadoop103</td><td>192.168.128.173</td><td></td><td align="center"></td><td></td></tr><tr><td align="left"></td><td>hadoop104</td><td>192.168.128.174</td><td></td><td align="center"></td><td></td></tr></tbody></table><p> </p><p>本文中使用的各种包，后续我会进行上传，以方便读者的使用</p><h2 id="搭建步骤详解"><a href="#搭建步骤详解" class="headerlink" title="搭建步骤详解"></a>搭建步骤详解</h2><h3 id="1-修改各节点的网络配置"><a href="#1-修改各节点的网络配置" class="headerlink" title="1.修改各节点的网络配置"></a>1.修改各节点的网络配置</h3><p>在虚拟机中输入</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">vim /etc/sysconfig/network-scripts/ifcfg-eth0</span><br></pre></td></tr></table></figure><p>可以进入虚拟机的网卡配置</p><p>我们需要修改虚拟机的网卡默认配置，将我们的虚拟机的网卡配置设置为静态ip</p><p>ip地址 根据 VMware 虚拟网络进行相关配置 如图</p><p><img src="http://qqa.nghqqa.cn/blog/20190811/lihMS1cmJ5ds.png?imageslim" alt="mark"></p><p><img src="http://qqa.nghqqa.cn/blog/20190811/628Xq2EJFY3r.png?imageslim" alt="mark"></p><p>在虚拟机里修改配置可以如图所示</p><p><img src="http://qqa.nghqqa.cn/blog/20190811/6mobrzRqbbHQ.png?imageslim" alt="mark"></p><p>修改完输入</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">service network restart</span><br></pre></td></tr></table></figure><p>重启网络服务</p><p>对其他两个hadoop节点也同样做上述操作，只不过在IPADDR值不一样，分别填其节点对应的ip</p><h3 id="2-修改节点主机名，并且添加各节点映射"><a href="#2-修改节点主机名，并且添加各节点映射" class="headerlink" title="2.修改节点主机名，并且添加各节点映射"></a>2.修改节点主机名，并且添加各节点映射</h3><p>在命令行中输入</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">vim /etc//sysconfig/network</span><br></pre></td></tr></table></figure><p>进入文件中修改hostname名称，如图所示</p><p><img src="http://qqa.nghqqa.cn/blog/20190811/vzbBoY0iVkob.png?imageslim" alt="mark"></p><p>在其他两个子节点的hostname处分别填hadoop103和hadoop104</p><p>添加节点映射，输入</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">vim /etc/hosts</span><br></pre></td></tr></table></figure><p>添加节点映射为</p><p><img src="http://qqa.nghqqa.cn/blog/20190811/cbqc8daw1Izy.png?imageslim" alt="mark"></p><p>我这里是多写了一些，对于这次搭建我们只需要添加hadoop102，hadoop103，hadoop104的节点映射即可</p><h2 id="3-关闭防火墙"><a href="#3-关闭防火墙" class="headerlink" title="3.关闭防火墙"></a>3.关闭防火墙</h2><p>我们只有关闭防火墙后才能在三台机器之间互相通信</p><p>所以关闭防火墙是很有必要的</p><p>我们可以使用这条命令来检查我们虚拟机开机时的防火墙状态</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">chkconfig iptables --list</span><br></pre></td></tr></table></figure><p>如果是已经关闭应该会如下图所示</p><p><img src="http://qqa.nghqqa.cn/blog/20190811/NNdzq6Nt2Prm.png?imageslim" alt="mark"></p><p>如果没有关闭我们可以使用这两条命令来关闭我们的防火墙</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">service iptables stop</span><br></pre></td></tr></table></figure><p>这条命令是在本次虚拟机开启过程中关闭防火墙，也就是一次性关闭</p><p>我们还需要这条命令</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">chkconfig iptables off</span><br></pre></td></tr></table></figure><p>禁止防火墙关机自启动，这样防火墙就是是关闭了</p><p>当hadoop102关闭防火墙后，对于hadoop103与hadoop104也要做同样的操作</p><p>在防火墙关闭完成后，输入</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">reboot</span><br></pre></td></tr></table></figure><p>重启虚拟机，检查防火墙是否已经关闭</p><h2 id="4-配置节点间ssh免密登陆"><a href="#4-配置节点间ssh免密登陆" class="headerlink" title="4.配置节点间ssh免密登陆"></a>4.配置节点间ssh免密登陆</h2><p>在hadoop102上输入</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">ssh-keygen -t rsa</span><br></pre></td></tr></table></figure><p>一直按回车</p><p>完成后在保证三台虚拟机开启且完成之前所有配置的情况下输入</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop102 ~]# ssh-copy-id hadoop102</span><br><span class="line">[root@hadoop102 ~]# ssh-copy-id hadoop103</span><br><span class="line">[root@hadoop102 ~]# ssh-copy-id hadoop104</span><br></pre></td></tr></table></figure><p>在hadoop102上完成后，在其他两个节点上重复上述操作</p><p>验证ssh免密登录是否成功</p><p><img src="http://qqa.nghqqa.cn/blog/20190811/djEjODwXsuF8.png?imageslim" alt="mark"></p><p>这里可以看到我们可以自己使用ssh转到hadoop103这台机器上</p><h2 id="5-安装java和hadoop"><a href="#5-安装java和hadoop" class="headerlink" title="5.安装java和hadoop"></a>5.安装java和hadoop</h2><p>我们先使用xftp将hadoop和java的压缩包上传到我们新建的/opt/software上同时新建一个module文件夹放置解压后的hadoop和java，新建文件夹代码如下</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop102 ~]# cd /opt/</span><br><span class="line">[root@hadoop102 ~]# mkdir software</span><br><span class="line">[root@hadoop102 ~]# mkdir module</span><br></pre></td></tr></table></figure><p>上传完成之后我们需要解压java和hadoop到/opt/module下，以便未来的管理</p><p>解压代码如下</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">tar -zxf jdk-8u191-linux-x64.tar.gz -C /opt/module/</span><br><span class="line">tar -zxf hadoop-2.7.2.tar.gz -C /opt/module/</span><br></pre></td></tr></table></figure><p>解压完成后在/opt/module下应该是这样的，如图所示</p><p><img src="http://qqa.nghqqa.cn/blog/20190811/tVfrEHmhKgeh.png?imageslim" alt="mark"></p><p>之后我们就需要配置环境变量</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">vi /etc/profile</span><br></pre></td></tr></table></figure><p>在最后添加</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">export JAVA_HOME=/opt/module/jdk1.8.0_191</span><br><span class="line">export PATH=$PATH:$JAVA_HOME/bin</span><br><span class="line"></span><br><span class="line">export HADOOP_HOME=/opt/module/hadoop-2.7.2</span><br><span class="line">export PATH=$PATH:$HADOOP_HOME/bin</span><br><span class="line">export PATH=$PATH:$HADOOP_HOME/sbin</span><br></pre></td></tr></table></figure><p>退出后，输入</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">source /etc/profile</span><br></pre></td></tr></table></figure><p>使其生效</p><p>验证java和hadoop环境变量是否配置完成</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop103 module]# java -version</span><br><span class="line">java version "1.8.0_191"</span><br><span class="line">Java(TM) SE Runtime Environment (build 1.8.0_191-b12)</span><br><span class="line">Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)</span><br><span class="line">[root@hadoop103 module]# hadoop version</span><br><span class="line">Hadoop 2.7.2</span><br><span class="line">Subversion Unknown -r Unknown</span><br><span class="line">Compiled by root on 2017-05-22T10:49Z</span><br><span class="line">Compiled with protoc 2.5.0</span><br><span class="line">From source with checksum d0fda26633fa762bff87ec759ebe689c</span><br><span class="line">This command was run using /opt/module/hadoop-2.7.2/share/hadoop/common/hadoop-common-2.7.2.jar</span><br><span class="line">[root@hadoop103 module]#</span><br></pre></td></tr></table></figure><p>环境变量已经配置完成， 在其他两个节点上重复上述操作</p><h2 id="6-配置hadoop中的文件"><a href="#6-配置hadoop中的文件" class="headerlink" title="6.配置hadoop中的文件"></a>6.配置hadoop中的文件</h2><h3 id="6-1配置文件core-site-xml"><a href="#6-1配置文件core-site-xml" class="headerlink" title="6.1配置文件core-site.xml"></a>6.1配置文件core-site.xml</h3><p><strong>core-site.xml文件包含了NameNode主机地址，监听端口等信息，对于这个伪分布式模型来说，我的主机地址为hadoo101，NameNode默认使用的端口为8020。</strong></p><p>修改core-site.xml</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><configuration></span><br><span class="line"><!-- 指定HDFS中NameNode的地址 --></span><br><span class="line"><property></span><br><span class="line"> <name>fs.defaultFS</name></span><br><span class="line"> <value>hdfs://hadoop101:8020</value></span><br><span class="line"></property></span><br><span class="line"> <!-- 指定hadoop运行时产生文件的存储目录 --></span><br><span class="line"> <property></span><br><span class="line"> <name>hadoop.tmp.dir</name></span><br><span class="line"> <value>/opt/module/hadoop-2.7.2/data/tmp</value></span><br><span class="line"> </property></span><br><span class="line"></configuration></span><br></pre></td></tr></table></figure><h3 id="6-2配置文件hdfs-site-xml"><a href="#6-2配置文件hdfs-site-xml" class="headerlink" title="6.2配置文件hdfs-site.xml"></a>6.2配置文件hdfs-site.xml</h3><p><strong>hdfs-site.xml用于配置/HDFS的相关属性，例如数据块的副本参数，数据块的副本对于完全分布式来说应该为3</strong></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><configuration></span><br><span class="line"></span><br><span class="line"><property></span><br><span class="line"><name>dfs.replication</name></span><br><span class="line"><value>3</value></span><br><span class="line"></property></span><br><span class="line"></span><br><span class="line"><property></span><br><span class="line"><name>dfs.namenode.secondary.http-address</name></span><br><span class="line"> <value>hadoop104:50090</value></span><br><span class="line"></property></span><br><span class="line"></span><br><span class="line"></configuration></span><br></pre></td></tr></table></figure><h3 id="6-3配置文件slaves"><a href="#6-3配置文件slaves" class="headerlink" title="6.3配置文件slaves"></a>6.3配置文件slaves</h3><p><strong>slaves文件里面记录的是集群里所有DataNode的主机名</strong></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop102 ~]#vim slaves</span><br><span class="line">hadoop102</span><br><span class="line">hadoop103</span><br><span class="line">hadoop104</span><br></pre></td></tr></table></figure><h3 id="6-4配置文件yarn-site-xml"><a href="#6-4配置文件yarn-site-xml" class="headerlink" title="6.4配置文件yarn-site.xml"></a>6.4配置文件yarn-site.xml</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><configuration></span><br><span class="line"><!-- Site specific YARN configuration properties --></span><br><span class="line"><!-- reducer获取数据的方式 --></span><br><span class="line"> <property></span><br><span class="line"> <name>yarn.nodemanager.aux-services</name></span><br><span class="line"> <value>mapreduce_shuffle</value></span><br><span class="line"> </property></span><br><span class="line"></span><br><span class="line"><!-- 指定YARN的ResourceManager的地址 --></span><br><span class="line"><property></span><br><span class="line"><name>yarn.resourcemanager.hostname</name></span><br><span class="line"><value>hadoop103</value></span><br><span class="line"></property></span><br><span class="line"></configuration></span><br></pre></td></tr></table></figure><h3 id="6-5配置文件yarn-env-sh"><a href="#6-5配置文件yarn-env-sh" class="headerlink" title="6.5配置文件yarn-env.sh"></a>6.5配置文件yarn-env.sh</h3><p>在其中修改java的路径</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">export JAVA_HOME=/opt/module/jdk1.8.0_191</span><br></pre></td></tr></table></figure><h3 id="6-6配置hadoop-env-sh"><a href="#6-6配置hadoop-env-sh" class="headerlink" title="6.6配置hadoop-env.sh"></a>6.6配置hadoop-env.sh</h3><p><strong>hadoop-env.sh 由于Hadoop是java进程，所以需要添加jdk</strong></p><p>修改hadoop-env.sh</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">export JAVA_HOME=/opt/module/jdk1.8.0_191</span><br></pre></td></tr></table></figure><h3 id="6-7配置文件mapred-site-xml"><a href="#6-7配置文件mapred-site-xml" class="headerlink" title="6.7配置文件mapred-site.xml"></a>6.7配置文件mapred-site.<strong>xml</strong></h3><p>先改名，因为本身是没有mapred-site.xml这个文件的</p><p>输入</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">mv mapred-site.xml.template mapred-site.xml</span><br></pre></td></tr></table></figure><p>改名完成后</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop102 ~]#vim mapred-site.xml</span><br><span class="line"></span><br><span class="line"><configuration></span><br><span class="line"><!-- 指定mr运行在yarn上 --></span><br><span class="line"> <property></span><br><span class="line"> <name>mapreduce.framework.name</name></span><br><span class="line"> <value>yarn</value></span><br><span class="line"> </property></span><br><span class="line"></configuration></span><br></pre></td></tr></table></figure><p>这些配置文件改好后，返回/opt/module目录下</p><p>把hadoop102下修改的文件分发到hadoop103和hadoop104下</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop102 module]# scp -r hadoop root@hadoop103:/opt/module/</span><br><span class="line">[root@hadoop102 module]# scp -r hadoop root@hadoop104:/opt/module/</span><br></pre></td></tr></table></figure><h2 id="7-测试集群"><a href="#7-测试集群" class="headerlink" title="7.测试集群"></a>7.测试集群</h2><p>在完成配置文件等一系列工作后，我们要开始测试集群了</p><p>先格式化</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop2 hadoop-2.7.2]# bin/hdfs namenode –format</span><br></pre></td></tr></table></figure><p>之后启动hdfs</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">[root@hadoop2 hadoop-2.7.2]# sbin/start-dfs.sh</span><br><span class="line"></span><br><span class="line">[root@hadoop2 hadoop-2.7.2]# jps</span><br><span class="line">4166 NameNode</span><br><span class="line">4482 Jps</span><br><span class="line">4263 DataNode</span><br><span class="line"></span><br><span class="line">[root@hadoop3 桌面]# jps</span><br><span class="line">3218 DataNode</span><br><span class="line">3288 Jps</span><br><span class="line"></span><br><span class="line">[root@hadoop4 桌面]# jps</span><br><span class="line">3221 DataNode</span><br><span class="line">3283 SecondaryNameNode</span><br><span class="line">3364 Jps</span><br></pre></td></tr></table></figure><p>如果是这样这表示启动hdfs成功</p><p>下面启动yarn</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sbin/start-yarn.sh</span><br></pre></td></tr></table></figure><p>启动完成后</p><p>在浏览器上访问可视化页面：<a href="http://192.168.128.172:50070" target="_blank" rel="noopener">http://192.168.128.172:50070</a></p><p><img src="http://qqa.nghqqa.cn/blog/20190811/CFi7BSr2G8Ws.png?imageslim" alt="mark"></p><p>到此为止，hadoop配置就结束了，谢谢大家的阅读。</p>]]></content>
<summary type="html">
<h1 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h1><p>我们之前搭建了hadoop的伪分布式的集群，伪分布式的集群对于学习一些基础的hadoop操作是足够的，但是当你涉及到更复杂的操作时，伪分布式就不够了，完全分布式的性能比伪分布的要强，而且完全分布式可操作性也更高，与此同时完全分布式对于电脑的配置也要求更高，完全分布式一般需要3台虚拟机来完成,下面我们开始搭建自己的完全分布式。 </p>
</summary>
<category term="搭建自己的hadoop学习集群" scheme="www.nghqqa.cn/categories/%E6%90%AD%E5%BB%BA%E8%87%AA%E5%B7%B1%E7%9A%84hadoop%E5%AD%A6%E4%B9%A0%E9%9B%86%E7%BE%A4/"/>
<category term="hadoop" scheme="www.nghqqa.cn/tags/hadoop/"/>
</entry>
<entry>
<title>Linux学习</title>
<link href="www.nghqqa.cn/2019/08/06/Linux%E5%AD%A6%E4%B9%A0/"/>
<id>www.nghqqa.cn/2019/08/06/Linux学习/</id>
<published>2019-08-06T11:28:07.000Z</published>
<updated>2019-08-22T04:32:08.582Z</updated>
<content type="html"><![CDATA[<h1 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h1><p>对于之前写的hadoop伪分布式的安装文章，可能对一些没有Linux基础的读者来说有一些的难度，所以建议大家先了解Linux的一些基础知识在来看之前的文章，可能就会觉得很简单了</p><h2 id="学习建议"><a href="#学习建议" class="headerlink" title="学习建议"></a>学习建议</h2><p>我个人是比较推荐大家可以去b站看看韩顺平老师的Linux教程，可以在b站直接找到，这里放出b站链接</p><p><a href="https://www.bilibili.com/video/av21303002?from=search&seid=3872508779266125537" target="_blank" rel="noopener">https://www.bilibili.com/video/av21303002?from=search&seid=3872508779266125537</a></p><p>视频是18年上传的，但是知识是不会过时的，当然如果大家不想那么系统的了解Linux的话，可以去看看这个老哥的博客文章，这里放出链接</p><p><a href="https://blog.csdn.net/weixin_41710054/article/details/89081599#22_vivim_19" target="_blank" rel="noopener">https://blog.csdn.net/weixin_41710054/article/details/89081599#22_vivim_19</a></p><p>这个文章写的比较的细，基本不知道的命令或者是快捷键都可以去文章中看看，自己看文章学习，可以比看视频省下不少时间，里面的命令可以基本满足正常操作Linux系统的要求。</p><p>好的，本次Linux的学习建议就到这里了，希望大家生活愉快。</p>]]></content>
<summary type="html">
<h1 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h1><p>对于之前写的hadoop伪分布式的安装文章，可能对一些没有Linux基础的读者来说有一些的难度，所以建议大家先了解Linux的一些基础知识在
</summary>
<category term="自己学习Linux" scheme="www.nghqqa.cn/categories/%E8%87%AA%E5%B7%B1%E5%AD%A6%E4%B9%A0Linux/"/>
<category term="Linux" scheme="www.nghqqa.cn/tags/Linux/"/>
</entry>
<entry>
<title>hadoop伪分布式安装</title>
<link href="www.nghqqa.cn/2019/07/25/hadoop%E4%BC%AA%E5%88%86%E5%B8%83%E5%BC%8F%E5%AE%89%E8%A3%85/"/>
<id>www.nghqqa.cn/2019/07/25/hadoop伪分布式安装/</id>
<published>2019-07-25T05:38:40.000Z</published>
<updated>2019-08-22T04:31:24.460Z</updated>
<content type="html"><![CDATA[<h1 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h1><p>本文介绍的主要是Hadoop的伪分布式的搭建以及遇到的相关问题的解决，做一下记录，jdk的安装这里就不做太多的介绍了，相信大家根据网上的安装介绍很快就能安装成功。</p> <a id="more"></a><p>hadoop集群分为3种模型</p><ol><li>单机模型：测试使用 </li><li>伪分布式模型：运行于单机 </li><li>完全分布式模型：适用于多台机器</li></ol><p>以下是使用的环境</p><table><thead><tr><th align="left">操作环境</th><th>主机名</th><th>IP地址</th><th>jdk</th><th align="center">hadoop版本</th><th></th></tr></thead><tbody><tr><td align="left">centos6.8</td><td>hadoop101</td><td>192.168.128.171</td><td>jdk1.8.0_191</td><td align="center">hadoop-2.7.2</td><td></td></tr><tr><td align="left"></td><td></td><td></td><td></td><td align="center"></td><td></td></tr></tbody></table><p>本文中使用的各种包，后续我会进行上传，以方便读者的使用</p><h1 id="安装hadoop"><a href="#安装hadoop" class="headerlink" title="安装hadoop"></a>安装hadoop</h1><h2 id="hadoop上传与解压"><a href="#hadoop上传与解压" class="headerlink" title="hadoop上传与解压"></a>hadoop上传与解压</h2><p> 当我们配置好自己的虚拟机后，可以自行在网上下载xftp和xshell，来对于自己的虚拟机进行远程上传文件和远程操作，这两款软件对于学生而言都是免费的，大家可以自行在网站上下载，速度可能会有点慢。</p><p> 当我们下载好这两款软件后，就可以将hadoop的解压包上传至自己的虚拟机上去，我们将解压包上传至/opt/software中，开始解压hadoop，将hadoop解压至/opt/module/中，同时建议将java也解压至/opt/module/中，方便后面的管理。</p><h3 id="配置环境变量"><a href="#配置环境变量" class="headerlink" title="配置环境变量"></a>配置环境变量</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">vi /etc/profile</span><br></pre></td></tr></table></figure><p>具体配置</p><p>在/etc/profile的最后面加上，关于vi编辑器的用法可以自行百度一下，简单用法应该几分钟就能学会</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">export JAVA_HOME=/opt/module/jdk1.8.0_191</span><br><span class="line">export PATH=$PATH:$JAVA_HOME/bin</span><br><span class="line"></span><br><span class="line">export HADOOP_HOME=/opt/module/hadoop-2.7.2</span><br><span class="line">export PATH=$PATH:$HADOOP_HOME/bin</span><br><span class="line">export PATH=$PATH:$HADOOP_HOME/sbin</span><br></pre></td></tr></table></figure><p>在配置环境变量完成后，记得要进行让它生效</p><p>输入</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">source /etc/profile</span><br></pre></td></tr></table></figure><p>即可生效</p><p>可以输入</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">hadoop version</span><br></pre></td></tr></table></figure><p>如果成功则显示</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">Hadoop 2.7.2</span><br><span class="line">Subversion Unknown -r Unknown</span><br><span class="line">Compiled by root on 2017-05-22T10:49Z</span><br><span class="line">Compiled with protoc 2.5.0</span><br><span class="line">From source with checksum d0fda26633fa762bff87ec759ebe689c</span><br><span class="line">This command was run using /opt/module/hadoop-2.7.2/share/hadoop/common/hadoop-common-2.7.2.jar</span><br></pre></td></tr></table></figure><p>到这里hadoop就算是安装好了</p><h1 id="配置hadoop"><a href="#配置hadoop" class="headerlink" title="配置hadoop"></a>配置hadoop</h1><p>在伪分布式里我们只需要改三个配置文件core-site.xml和hdfs-site.xml还有hadoop-env.sh</p><p>这三个文件在hadoop目录下的etc/hadoop文件夹下</p><p><strong>core-site.xml文件包含了NameNode主机地址，监听端口等信息，对于这个伪分布式模型来说，我的主机地址为hadoo101，NameNode默认使用的端口为8020。</strong></p><p>修改core-site.xml</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><configuration></span><br><span class="line"><!-- 指定HDFS中NameNode的地址 --></span><br><span class="line"><property></span><br><span class="line"> <name>fs.defaultFS</name></span><br><span class="line"> <value>hdfs://hadoop101:8020</value></span><br><span class="line"></property></span><br><span class="line"> <!-- 指定hadoop运行时产生文件的存储目录 --></span><br><span class="line"> <property></span><br><span class="line"> <name>hadoop.tmp.dir</name></span><br><span class="line"> <value>/opt/module/hadoop-2.7.2/data/tmp</value></span><br><span class="line"> </property></span><br><span class="line"></configuration></span><br></pre></td></tr></table></figure><p><strong>hdfs-site.xml用于配置/HDFS的相关属性，例如数据块的副本参数，数据块的副本对于伪分布式来说应该为1</strong></p><p>修改hdfs-site.xml</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><configuration></span><br><span class="line"><!-- 指定HDFS副本的数量 --></span><br><span class="line"> <property></span><br><span class="line"> <name>dfs.replication</name></span><br><span class="line"> <value>1</value></span><br><span class="line"> </property></span><br><span class="line"></configuration></span><br></pre></td></tr></table></figure><p><strong>hadoop-env.sh 由于Hadoop是java进程，所以需要添加jdk</strong></p><p>修改hadoop-env.sh</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">export JAVA_HOME=/opt/module/jdk1.8.0_191</span><br></pre></td></tr></table></figure><p>对于伪分布式来说，改这三个配置文件够了。</p><p>在配置文件完成后，我们需要对hadoop进行初始化</p><p>在hadoop-2.7.2的目录下输入</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">bin/hdfs namenode -format</span><br></pre></td></tr></table></figure><p>如果初始化成功的话，一个和下图相似</p><p><img src="http://pv4lxcno2.bkt.clouddn.com/blog/20190725/9c98NJ18De35.png?imageslim" alt="mark"></p><p>到这里hadoop的配置就已经完成了</p><h1 id="启动集群"><a href="#启动集群" class="headerlink" title="启动集群"></a>启动集群</h1><p>在hadoop-2.7.2目录下输入</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sbin/start-dfs.sh</span><br></pre></td></tr></table></figure><p>启动dfs</p><p>输入</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sbin/start-yarn.sh</span><br></pre></td></tr></table></figure><p>启动yarn节点</p><p>启动成功应该和下图相似</p><p><img src="http://pv4lxcno2.bkt.clouddn.com/blog/20190725/15Fa0e8qUDCR.png?imageslim" alt="mark"></p><p>到这里我们的集群就算是启动成功了</p><p>我们可以在web端查看HDFS文件系统</p><p><a href="http://192.168.128.171:50070/" target="_blank" rel="noopener">http://192.168.128.171:50070</a></p><p>192.168.128.171是我的ip地址，如果配置的不同，改一下即可</p><p>web端的hdfs文件系统如下图所示</p><p><img src="http://pv4lxcno2.bkt.clouddn.com/blog/20190725/fKgRPKMecBlX.png?imageslim" alt="mark"></p><h1 id="测试集群"><a href="#测试集群" class="headerlink" title="测试集群"></a>测试集群</h1><p>在HDFS文件系统上创建一个input文件夹</p><p>输入</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">hadoop fs -mkdir /input</span><br></pre></td></tr></table></figure><p>在web端应该可以看到</p><p><img src="http://pv4lxcno2.bkt.clouddn.com/blog/20190725/GK25hp1DN8GK.png?imageslim" alt="mark"></p><p>我们上传一个文件看看</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">hadoop fs -put LICENSE.txt /input</span><br></pre></td></tr></table></figure><p>LICENSE.txt是hadoop自带的一个TXT文件</p><p>如果上传成功在web端应该可以看到</p><p><img src="http://pv4lxcno2.bkt.clouddn.com/blog/20190725/C4OldlfJiDzs.png?imageslim" alt="mark"></p><p>这样就是上传成功了</p><p>我们在HDFS上跑一下MapReduce程序</p><p>输入</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">hadoop jar /opt/module/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount /input/LICENSE.txt /output</span><br></pre></td></tr></table></figure><p>这里说明一下MapReduce要在启动yarn下运行</p><p>查看运行结果</p><p>在web端：</p><p><img src="http://pv4lxcno2.bkt.clouddn.com/blog/20190725/4wgLyU3syVSv.png?imageslim" alt="mark"></p><p>part-r-00000这个就是运行出来的结果</p><p>我们可以使用命令行查看结果也可以把这个文件下载到本地，这里我们使用命令行查看</p><p>输入</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">bin/hdfs dfs -cat /output/part-r-00000</span><br></pre></td></tr></table></figure><p>返回如下结果</p><p><img src="http://pv4lxcno2.bkt.clouddn.com/blog/20190725/FHhHjKyLhTk9.png?imageslim" alt="mark"></p><p>到这里基本可以了，我们的hadoop已经安装配置好了，可以进行下一步的学习了</p><p>关于MapReduce的WordCount程序详解可以看这个</p><p><a href="https://blog.csdn.net/gulu_gulu_jp/article/details/51298164/" target="_blank" rel="noopener">https://blog.csdn.net/gulu_gulu_jp/article/details/51298164/</a></p><p>本次伪分布的配置就到这里了，如果还有问题可以向我留言，谢谢阅读，下次的文章应该是完全分布式的hadoop的安装教程了</p>]]></content>
<summary type="html">
<h1 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h1><p>本文介绍的主要是Hadoop的伪分布式的搭建以及遇到的相关问题的解决，做一下记录，jdk的安装这里就不做太多的介绍了，相信大家根据网上的安装介绍很快就能安装成功。</p>
</summary>
<category term="搭建自己的hadoop学习集群" scheme="www.nghqqa.cn/categories/%E6%90%AD%E5%BB%BA%E8%87%AA%E5%B7%B1%E7%9A%84hadoop%E5%AD%A6%E4%B9%A0%E9%9B%86%E7%BE%A4/"/>
<category term="hadoop" scheme="www.nghqqa.cn/tags/hadoop/"/>
</entry>
</feed>