一:前言GlusterFS是Scale-Out存儲解決方案Gluster的核心,它是一個開源的分布式文件系統(tǒng),具有強大的橫向擴展能力,通過擴展能夠支持數(shù)PB存儲容量和處理數(shù)千客戶端。GlusterFS借助TCP/IP或InfiniBand RDMA網(wǎng)絡(luò)將物理分布的存儲資源聚集在一起,使用單一全局命名空間來管理數(shù)據(jù)。GlusterFS基于可堆疊的用戶空間設(shè)計,可為各種不同的數(shù)據(jù)負載提供優(yōu)異的性能。
服務(wù)器:
10.116.137.196 k8s_master
10.116.82.28 k8s_node1
10.116.36.57 k8s_node2
二:安裝glusterfs我們直接在物理機上使用yum安裝,在三臺服務(wù)器上分別執(zhí)行以下安裝命令。
1.先安裝 gluster 源yum install centos-release-gluster -y
2.安裝 glusterfs 組件yum install -y glusterfs glusterfs-server glusterfs-fuse glusterfs-rdma glusterfs-geo-replication glusterfs-devel
3.創(chuàng)建 glusterfs 目錄mkdir /mnt/glusterd
4.修改 glusterd 目錄
-
volume management
-
type mgmt/glusterd
-
option working-directory /mnt/glusterd
-
option transport-type socket,rdma
-
option transport.socket.keepalive-time 10
-
option transport.socket.keepalive-interval 2
-
option transport.socket.read-fail-log off
-
option ping-timeout 0
-
option event-threads 1
-
#option lock-timer 180
-
#option transport.address-family inet6
-
#option base-port 49152
-
end-volume
5.啟動glusterfssystemctl enable glusterd.service
systemctl start glusterd.service
systemctl status glusterd.service
三:配置 glusterfs1.開放端口iptables -I INPUT -p tcp --dport 24007 -j ACCEPT
2.創(chuàng)建存儲目錄mkdir /opt/gfs_data
3.添加節(jié)點到集群在10.116.137.196上執(zhí)行以下操作:
gluster peer probe 10.116.82.28
gluster peer probe 10.116.36.57
4.查看集群狀態(tài)gluster peer status
-
Number of Peers: 2
-
-
Hostname: 10.116.82.28
-
Uuid: f73138ca-e32e-4d87-a99d-cf842fc29447
-
State: Peer in Cluster(Connected)
-
-
Hostname: 10.116.36.57
-
Uuid: 18e22d2c-049b-4b0c-8cc7-2560319e6c05
-
State: Peer in Cluster(Connected)
-
[root@iZwz95trb3stk6afg8oozuZ kubernetes]#clear
-
[root@iZwz95trb3stk6afg8oozuZ kubernetes]# glustervolume create k8s-volume transport tcp 10.116.137.196:/mnt/gfs_data 10.116.82.28:/mnt/gfs_data 10.116.36.57:/mnt/gfs_data force
-
volume create: k8s-volume: success: please start thevolume to access data
-
[root@iZwz95trb3stk6afg8oozuZ kubernetes]# glustervolume info
-
-
Volume Name: k8s-volume
-
Type: Distribute
-
Volume ID: 62900029-02c9-4870-951c-38fafd5f5d9b
-
Status: Created
-
Snapshot Count: 0
-
Number of Bricks: 3
-
Transport-type: tcp
-
Bricks:
-
Brick1: 10.116.137.196:/mnt/gfs_data
-
Brick2: 10.116.82.28:/mnt/gfs_data
-
Brick3: 10.116.36.57:/mnt/gfs_data
-
Options Reconfigured:
-
transport.address-family: inet
-
nfs.disable: on
四:配置 volume1.volume的模式a.分布卷(默認模式):即DHT, 也叫 分布卷: 將文件已hash算法隨機分布到 一臺服務(wù)器節(jié)點中存儲。
b.復(fù)制模式:即AFR, 創(chuàng)建volume 時帶 replica x 數(shù)量: 將文件復(fù)制到 replica x 個節(jié)點中。
c.條帶模式:即Striped, 創(chuàng)建volume 時帶 stripe x 數(shù)量: 將文件切割成數(shù)據(jù)塊,分別存儲到 stripe x 個節(jié)點中 ( 類似raid 0 )。
d.分布式條帶模式:最少需要4臺服務(wù)器才能創(chuàng)建。 創(chuàng)建volume 時 stripe 2 server = 4 個節(jié)點: 是DHT 與 Striped 的組合型。
e.分布式復(fù)制模式:最少需要4臺服務(wù)器才能創(chuàng)建。 創(chuàng)建volume 時 replica 2 server = 4 個節(jié)點:是DHT 與 AFR 的組合型。
f.條帶復(fù)制卷模式:最少需要4臺服務(wù)器才能創(chuàng)建。 創(chuàng)建volume 時 stripe 2 replica 2 server = 4 個節(jié)點: 是 Striped 與 AFR 的組合型。
g.三種模式混合: 至少需要8臺 服務(wù)器才能創(chuàng)建。 stripe 2 replica 2 , 每4個節(jié)點 組成一個 組。
2.創(chuàng)建分布卷gluster volume create k8s-volume transport tcp 10.116.137.196:/mnt/gfs_data 10.116.82.28:/mnt/gfs_data 10.116.36.57:/mnt/gfs_data force
3.查看volume狀態(tài)gluster volume info
-
Volume Name: k8s-volume
-
Type: Distribute
-
Volume ID: 62900029-02c9-4870-951c-38fafd5f5d9b
-
Status: Created
-
Snapshot Count: 0
-
Number of Bricks: 3
-
Transport-type: tcp
-
Bricks:
-
Brick1: 10.116.137.196:/mnt/gfs_data
-
Brick2: 10.116.82.28:/mnt/gfs_data
-
Brick3: 10.116.36.57:/mnt/gfs_data
-
Options Reconfigured:
-
transport.address-family: inet
-
nfs.disable: on
4.啟動分布卷gluster volume start k8s-volume
五:Glusterfs調(diào)優(yōu)1.開啟指定volume的配額gluster volume quota k8s-volume enable
2.限制指定volume的配額gluster volume quota k8s-volume limit-usage / 5GB
3.設(shè)置cache大小, 默認32MBgluster volume set k8s-volume performance.cache-size 64MB
4.設(shè)置io線程, 太大會導(dǎo)致進程崩潰gluster volume set k8s-volume performance.io-thread-count 16
5.設(shè)置網(wǎng)絡(luò)檢測時間, 默認42sgluster volume set k8s-volume network.ping-timeout 10
6.設(shè)置寫緩沖區(qū)的大小, 默認1Mgluster volume set k8s-volume performance.write-behind-window-size 512MB
六:Kubernetes中配置glusterfs1.配置 endpoints(glusterfs-endpoints.json)
-
{
-
"kind": "Endpoints",
-
"apiVersion": "v1",
-
"metadata": {
-
"name": "glusterfs-cluster"
-
},
-
"subsets": [
-
{
-
"addresses": [
-
{
-
"ip": "10.116.137.196"
-
}
-
],
-
"ports": [
-
{
-
"port": 1990
-
}
-
]
-
}
-
]
-
}
kubectl apply -f glusterfs-endpoints.json
2.配置 service(glusterfs-service.json)
-
{
-
"kind": "Service",
-
"apiVersion": "v1",
-
"metadata": {
-
"name": "glusterfs-cluster"
-
},
-
"spec": {
-
"ports": [
-
{"port": 1990}
-
]
-
}
-
}
kubectl apply -f glusterfs-service.json
3.創(chuàng)建測試 pod
-
{
-
"apiVersion": "v1",
-
"kind": "Pod",
-
"metadata": {
-
"name": "glusterfs"
-
},
-
"spec": {
-
"containers": [
-
{
-
"name": "glusterfs",
-
"image": "registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0",
-
"volumeMounts": [
-
{
-
"mountPath": "/mnt/glusterfs",
-
"name": "glusterfsvol"
-
}
-
]
-
}
-
],
-
"volumes": [
-
{
-
"name": "glusterfsvol",
-
"glusterfs": {
-
"endpoints": "glusterfs-cluster",
-
"path": "k8s-volume",
-
"readOnly": true
-
}
-
}
-
]
-
}
-
}
kubectl apply -f glusterfs-pod.json 在Pod所在的Node節(jié)點上執(zhí)行 df -h 可以查看k8s-volume掛載目錄
4.配置PersistentVolume(glusterfs-pv.yaml)
-
apiVersion: v1
-
kind: PersistentVolume
-
metadata:
-
name: gluster-dev-volume
-
spec:
-
capacity:
-
storage: 1Gi
-
accessModes:
-
- ReadWriteMany
-
glusterfs:
-
endpoints: "glusterfs-cluster"
-
path: "k8s-volume"
-
readOnly: false
kubectl apply -f glusterfs-pv.yaml
5.配置PVC(glusterfs-pvc.yaml)
-
kind: PersistentVolumeClaim
-
apiVersion: v1
-
metadata:
-
name: glusterfs-nginx
-
spec:
-
accessModes:
-
- ReadWriteMany
-
resources:
-
requests:
-
storage: 1Gi
kubectl apply -f glusterfs-pvc.yaml
6.創(chuàng)建nginx deployment 掛載 volume(nginx-deployment.yaml)
-
apiVersion: extensions/v1beta1
-
kind: Deployment
-
metadata:
-
name: nginx-dm
-
spec:
-
replicas: 2
-
template:
-
metadata:
-
labels:
-
name: nginx
-
spec:
-
containers:
-
- name: nginx
-
image: nginx:latest
-
imagePullPolicy: IfNotPresent
-
ports:
-
- containerPort: 80
-
volumeMounts:
-
- name: gluster-dev-volume
-
mountPath: "/usr/share/nginx/html"
-
volumes:
-
- name: gluster-dev-volume
-
persistentVolumeClaim:
-
claimName: glusterfs-nginx
kubectl apply -f nginx-deployment.yaml
a.查看 deployment
kubectl get pods |grep nginx-dm
-
nginx-dm-64dcbb8d55-pp9vn 1/1 Running 0 1h
-
nginx-dm-64dcbb8d55-sclj6 1/1 Running 0 1h
b.查看掛載
kubectl exec -it nginx-dm-64dcbb8d55-pp9vn -- df -h
-
FilesystemSize Used Avail Use% Mounted on
-
overlay 40G 23G 16G 60% /
-
tmpfs 3.7G 0 3.7G 0% /dev
-
tmpfs 3.7G 0 3.7G 0% /sys/fs/cgroup
-
/dev/xvda1 40G 23G 16G 60% /etc/hosts
-
shm 64M 0 64M 0% /dev/shm
-
10.116.137.196:k8s-volume 5.0G 0 5.0G 0% /usr/share/nginx/html
-
tmpfs 3.7G 12K 3.7G 1% /run/secrets/kubernetes.io/serviceaccount
c.創(chuàng)建文件測試
kubectl exec -it nginx-dm-64dcbb8d55-pp9vn -- touch /usr/share/nginx/html/index.html
kubectl exec -it nginx-dm-64dcbb8d55-sclj6 -- ls -lt /usr/share/nginx/html/index.html
d.驗證 glusterfs
因為我們使用分布卷,所以可以看到某個節(jié)點中有文件
ls /mnt/gfs_data/
index.html
分享名稱:kubernetes實踐之二十六:GlusterFS
分享網(wǎng)址:http://chinadenli.net/article4/pgpdie.html
成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián),為您提供動態(tài)網(wǎng)站、網(wǎng)站設(shè)計、企業(yè)建站、網(wǎng)站建設(shè)、手機網(wǎng)站建設(shè)、小程序開發(fā)
廣告
聲明:本網(wǎng)站發(fā)布的內(nèi)容(圖片、視頻和文字)以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主,如果涉及侵權(quán)請盡快告知,我們將會在第一時間刪除。文章觀點不代表本網(wǎng)站立場,如需處理請聯(lián)系客服。電話:028-86922220;郵箱:631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載,或轉(zhuǎn)載時需注明來源:
創(chuàng)新互聯(lián)