Kubernetes集群实践(八)Kubernetes安装云原生存储系统Rook

本文主要介绍如何安装和使用云原生存储系统Rook。

关键词:k8s

准备工作

  • Ubuntu 20.04 LTS amd64,32GB,作为Master节点

假定已经安装了K8s集群,这里以单节点集群为例

Rook

  • Rook是一个开源的云原生存储协调器,提供平台、框架和对各种存储解决方案的支持,以与云原生环境进行本地集成。
  • Rook将存储软件转变为自我管理,自我扩展和自我修复的存储服务。它通过自动化部署、引导、配置、供应、扩展、升级、迁移、灾难恢复、监视和资源管理来实现。Rook使用底层云原生容器管理,调度和编排平台提供的功能来执行其职责。
  • Rook利用扩展点将其深度集成到云原生环境中,并为调度、生命周期管理、资源管理、安全性、监视和用户体验提供无缝的体验。

部署Rook

下载源码,项目地址:https://github.com/rook/rook/releases

最新源码为1.9.7

1
git clone --single-branch --branch v1.9.7 https://github.com/rook/rook.git

以块存储RBD为例

部署准备工作

1
2
cd rook/deploy/examples
kubectl create -f crds.yaml -f common.yaml -f operator.yaml

确保存储设备没有文件系统

1
sudo fdisk -l

如果有GPT或者dos等描述文字,说明存在分区表,需要把这个删掉

1
sudo wipefs /dev/sda

可以看到分区表

执行删除命令

1
sudo wipefs -a -f /dev/sda

根据需要修改一下集群

1
vim cluster.yaml

主要是指定设备,设定mon为1,mgr为1等操作

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
#################################################################################################################
# Define the settings for the rook-ceph cluster with common settings for a small test cluster.
# All nodes with available raw devices will be used for the Ceph cluster. One node is sufficient
# in this example.

# For example, to create the cluster:
# kubectl create -f crds.yaml -f common.yaml -f operator.yaml
# kubectl create -f cluster-test.yaml
#################################################################################################################
kind: ConfigMap
apiVersion: v1
metadata:
name: rook-config-override
namespace: rook-ceph # namespace:cluster
data:
config: |
[global]
osd_pool_default_size = 1
mon_warn_on_pool_no_redundancy = false
bdev_flock_retry = 20
bluefs_buffered_io = false
---
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
name: my-cluster
namespace: rook-ceph # namespace:cluster
spec:
dataDirHostPath: /var/lib/rook
cephVersion:
image: quay.io/ceph/ceph:v17
allowUnsupported: true
mon:
count: 1
allowMultiplePerNode: true
mgr:
count: 1
allowMultiplePerNode: true
dashboard:
enabled: true
crashCollector:
disable: true
storage:
useAllNodes: false
useAllDevices: false
#deviceFilter:
config:
# crushRoot: "custom-root" # specify a non-default root label for the CRUSH map
# metadataDevice: "md0" # specify a non-rotational storage so ceph-volume will use it as block db device of bluestore.
# databaseSizeMB: "1024" # uncomment if the disks are smaller than 100 GB
# journalSizeMB: "1024" # uncomment if the disks are 20 GB or smaller
osdsPerDevice: "1" # this value can be overridden at the node or device level
# encryptedDevice: "true" # the default value for this option is "false"
# Individual nodes and their config can be specified as well, but 'useAllNodes' above must be set to false. Then, only the named
# nodes below will be used as storage resources. Each node's 'name' field should match their 'kubernetes.io/hostname' label.
nodes:
- name: "amd-5700g"
devices: # specific devices to use for storage can be specified for each node
- name: "sda"
deviceFilter:
healthCheck:
daemonHealth:
mon:
interval: 45s
timeout: 600s
priorityClassNames:
all: system-node-critical
mgr: system-cluster-critical
disruptionManagement:
managePodBudgets: true
---
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
name: builtin-mgr
namespace: rook-ceph # namespace:cluster
spec:
name: .mgr
replicated:
size: 1
requireSafeReplicaSize: false

现在可以部署了

1
kubectl apply -f cluster.yaml

安装ceph工具

1
kubectl applt -f toolbox.yaml

等待相关操作执行完毕

此时进入ceph-tools,查看ceph集群工作状态

1
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash

挂载Rook块存储到Pod容器

首先需要创建strongeClass,才可以让k8s控制器根据声明的PVC创建PV

1
2
3
4
5
6
7
8
9
10
11
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-standalone-pvc
spec:
accessModes:
- ReadWriteOnce
storageClassName: rook-ceph-block
resources:
requests:
storage: 1Gi

然后在Pod里创建PVC

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
apiVersion: v1
kind: Pod
metadata:
name: mysql-standalone-pod
spec:
containers:
- name: mysql
image: mysql:8-debian
args:
- --max_connections=10000
- --character-set-server=utf8mb4
- --collation-server=utf8mb4_general_ci
- --default-authentication-plugin=mysql_native_password
env:
- name: MYSQL_ROOT_PASSWORD
value: "root"
volumeMounts:
- name: mysql-standalone-data
mountPath: /var/lib/mysql # 容器挂载路径
subPath: mysql
- name: mysql-conf
mountPath: /etc/mysql/conf.d
volumes:
- name: mysql-standalone-data
persistentVolumeClaim:
claimName: mysql-standalone-pvc
- name: mysql-conf
configMap:
name: mysql-standalone-config