调整kubernetes中持久卷的大小时出错
我有一个Prometheus服务器pod,它使用8Gi持久块卷。 卷提供程序是rook ceph pod处于crashloopbackoff状态,因为没有更多可用空间:调整kubernetes中持久卷的大小时出错,kubernetes,persistent-volumes,rook-storage,Kubernetes,Persistent Volumes,Rook Storage,我有一个Prometheus服务器pod,它使用8Gi持久块卷。 卷提供程序是rook ceph pod处于crashloopbackoff状态,因为没有更多可用空间: [root@node4 ~]# df -h | grep rbd /dev/rbd0 8.0G 8.0G 36K 100% /var/lib/kubelet/plugins/ceph.rook.io/rook-ceph/mounts/pvc-80f98193-deae-11e9-a240-0025b50a01df
[root@node4 ~]# df -h | grep rbd
/dev/rbd0 8.0G 8.0G 36K 100% /var/lib/kubelet/plugins/ceph.rook.io/rook-ceph/mounts/pvc-80f98193-deae-11e9-a240-0025b50a01df
pod需要更多的空间,所以我决定将音量调整为20Gi
通过以下文件:
我在持久卷声明中编辑了resources.requests.storage:20Gi。
并升级了头盔释放
现在我可以看到PV已经调整到20Gi。但PVC仍然表明它声称拥有8Gi
$ kubectl get pvc -n prometheus
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
prometheus-alertmanager Bound pvc-80f5eb1a-deae-11e9-a240-0025b50a01df 2Gi RWO rook-ceph-block 22d
prometheus-server Bound pvc-80f98193-deae-11e9-a240-0025b50a01df 8Gi RWO rook-ceph-block 22d
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-80f5eb1a-deae-11e9-a240-0025b50a01df 2Gi RWO Delete Bound prometheus/prometheus-alertmanager rook-ceph-block 22d
pvc-80f98193-deae-11e9-a240-0025b50a01df 20Gi RWO Delete Bound prometheus/prometheus-server rook-ceph-block 22d
pvc-fb73b383-deb2-11e9-a240-0025b50a01df 10Gi RWO Delete Bound grafana/grafana rook-ceph-block 22d
PVC说明如下:
Conditions:
Type Status LastProbeTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
FileSystemResizePending True Mon, 01 Jan 0001 00:00:00 +0000 Thu, 17 Oct 2019 15:49:05 +0530 Waiting for user to (re-)start a pod to finish file system resize of volume on node.
Warning FailedMount 2m17s (x2 over 2m17s) kubelet, node4 MountVolume.SetUp failed for volume "pvc-80f98193-deae-11e9-a240-0025b50a01df" : mount command failed, status: Failure, reason: Rook: Mount volume failed: failed to attach volume pvc-80f98193-deae-11e9-a240-0025b50a01df for pod prometheus/prometheus-server-756c8495ff-wtx84. Volume is already attached by pod prometheus/prometheus-server-756c8495ff-hcd85. Status Running
然后我删除了pod以重新启动它
但pod仍处于紧急回退状态。吊舱描述说:
Conditions:
Type Status LastProbeTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
FileSystemResizePending True Mon, 01 Jan 0001 00:00:00 +0000 Thu, 17 Oct 2019 15:49:05 +0530 Waiting for user to (re-)start a pod to finish file system resize of volume on node.
Warning FailedMount 2m17s (x2 over 2m17s) kubelet, node4 MountVolume.SetUp failed for volume "pvc-80f98193-deae-11e9-a240-0025b50a01df" : mount command failed, status: Failure, reason: Rook: Mount volume failed: failed to attach volume pvc-80f98193-deae-11e9-a240-0025b50a01df for pod prometheus/prometheus-server-756c8495ff-wtx84. Volume is already attached by pod prometheus/prometheus-server-756c8495ff-hcd85. Status Running
当列出pod时,我只能看到新的podprometheus-server-756c8495ff-wtx84
(而不是旧的podprometheus-server-756c8495ff-hcd85
):
我如何解决这个问题
编辑:
部署的战略是:
StrategyType: RollingUpdate
RollingUpdateStrategy: 1 max unavailable, 1 max surge
我可以看到,即使kubectl get pv
显示pv的容量为20Gi,rook ceph的实际rbd块大小仅为8Gi:
[root@rook-ceph-operator-775cf575c5-dfpql /]# rbd info replicated-metadata-pool/pvc-80f98193-deae-11e9-a240-0025b50a01df
rbd image 'pvc-80f98193-deae-11e9-a240-0025b50a01df':
size 8 GiB in 2048 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 434b1922b4b40a
data_pool: ec-data-pool
block_name_prefix: rbd_data.1.434b1922b4b40a
format: 2
features: layering, data-pool
op_features:
flags:
create_timestamp: Tue Sep 24 09:34:28 2019
access_timestamp: Tue Sep 24 09:34:28 2019
modify_timestamp: Tue Sep 24 09:34:28 2019
storageclass.yaml是:
$ kubectl get sc -n prometheus -o yaml
apiVersion: v1
items:
- allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
creationTimestamp: "2019-08-01T11:27:31Z"
name: rook-ceph-block
resourceVersion: "15172025"
selfLink: /apis/storage.k8s.io/v1/storageclasses/rook-ceph-block
uid: 59e3b081-b44f-11e9-a240-0025b50a01df
parameters:
blockPool: replicated-metadata-pool
clusterNamespace: rook-ceph
dataBlockPool: ec-data-pool
fstype: xfs
provisioner: ceph.rook.io/block
reclaimPolicy: Delete
volumeBindingMode: Immediate
kind: List
metadata:
resourceVersion: ""
selfLink: ""
您可以尝试手动调整ext4格式的大小。这是打开的问题()您可以尝试手动调整ext4格式的大小。这是一个开放的问题()但上面写着:rbd:option'-size'的参数无效,但上面写着:rbd:option'-size'的参数无效你能为你的
存储类发布yaml吗?@Crou我编辑了这个问题。请看一看。你能为你的StorageClass
发布yaml吗?@Crou我编辑了这个问题。请看一看。