[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Provisioning persistence for metrics with GlusterFS



Hi Dan!

Now what I also don't understand is how did the initial volume group for the registry got created with just 26GB of storage if the default is for 100GB? Is there a rule such as: "create block-hosting volume of default size=100GB or max available"?
The integrated registry's persistence is set to 5GB. This is, I believe, a default value, as I haven't set anything related to it in my inventory file when installing Openshift Origin. How can I use the remaining storage in my vg with glusterFS and Openshift?

The registry-storage volume is not a block-volume is a file volume, witch has no "minimum" size. So you can create many other small file volumes with no problems. The only restriction will be to create block-volumes, that need a least 100GB.

Best regards,


Rodrigo Bersa

Cloud Consultant, RHCVA, RHCE

Red Hat Brasil

rbersa redhat com    M: +55-11-99557-5841    

Red Hat é reconhecida entre as melhores empresas para trabalhar no Brasil pelo Great Place to Work.

On Mon, May 21, 2018 at 7:49 AM, Dan Pungă <dan punga gmail com> wrote:
Hello Rodrigo, I appreciate your answer!

In the meantime I had reached for the heketi-cli related support(chat) and I got the same reference. There's a config map generated by the installer for the heketi-registry pod that has the default for block-hosting volumes size set at 100GB.
What I thought was that the "block hosting volume" would be an equivalent of a logical volume and it(heketi-cli) tries to create a lv of size 100GB inside the already created
vg_bd61a1e6f317bb9decade964449c12e8(which has 26GB).

I've actually modified the encrypted json config and tried to restart the heketi-registry pod, which failed. So I ended up with some unmanaged glusterFS storage, but since I'm on a test envionment, it's fine. Otherwise, good to know for the future.

Now what I also don't understand is how did the initial volume group for the registry got created with just 26GB of storage if the default is for 100GB? Is there a rule such as: "create block-hosting volume of default size=100GB or max available"?
The integrated registry's persistence is set to 5GB. This is, I believe, a default value, as I haven't set anything related to it in my inventory file when installing Openshift Origin. How can I use the remaining storage in my vg with glusterFS and Openshift?

Thank you!

On 19.05.2018 02:43, Rodrigo Bersa wrote:
Hi Dan,

The Gluster Block volumes works with the concept of block-hosting volume, and these ones are created with 100GB by default.

To clarify, the block volumes will be provisioned over the block hosting volumes.

Let's say you need a 10GB block volume, it will create a block hosting volume with 100GB and then the 10GB block volume over it, as the next block volumes requested until it reaches the 100GB. After that a new block hosting volume will be created and so on.

So, if you have just 26GB available in each server, it's not enough to create the block hosting volume. You may need to add more devices to your CNS Cluster to grow your free space.


Kind regards,


Rodrigo Bersa

Cloud Consultant, RHCVA, RHCE

Red Hat Brasil

rbersa redhat com    M: +55-11-99557-5841    

Red Hat é reconhecida entre as melhores empresas para trabalhar no Brasil pelo Great Place to Work.

On Wed, May 16, 2018 at 10:35 PM, Dan Pungă <dan punga gmail com> wrote:
Hello all!

I have setup a cluster with 3 glusterFS nodes for disk persistence just as specified in the docs. I have configured the inventory file to install the containerized version to be used by Openshift's integrated registry. This works fine.

Now I wanted to install the metrics component and I followed the procedure described here: https://docs.openshift.org/latest/install_config/persistent_storage/persistent_storage_glusterfs.html#install-example-infra

I end up with openshift-infra project set up, but with 3 pods failing to start and I think this has to do with the PVC for cassandra that fails to create.

oc get pvc metrics-cassandra-1 -o yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"8ef584d1-5923-11e8-8730-0a580a830040","leaseDurationSeconds":15,"acquireTime":"2018-05-17T00:38:34Z","renewTime":"2018-05-17T00:55:33Z","leaderTransitions":0}'
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{"volume.beta.kubernetes.io/storage-provisioner":"gluster.org/glusterblock"},"labels":{"metrics-infra":"hawkular-cassandra"},"name":"metrics-cassandra-1","namespace":"openshift-infra"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"6Gi"}},"storageClassName":"glusterfs-registry-block"}}
    volume.beta.kubernetes.io/storage-provisioner: gluster.org/glusterblock
  creationTimestamp: 2018-05-17T00:38:34Z
  labels:
    metrics-infra: hawkular-cassandra
  name: metrics-cassandra-1
  namespace: openshift-infra
  resourceVersion: "1204482"
  selfLink: /api/v1/namespaces/openshift-infra/persistentvolumeclaims/metrics-cassandra-1
  uid: a18b8c20-596a-11e8-8a63-fa163ed601cb
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 6Gi
  storageClassName: glusterfs-registry-block
status:
  phase: Pending


oc describe pvc metrics-cassandra-1 shows these warnings:

 36m        23m        13    gluster.org/glusterblock glusterblock-registry-provisioner-dc-1-tljbb 8ef584d1-5923-11e8-8730-0a580a830040            Warning        ProvisioningFailed    Failed to provision volume with StorageClass "glusterfs-registry-block": failed to create volume: [heketi] failed to create volume: Failed to allocate new block volume: No space
  36m        21m        14    gluster.org/glusterblock glusterblock-registry-provisioner-dc-1-tljbb 8ef584d1-5923-11e8-8730-0a580a830040            Normal        Provisioning        External provisioner is provisioning volume for claim "openshift-infra/metrics-cassandra-1"
  21m        21m        1    gluster.org/glusterblock glusterblock-registry-provisioner-dc-1-tljbb 8ef584d1-5923-11e8-8730-0a580a830040            Warning        ProvisioningFailed    Failed to provision volume with StorageClass "glusterfs-registry-block": failed to create volume: [heketi] failed to create volume: Post http://heketi-registry-default.apps.my.net/blockvolumes: dial tcp: lookup heketi-registry-default.apps.my.net on 192.168.150.16:53: no such host


In the default project, if I check the logs for heketi-registry, I get a lot of

[heketi] ERROR 2018/05/17 00:46:47 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:909: Create Block Volume Build Failed: No space
[negroni] Started POST /blockvolumes
[heketi] INFO 2018/05/17 00:49:02 Loaded simple allocator
[heketi] INFO 2018/05/17 00:49:02 brick_num: 0
[heketi] INFO 2018/05/17 00:49:02 brick_num: 0
[heketi] INFO 2018/05/17 00:49:02 brick_num: 0
[heketi] INFO 2018/05/17 00:49:02 brick_num: 0
[heketi] INFO 2018/05/17 00:49:02 brick_num: 1
[negroni] Completed 500 Internal Server Error in 7.091238ms


For the other glusterFS-related pod, I see the same errors reported by the pvc creation

oc logs -f glusterblock-registry-provisioner-dc-1-tljbb -n default

I0516 22:38:49.136388       1 controller.go:1167] scheduleOperation[lock-provision-openshift-infra/metrics-cassandra-1[1191fb8d-5959-11e8-94c9-fa163e1cba7f]]
I0516 22:38:49.166658       1 leaderelection.go:156] attempting to acquire leader lease...
I0516 22:38:49.197051       1 leaderelection.go:178] successfully acquired lease to provision for pvc openshift-infra/metrics-cassandra-1
I0516 22:38:49.197122       1 controller.go:1167] scheduleOperation[provision-openshift-infra/metrics-cassandra-1[1191fb8d-5959-11e8-94c9-fa163e1cba7f]]
E0516 22:38:49.207257       1 glusterblock-provisioner.go:441] BLOCK VOLUME NAME I RECEIEVED:
E0516 22:38:49.207288       1 glusterblock-provisioner.go:449] BLOCK VOLUME CREATE REQUEST: &{Size:6 Clusters:[] Name: Hacount:3 Auth:true}
E0516 22:38:49.355122       1 glusterblock-provisioner.go:451] BLOCK VOLUME RESPONSE: <nil>
E0516 22:38:49.355204       1 glusterblock-provisioner.go:453] [heketi] failed to create volume: Failed to allocate new block volume: No space
E0516 22:38:49.355262       1 controller.go:895] Failed to provision volume for claim "openshift-infra/metrics-cassandra-1" with StorageClass "glusterfs-registry-block": failed to create volume: [heketi] failed to create volume: Failed to allocate new block volume: No space
E0516 22:38:49.355365       1 goroutinemap.go:165] Operation for "provision-openshift-infra/metrics-cassandra-1[1191fb8d-5959-11e8-94c9-fa163e1cba7f]" failed. No retries permitted until 2018-05-16 22:40:51.355301022 +0000 UTC m=+23465.283195247 (durationBeforeRetry 2m2s). Error: "failed to create volume: [heketi] failed to create volume: Failed to allocate new block volume: No space"
I0516 22:38:51.241605       1 leaderelection.go:198] stopped trying to renew lease to provision for pvc openshift-infra/metrics-cassandra-1, task failed


Regarding the no space message, I am certain that there is space on the device (if there isn't some glusterFS config that's done on the servers which prevents them to extend/create the volumes). All disks have the same 26GB capacity and lvs  on one of the machines shows:

 
LV                                     VG                                  Attr       LSize  Pool                                Origin Data%  Meta%  Move Log Cpy%Sync Convert
  docker-pool                            rootvg                              twi-aot--- <4,16g                                            52,37  2,62                           
  home                                   rootvg                              -wi-ao----  1,00g                                                                                  
  root                                   rootvg                              -wi-ao----  2,00g                                                                                  
  swap                                   rootvg                              -wi-a-----  2,00g                                                                                  
  tmp                                    rootvg                              -wi-ao----  1,17g                                                                                  
  usr                                    rootvg                              -wi-ao----  4,00g                                                                                  
  var                                    rootvg                              -wi-ao----  4,00g                                                                                  
  brick_7aa3a789badd1ae620a2bbefe51b8c73 vg_bd61a1e6f317bb9decade964449c12e8 Vwi-aotz--  2,00g tp_7aa3a789badd1ae620a2bbefe51b8c73        0,71                                  
  brick_8818ffee7ab2244ca721b7d15ea1e514 vg_bd61a1e6f317bb9decade964449c12e8 Vwi-aotz--  5,00g tp_8818ffee7ab2244ca721b7d15ea1e514        7,57                                  
  tp_7aa3a789badd1ae620a2bbefe51b8c73    vg_bd61a1e6f317bb9decade964449c12e8 twi-aotz--  2,00g                                            0,71   0,33                           
  tp_8818ffee7ab2244ca721b7d15ea1e514    vg_bd61a1e6f317bb9decade964449c12e8 twi-aotz--  5,00g                                            7,57   0,29


Any ideas where to look for misconfigurations?

Thank you,
Dan Pungă

_______________________________________________
users mailing list
users lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users





[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]