[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Issues with logging and metrics on Origin 3.7



I'm hitting a number of issues with installing logging and metrics on Origin 3.7. This is using Centos7 hosts, the release-3.7 branch of openshift-ansible and NFS for persistent storage.

I first do a minimal deploy with logging and metrics turned off.
This goes fine. On the NFS server I see various volumes exported under /exports for logging, metrics, prometheus, even thought these are not deployed, but that's fine,  they are there if they become needed.
As epxected there are no PVs related to metrics and logging.

So I try to install metrics. I add this to the inventory file:

openshift_metrics_install_metrics=true
openshift_metrics_storage_kind=nfs
openshift_metrics_storage_access_modes=['ReadWriteOnce']
openshift_metrics_storage_nfs_directory=/exports
openshift_metrics_storage_nfs_options='*(rw,root_squash)'
openshift_metrics_storage_volume_name=metrics
openshift_metrics_storage_volume_size=10Gi
openshift_metrics_storage_labels={'storage': 'metrics'}

and run:

ansible-playbook openshift-ansible/playbooks/byo/openshift-cluster/openshift-metrics.yml

All seems to install OK, but metrics can't start, and it turns out that no PV is created so the PVC needed by Casandra can't be satisfied.
So I manually create the PV using this definition:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: metrics-pv
  labels:
    storage: metrics
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /exports/metrics
    server: nfsserver

Now the PVC is satisfied and metrics can be started (though pods may need to be bounced because they have timed out).

ISSUE 1: why does the metrics PV not get created?


So now on to trying to install logging. The approach is similar. Add this to the inventory file:

openshift_logging_install_logging=true
openshift_logging_storage_kind=nfs
openshift_logging_storage_access_modes=['ReadWriteOnce']
openshift_logging_storage_nfs_directory=/exports
openshift_logging_storage_nfs_options='*(rw,root_squash)'
openshift_logging_storage_volume_name=logging
openshift_logging_storage_volume_size=10Gi
openshift_logging_storage_labels={'storage': 'logging'}

and run:
ansible-playbook openshift-ansible/playbooks/byo/openshift-cluster/openshift-logging.yml

Logging installs fine, and is running fine. Kibana shows logs.
But look at what has been installed and there are no PVs or PVs for logging. It seems it has  ignored the instructions to use NFS and and deployed using ephemeral storage.

ISSUE 2: why does the persistence definitions get ignored?


And finally, looking at the metrics and logging images on Docker Hub there are none with v3.7.0 or v3.7 tags. The only tag related to 3.7 is v3.7.0-rc.0. For example look here:
https://hub.docker.com/r/openshift/origin-metrics-hawkular-metrics/tags/
But for other openshift components there is a v3.7.0 tag present.
Without specifying any particular tag to use for metrics or logging it seems you get 'latest' installed.

ISSUE 3: is 3.7 officially released yet (there's no docs for this here either: https://docs.openshift.org/index.html)?


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]