Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions openshift_aws_nvme_disk_order_fix/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# README

Description:

It was discovered in an OpenShift Cluster deployed into AWS that the AWS NVME disk order changes after reboot in the Red Hat Enterprise Linux 7 operating system. This impacts docker's storage configuration negatively if you have multiple disks configured. This repo contains two basic playbooks that address the two core use cases.

# Use Case 1: Pre Docker Storage Setup

If you are configuring docker storage for the first time:

ocp-aws-nvme-docker-preinstall.yml

# Use Case 2: Post Docker storage setup

If you have already configured docker storage, disks have changed order, and you need to fix dockers configuration in order to bring up the service:

ocp-aws-nvme-docker-preinstall-fix.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
---
- name: Update /etc/sysconfig/docker-storage-setup configuration to use a persistent path from AWS
hosts: all
tasks:
- name: Determine disk used for PV in docker-vg volume group
shell: vgdisplay docker-vg -v | grep "PV Name" -A1 | awk '{print $3}' | sed -n '1p' | sed 's/.\{2\}$//'
register: UUID

- debug: msg={{ UUID.stdout }}

- name: Grab NVME Path from udevinfo info {{ UUID.stdout }}
shell: udevadm info {{ UUID.stdout }} | sed -n '3p' | sed 's/^.\{,14\}//'
register: NVME_PATH

- debug: msg={{ NVME_PATH.stdout }}

- name: Backup /etc/sysconfig/docker-storage-setup to /etc/sysconfig/docker-storage-setup.{{ ansible_date_time.date }}.{{ ansible_date_time.time }}
copy:
src: /etc/sysconfig/docker-storage-setup
dest: /etc/sysconfig/docker-storage-setup.{{ ansible_date_time.date }}.{{ ansible_date_time.time }}
remote_src: yes

- name: Update /etc/sysconfig/docker-storage-setup with Persistent Device Patch
lineinfile:
path: /etc/sysconfig/docker-storage-setup
regexp: '^DEVS'
line: 'DEVS=/dev/disk/by-id/{{ NVME_PATH.stdout }}'

- name: Remove Incorrrect Disk from Volume Group
shell: vgreduce --removemissing docker-vg

- name: Activate Volume Group
shell: lvchange -a y docker-vg
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
---
- name: Update /etc/sysconfig/docker-storage-setup configuration to use a persistent path from AWS
hosts: all
vars:
ocp_docker_storage_size: 151 # Define the size of the disk that you want to use for docker storage
tasks:

- name: Get disk for Docker Storage
shell: "lsblk | grep disk | grep {{ ocp_docker_storage_size }} |cut -f1 -d' '"
register: docker_device

- name: Grab NVME Path from udevinfo info {{ docker_device.stdout }}
shell: udevadm info /dev/{{ docker_device.stdout }} | sed -n '3p' | sed 's/^.\{,14\}//'
register: NVME_PATH

- name: Configure Docker Storage
copy:
dest: /etc/sysconfig/docker-storage-setup
content: |
DEVS="/dev/disk/by-id/{{ NVME_PATH.stdout }}"
VG=docker-vg

- name: Docker storage setup
command: docker-storage-setup

- name: Start and Enable docker
service:
name: docker
state: started
enabled: yes