Prepare to Install OKD 4.4

I have provided a set of utility scripts to automate a lot of the tasks associated with deploying and tearing down an OKD cluster. In your ~/bin/lab-bin directory you will see the following: Creates a virtual machine that will kickstart based on a specific role. We will use it to configure the HA-Proxy load balancer Destroys a guest VM and supporting infrastructure Creates the Bootstrap, Master, and Worker VMs from an inventory file, (described below) Destroys the OKD cluster and all supporting infrastructure Helper script that uses IPMI to power on the VMs listed in an inventory file
  1. Deploy the load-balancer: (Replace bastion with the hostname of your bastion host if you used a different name.) -h=okd4-lb01 -n=bastion -r=lb-node -c=2 -m=4096 -d=50 -v=6228
    -h Sets Hostname: okd4-lb01 DNS A and PTR records must exist
    -n Sets the Hypervisor Host bastion  
    -r Sets the kickstart file to lb-node.ks    
    -c Sets the vCPU count    
    -m Sets the guest memory    
    -d Sets the guest root disk volume in GB    
    -v Sets the VBMC Port 6228  

    This will create a VM which will do a kickstart install of CentOS with HA-Proxy. It will pull the haproxy.cfg file that we prepared earlier when we set up Nginx. If you are curious about the installation, take a look at:

    ${INSTALL_ROOT}/kickstart/lb-node.ks The kickstart file that will direct the installation
    ${INSTALL_ROOT}/firstboot/lb-node.fb The script that will execute on the first boot to install haproxy
    ${INSTALL_ROOT}/haproxy.cfg The haproxy configuration file for our OKD cluster

    Let’s power it on and watch the installation.

    ipmitool -I lanplus -H10.11.11.10 -p6228 -Uadmin -Ppassword chassis power on
    virsh console okd4-lb01

    You should see your VM do an iPXE boot and begin an unattended installation of CentOS 7.

  2. Now let’s prepare to deploy the VMs for our OKD cluster by preparing the Cluster VM inventory file:

    This is not an ansible inventory like you might have encountered with OKD 3.11. This is something I made up for my lab that allows me to quickly create, manage, and destroy virtual machines.

    I have provided an example that will create the virtual machines for this deployment. It is located at ./Provisioning/guest_inventory/okd4_lab. The file is structured in such a way that it can be parsed by the utility scripts provided in this project. The columns in the comma delimited file are used for the following purposes:

    Column Name Description
    1 KVM_HOST_NODE The hypervisor host that this VM will be provisioned on
    2 GUEST_HOSTNAME The hostname of this VM, must be in DNS with A and PTR records
    3 MEMORY The amount of RAM in MB to allocate to this VM
    4 CPU The number of vCPUs to allocate to this VM
    5 ROOT_VOL The size in GB of the first HDD to provision
    6 DATA_VOL The size in GB of the second HDD to provision; 0 for none
    7 NUM_OF_NICS The number of NICs to provision for thie VM; 1 or 2
    8 ROLE The OKD role that this VM will play: BOOSTRAP, MASTER, or WORKER
    9 VBMC_PORT The port that VBMC will bind to for IPMI control of this VM

    It looks like this: (The entries for the three worker nodes are commented out, if you have two KVM hosts with 64GB RAM each, then you can uncomment those lines and have a full 6-node cluster)

    # kvm-host02,okd4-worker-0,20480,4,100,0,1,WORKER,6233
    # kvm-host02,okd4-worker-1,20480,4,100,0,1,WORKER,6234
    # kvm-host02,okd4-worker-2,20480,4,100,0,1,WORKER,6235

    Copy this file into place, and modify it if necessary:

    mkdir -p ${OKD4_LAB_PATH}/guest-inventory
    cp ./Provisioning/guest_inventory/okd4_lab ${OKD4_LAB_PATH}/guest-inventory
  3. Retrieve the oc command. We’re going to grab an older version of oc, but that’s OK. We just need it to retrieve to current versions of oc and openshift-install

    Go to: and retrieve the openshift-client-linux-4.4.0-0.okd-2020-01-28-022517.tar.gz archive.

  4. Uncompress the archive and move the oc executable to your ~/bin directory. Make sure ~/bin is in your path.

    tar -xzf openshift-client-linux-4.4.0-0.okd-2020-01-28-022517.tar.gz
    mv oc ~/bin

    The script will pull the correct version of oc and openshift-install when we run it. It will over-write older versions in ~/bin.

  5. Now, we need a couple of pull secrets.

    The first one is for If you don’t already have an account, go to and create a free account.

    Once you have your account, you need to extract your pull secret.

    1. Log into your new account.
    2. In the top left corner, click the down arrow next to your user name. This will expand a menu
    3. Select Account Settings
    4. Under Docker CLI Password, click on Generate Encrypted Password
    5. Type in your password
    6. Select Kubernetes Secret
    7. Select View <your-userid>-secret.yml
    8. Copy the base64 encoded string under .dockerconfigjson

      It will look something like:


      But much longer…

    9. We need to put the pull secret into a JSON file that we will use to mirror the OKD images into our Nexus registry. We’ll also need the pull secret for our cluster install.

      echo "PASTE THE COPIED BASE64 STRING HERE" | base64 -d > ${OKD4_LAB_PATH}/pull_secret.json 

    So, that’s the first pull secret. The second is for our local Nexus install. We need to push the mirrored images into our Nexus, and the OKD install will need to pull them out.

    1. Create the pull secret for Nexus. Use a username and password that has write authority to the origin repository that we created earlier.

      echo -n "admin:your_admin_password" | base64 -w0
    2. Copy the resulting string, and edit the pull_secret.json file we created above.

      vi ${OKD4_LAB_PATH}/pull_secret.json

      The file should look something like:

      {"auths": {"": {"auth": "Y2dydXZlcREDACTEDrNmpiL34MVZHeGt2YpREDACTEDptNVlxS9ZTFIUREDACTEDM4bmZB", "email": ""}}}

      Add an entry for your Nexus secret, so that the file looks like this:

      {"auths": {"": {"auth": "Y2dydXZlcREDACTEDrNmpiL34MVZHeGt2YpREDACTEDptNVlxS9ZTFIUREDACTEDM4bmZB", "email": ""},"": {"auth": "PASTE_NEW_SECRET_HERE", "email": ""}}}

      Save this file.

  6. We need to pull a current version of OKD. So point your browser at

    OKD Release

    Select the most recent 4.4.0-0.okd release that is in a Phase of Accepted, and copy the release name into an environment variable:

    export OKD_RELEASE=4.4.0-0.okd-2020-03-23-073327
  7. The next step is to prepare our install-config.yaml file that openshift-install will use to create the ignition files for bootstrap, master, and worker nodes.

    I have prepared a skeleton file for you in this project, ./Provisioning/install-config-upi.yaml.

    apiVersion: v1
    baseDomain: %%LAB_DOMAIN%%
      name: %%CLUSTER_NAME%%
      networkType: OpenShiftSDN
      - cidr: 
        hostPrefix: 23 
      - cidr:
    - name: worker
      replicas: 0
      name: master
      replicas: 3
      none: {}
    pullSecret: '%%PULL_SECRET%%'
    sshKey: %%SSH_KEY%%
    additionalTrustBundle: |
    - mirrors:
      - nexus.%%LAB_DOMAIN%%:5001/origin
    - mirrors:
      - nexus.%%LAB_DOMAIN%%:5001/origin

    Copy this file to our working directory.

     cp ./Provisioning/install-config-upi.yaml ${OKD4_LAB_PATH}/install-config-upi.yaml

    Patch in some values:

     sed -i "s|%%LAB_DOMAIN%%|${LAB_DOMAIN}|g" ${OKD4_LAB_PATH}/install-config-upi.yaml
     SECRET=$(cat ${OKD4_LAB_PATH}/pull_secret.json)
     sed -i "s|%%PULL_SECRET%%|${SECRET}|g" ${OKD4_LAB_PATH}/install-config-upi.yaml
     SSH_KEY=$(cat ~/.ssh/
     sed -i "s|%%SSH_KEY%%|${SSH_KEY}|g" ${OKD4_LAB_PATH}/install-config-upi.yaml

    For the last piece, you need to manually paste in a cert. No sed magic here for you…

    Copy the contents of: /etc/pki/ca-trust/source/anchors/nexus.crt and paste it into the blank line here in the config file:

    additionalTrustBundle: |

    You need to indent every line of the cert with two spaces for the yaml syntax.

    Your install-config-upi.yaml file should now look something like:

    apiVersion: v1
      name: %%CLUSTER_NAME%%
      networkType: OpenShiftSDN
      - cidr: 
        hostPrefix: 23 
    - name: worker
      replicas: 0
      name: master
      replicas: 3
      none: {}
    pullSecret: '{"auths": {"": {"auth": "Y2dydREDACTEDREDACTEDHeGtREDACTEDREDACTEDU55NWV5MREDACTEDREDACTEDM4bmZB", "email": ""},"": {"auth": "YREDACTEDREDACTED==", "email": ""}}}'
    additionalTrustBundle: |
      -----BEGIN CERTIFICATE-----
      -----END CERTIFICATE-----
    - mirrors:
    - mirrors:
  8. Now mirror the OKD images into the local Nexus:


    The output should look something like:

    Update image:
    Mirror prefix:
    To use the new mirrored repository to install, add the following section to the install-config.yaml:
    - mirrors:
    - mirrors:
    To use the new mirrored repository for upgrades, use the following to create an ImageContentSourcePolicy:
    kind: ImageContentSourcePolicy
      name: example
      - mirrors:
      - mirrors:
  9. Create the cluster virtual machines and set up for OKD installation: -i=${OKD4_LAB_PATH}/guest-inventory/okd4 -p -m -n=okd4

    This script does a whole lot of work for us.

    1. It will pull the current versions of oc and openshift-install based on the value of ${OKD_RELEASE} that we set previously.
    2. fills in the OKD version and %%CLUSTER_NAME%% in the install-config-upi.yaml file and copies that file to the install directory as install-config.yaml.
    3. Invokes the openshift-install command against our install-config to produce ignition files
    4. Copies the ignition files into place for FCOS install
    5. Sets up for a mirrored install by putting into a DNS sinkhole.
    6. Creates guest VMs based on the inventory file at ${OKD4_LAB_PATH}/guest-inventory/okd4
    7. Creates iPXE boot files for each VM and copies them to the iPXE server, (your router)

We are now ready to fire up our OKD cluster!!!

  1. Start the LB

    ipmitool -I lanplus -H10.11.11.10 -p6228 -Uadmin -Ppassword chassis power on
  2. Start the bootstrap node

    ipmitool -I lanplus -H10.11.11.10 -p6229 -Uadmin -Ppassword chassis power on
  3. Start the cluster master nodes

    for i in 6230 6231 6232
      ipmitool -I lanplus -H10.11.11.10 -p${i} -Uadmin -Ppassword chassis power on
  4. Start the cluster worker nodes (If you have any)

    for i in 6233 6234 6235
      ipmitool -I lanplus -H10.11.11.10 -p${i} -Uadmin -Ppassword chassis power on

Now let’s sit back and watch the install:

Note: It is normal to see logs which look like errors while bootkube and kublet are waiting for resources to be provisioned.

Don’t be alarmed if you see streams of connection refused errors for a minute or two. If the errors persist for more than a few minutes, then you might have real issues, but be patient.

Log into your new cluster console:

Point your browser to the url listed at the completion of install: Log in as kubeadmin with the password from the output at the completion of the install.

__If you forget the password for this initial account, you can find it in the file: ${OKD4_LAB_PATH}/okd4-install-dir/auth/kubeadmin-password

Note: the first time you try to log in, you may have to wait a bit for all of the console resources to initialize.

You will have to accept the certs for your new cluster.

Issue commands against your new cluster:

export KUBECONFIG="${OKD4_LAB_PATH}/okd4-install-dir/auth/kubeconfig"
oc get pods --all-namespaces

You may need to approve the certs of you master and or worker nodes before they can join the cluster:

oc get csr

If you see certs in a Pending state:

oc get csr -ojson | jq -r '.items[] | select(.status == {} ) |' | xargs oc adm certificate approve

Create an Empty volume for registry storage:

oc patch cluster --type merge --patch '{"spec":{"managementState":"Managed","storage":{"emptyDir":{}}}}'

If it all goes pancake shaped:

openshift-install --dir=okd4-install gather bootstrap --bootstrap --master --master --master


  1. Create an Image Pruner:

    oc patch --type merge -p '{"spec":{"schedule":"0 0 * * *","suspend":false,"keepTagRevisions":3,"keepYoungerThan":60,"resources":{},"affinity":{},"nodeSelector":{},"tolerations":[],"startingDeadlineSeconds":60,"successfulJobsHistoryLimit":3,"failedJobsHistoryLimit":3}}'
  2. Designate your Master Nodes as Infrastructure Nodes

    Do Not do this step if you do not have dedicated worker nodes.

    If you have dedicated worker nodes in addition to three master nodes, then I recommend this step to pin your Ingress Routers to the Master nodes. If they restart on worker nodes, you will lose Ingress access to your cluster unless you add the worker nodes to your external HA Proxy configuration. I prefer to use Infrasturcture nodes to run the Ingress routers and a number of other pods.

  3. Set up Htpasswd as an Identity Provider
  4. Deploy a Ceph cluster for block storage provisioning
  5. Create a MariaDB Galera StatefulSet
  6. Updating Your Cluster
  7. Coming soon… Tekton pipeline for Quarkus and Spring Boot applications.
  8. Gracefully shut down your cluster