Find the official installation docs for v2.20 at teamhephy.com.
If you have any issues, please be sure to report it on the slack community, or report them on the issue tracker on GitHub.
Find the official installation docs for v2.20 at teamhephy.com.
If you have any issues, please be sure to report it on the slack community, or report them on the issue tracker on GitHub.
Hephy Workflow v2.20.1 (codename: “Semver and the Point Release”) is out!
This release comes with several warnings! NB read more below
Users all should be aware that there have been some issues reported since this was released on Thursday, February 14, and some known bugs may now affect users of the current and previous release. Users please read all of this information carefully, especially if you have a Hephy Workflow installation in production and want to keep it safe.
GKE users take special note of this upgrade, as we have accidentally broken some compatibility with the latest GKE releases, due to their lack of adherance with SemVer standards. This break was an oversight on our part; we have not made any emergency patch, as the issue is quite complex it will take time to communicate with upstreams, and ascertain precisely how to correctly resolve this safely and without causing any harm.
If you have already upgraded and are affected by this, you can take special steps and read this blog post for details, on how to safely roll-back the Controller to v2.20.0. This will hopefully mitigate the issue which may be preventing the Hephy controller from becoming healthy after the upgrade, or from normally answering Deis API requests when Workflow is deployed on GKE.
There will be a deep-dive blog episode, coming soon, to explain exactly how the beautiful postgres database backup works, and how we kind-of broke it during v2.20.0, just a little bit, just so nobody noticed over 3 months. tl;dr: it has been ever-so-slightly possible to accidentally (and potentially irreversibly) wipe out your database during an upgrade.
For users that installed v2.20.0 prior to the February 14, 2019 release - NB: This is a bug that would only affect a narrow set of users and use cases. (Don’t worry, it’s really not as bad as I made it out to sound. But gee Kingdon, wipe out your database does sound kind of serious, I guess you’re right, it sounds awful bad the way you started to explain it at first!)
Another general advice for Hephy Workflow v2.20.0 users: platform admins should remember to thoroughly understand backups and retention schedules, and always test your platform database backups before performing an upgrade. Test upgrades in your staging environment first, if you have one, and as always recommended according to best-practices advice, regularly test your backups in any case.
Hephy users should run helm repo update first, to begin to safely mitigate
the most dangerous of the currently known issues.
The buggy chart referenced in v2.20.0 (This is a bug) post linked above,
has been de-fanged and https://charts.teamhephy.com, our canonical charts
repo, is no longer actively dangerous. In this case we have actually
overwritten a slightly bad chart so that nobody will ever install it again.
For details about that issue, please read the Release v2.20.1 Postmortem.
This upgrade is not reversible, so you must take care. It would be wise,
and is stronly recommended, to take a snapshot of any database S3 buckets
first before upgrading, if feasible within your cluster’s particular setup.
The v2.20.1 patch represents a major upgrade for database users, and many important security fixes for all Hephy Workflow users. Users should upgrade as soon as possible. GKE cluster users will have to take special steps to receive the upgrades safely, until v2.20.2 is made available.
This release is tested and known to work with K8s clusters on Amazon Web Services, Azure Kubernetes Service, we also support users on Google Kubernetes Engine and Kubernetes on DigitalOcean Engine. Every release is always tested on Minikube, too.
Copyright © Team Hephy
Team Hephy accepts Issues and Pull Requests from any interested contributors! If there are any questions or problems, please reach us directly on slack or just open an issue on the github repository.
Under CC Attribution-ShareAlike 4.0 International License, you are free to share and adapt this document for any purpose.
Email: team@teamhephy.com
Kingdon: kingdon@teamhephy.com
Anton: anton@teamhephy.com