Today the 50th edition of the weekly Flux Bug Scrub event was commenced and concluded! One of the members of Team Hephy, Kingdon, is also a Flux maintainer, and hosts these hour-long community events on alternating Wednesdays and Thursdays each week, an early and a late edition for different time zones. They’re easy to join and you can find them on the calendar with other regular Flux meetings. Hephy followers may have noticed that the monthly Open Roadmap meetings have been stopped since last year. Flux Bug Scrub was established in the spirit of those meetings, but with a different agenda.

Here I’ll tell a little bit about the format and why it is “this way,” whilst occasionally referring to myself in the third person as one of the NPCs that makes this event regularly possible with some good old face-to-face networking and some mechanized assistance, then explain how you can join!

This week marks the 50th time we have gathered to read FluxCD community issues and discussions: Flux maintainers and community members hanging out and informally following up on the little notes that people have left in our community forums. 50 Bug Scrubs means we have been doing this for over a year, given that a few times we have been on holiday or cancelled, it seems like a pretty good time to sit and write up a reflection on how it’s going, how it’s been, and what worked/what didn’t; shoot the shit, ya know, normal retro-type stuff to talk about once in a while, so we can all vibe together and try to get broadly better at how we’re doing?

We’ll also make a call to action, since this is not just our Bug Scrub, it’s also your Flux community, if you’re a Flux user or think you might like to try it out. Many eclectic topics under GitOps and cloud are discussed and we sometimes hold informal or semi-formal scheduled presenting sessions, where a Flux maintainer or someone else from the community joins to tell a story, deep-dive into an idea they’ve been itching to talk about, ask some questions, or even some strange thing we haven’t quite conceived of well enough yet to paint a picture of it here and now.

Often times I’m feeling drained and I think about cancelling Bug Scrub. Then I usually remember how good it makes us feel to connect with our community, and I rarely do actually follow through and cancel it. I made sure with automation that the activation energy required to start up Flux Bug Scrub each week would be low. Almost always someone shows up to read discussions with me, and we try to help whoever it is that posted the issue, whether they need help understanding Flux, or it’s a proposal that needs our consideration to decide whether it’s worth advancing, or if it needs re-work.

Usually we also have some random users who show up just to see what it’s about. Occasionally we get a repeat customer, but it’s a rotating door and that’s also fine. It’s really energizing to have that dichotomy of continuity and also subtle chaos. The docs all say you should RSVP, and nobody ever does, but now with increasing regularity, people are showing up … I think this may now be something like what one might call a “third place?”

The First Bug Scrub

Flux Bug Scrub was first created in homage to the Kubernetes SIG CLI Bug Scrub, that is another weekly or bi-weekly event that happens in a Zoom meeting and on Slack, and a SIG Node Bug Scrub that was a global special event hosted in 2021 with a piece of software called Triage Party. We didn’t attend any of those events, and we didn’t have experience with any of that global-scale team organizing or really know how to organize something like this, but Kingdon had a need for doing his new job to establish a regular habit of churning through a large number of issues.

It actually started with an assignment. Flux v1 was to be decommissioned. But not just yet. We had made a commitment to maintain Flux v1 while the new Flux v2 project was separately being undertaken and new problems dispatched. It proved challenging maintaining both projects with one team, so our team was expanded (I joined) and then bifurcated, as a single dev I was assigned to work on issues for the legacy project. A problem was that quite some time had passed since veteran maintainers already realized that developing an whole new Flux was necessarily going to take up 100% of their attention. So lots of issues looked abandoned.

The new guy had only just arrived on the scene, already familiar with Flux as a user and with some light duty answering support type issues for random drive-by Slack customers out in the open on the Weaveworks or CNCF Slack, formerly as a hobby/practice, but now professionally as part of day-job! But that didn’t make him an authority on anything else, and a great deal of research would be required to make any tangible progress.

A problem was this massive pile of stale issues that couldn’t just be thrown away. Flux users took time and care to write out those issues, many hours of research and discussion had been gathered in this space, and carefully curated by thoughtful and enthusiastic developers as well as Flux users. Nobody ever throws anything away. It seemed a terrible snub to simply close those issues unceremoniously without spending time reading each one carefully to consider whether it could really justifiably be closed. It was a challenge with a great value proposition.

We had committed to support Flux v1, but also written a support policy that emphasized Semantic Versioning and that meant no breaking changes in the same major series. Also many issues came from Flux v2 users that simply landed on the wrong repo. Maybe their issue was being addressed, but they’d need to upgrade. After feature parity was reached in the rewritten v2, we started advising everybody to upgrade. Predictably, many people didn’t immediately like that very much.

It became fast clear that to triage all the issues and/or eventually read and handle fairly or close every open issue, in order to honor our commitments to support Flux v1, and to build up overall knowledge about the system, the first step was to be boiling an entire ocean.

The task was clear even when the direction or a next important landmark was not immediately certain at first. It was vaguely important to get a real handle “on all the issues” first, in whatever order, so important details that may have been explored somewhere else could be considered thoughtfully before responding to each complex issue layered into other issues, and to avoid frustrating some already slightly agitated users from recently being told their important issue has been placed on indefinite hold, while they’ll also be made to endure breaking changes soon.

That much meant that “getting a handle” on the big picture behind all the issues was important before significant inroads on tangible progress could be made, and also so the New Guy could provide a more reliable estimate on how long it would take to close every last issue and honor our commitments to the end.

This path we’d be trailblazing should also provide that one day New Guy can be a productive contributor on the rest of the modern Flux v2 project, a fast-moving target that was actually going to be a lot more difficult to wrap our brain around than the old Flux v1 monolith, which was in a steady-state by comparison that might as well have been measured in Kelvins as it finished cooling off, rather than to use a proper semver.

Inventing a “Flux Bug Scrub” Method

A problem was that “getting a handle on all the issues” meant first lining them up from end to end by some metric in a way that wouldn’t be disturbed by thumbing through the list how oldest to newest might, or most time since last responded to, which again becomes a moving target once we start responding again (IDK, tallest to shortest?) – the GitHub interface did not provide well for this experience. If we sat down to work on “all the bugs” we could sort them by oldest-newest or most-recent-update to least-recent, but neither of these really made sense on their own without blending somehow. We don’t want to be always responding to newest issues, as actively supporting users on the legacy project was going quickly out of scope, but we always like to be helpful… people will notice and find a way to take advantage even if we don’t realize it.

So he decided on a spreadsheet. It would have to be machine-generated, probably once a week. The first few Flux Bug Scrub events didn’t have any supportive technology outside of what GitHub provided for us, and that was pretty limited. It became clear as soon as he tried to scale the problem beyond a single project repo, the Flux v1 repo, that scanning issues across many repositories in a somewhat orderly fashion was a task better suited to a spreadsheet than the GitHub.com chrome. There was a strong temptation to put more information into the spreadsheet, since we already identified that making notes on an issue directly would “bump” the thread, and it wasn’t really appropriate for the New Guy to spend all day reminding people who may have already moved on about the stale issues they filed and hey, are you still using this thing, please upgrade?

No. We couldn’t spam the users just to build up our own mental model and network I mean “schmooze” them into upgrading. That was out.

A piece of software was developed, flux-pr-review (still private today) which does two functions, broadly speaking. This builds a new spreadsheet with only the currently most relevant issues and mashes the “Updated At” and “Created At” date columns together with a logarithmic scale in order to come up with a single metric for “Stale-Age” that we can consider as mostly fair for sorting and managing a cursor, without a bias away from repeat visiting issues we’ve already seen before (until we’ve seen them all at least once.) A spreadsheet is a model supporting that.

That’s one function, building the new spreadsheet with metadata, and the other function is to carry over the metadata from last week’s spreadsheet if any issues that we wrote our notes on haven’t closed out yet. We’ve made some notes – we should get to keep them. We’re moving quickly from one issue to the next in Flux Bug Scrub, and any other time we’re following this process. We might not have time or opportunity to do anything other than read the issue and internalize something that we read, but that should still advance the cursor.

Not all issues can be moved ahead immediately, and “bumping” the thread without adding anything is a valueless process as we’ve established (with the risk of adding negative value, by pissing off someone whose issue still isn’t fixed simply by reminding them that it isn’t yet fixed… compounded by the New Guy who hasn’t even taken time to understand it thoroughly before asking questions? No, we must not…)

You can’t imagine the process that I went through to design this method.

I don’t want to be spammed even by someone whose goal is to be helpful, so I’d better not be that guy. We’ll use all the tools and take in as much data as we can to become expert first – then make a small contribution here and there and people will appreciate we’ve taken the time.

“And soon we’ll have closed all the issues.”

And indeed the process seemed to be working pretty well! But method and process isn’t everything, and there’s a certain character of intention and thoughtfulness that isn’t able to be captured in any way when writing process documentation, even when we’ve tried our best to do so.

Going Beyond Methodology

Part of Bug Scrub is about doing this work. We have this hour set aside every week to go into the issues list and make a dent so nobody feels like they’re dropping coins in a well and wasting their time reporting. People write us on the Discussions and Issues pages because they either have an idea, a problem, or a question – generally speaking one of those three. People who show up at Bug Scrub seemingly come for an entirely different reason. I’m writing today because I want to take out time to honor that reason! The reason is community feeling.

We like to belong to a community, and we especially like to be part of a community where we feel like we belong. At least, New Guy likes that, and it’s definitely one of the unspoken reasons for him starting Bug Scrub, at least it has been unspoken up to this point. I mean sure, we have this amazing technology product called Flux, and it’s designed to support this incredible delivery methodology called GitOps, with a goal to promote stable, resilient infrastructure that’s also flexible, extensible, progressive. And that’s what it’s really all about!

We’re here for the technology, much more than we’re here for the community. It’s kind of the point, we’re building something very real.

But also let’s not take ourselves too seriously. We are all here for each other, that’s a better point. Let’s all be excellent to each other.

A Call For Presenters

We can all present something. Let’s make it a community thing. I’m calling you out. You’ve got something awesome that you learned or made recently and you wanna show us. I know you do. We’re going to hear from you, I hope you just show up and let us have it. But if we’re not really that tight of friends yet, or you’d rather put something on the calendar in advance, we can always do that. You’ll be welcome to take up one of the regularly scheduled times, we don’t have to do spreadsheet every week.

Or we can also put you down for a different time. There’s lots of room on the calendar, for you to show us your stuff.

I’ll keep a time slot open for you. We can record the talk (or we don’t have to), it’ll go on YouTube if you want, then you can put it on your résumé and everyone will be impressed with your presenting skills, (we always are!) We get to hear your story, and you get to tell it! A win-win for all.

I’m asking everyone to dig deep right now and look for stories, shell scripts, magic terminal tricks, editor extensions, if you think if there’s something burning or novel that you can only tell the world about by putting it on a screen share at the Flux Bug Scrub…

Then I do hope you come to Bug Scrub, please drop in and say hello. It’s ok to interrupt us, even if nothing is really burning.

Keeping Up

An important part of Flux community is also keeping up with the news, and that means new releases. Today we added Flux v0.35.0, a milestone that carries the first support for Flux to verify OCI manifest sources against a keyless CoSign signature database.

There is so much new technology flying around the Cloud Native space, and it can seem like things move very fast. The Flux blog every month publishes an update to cover everything. We don’t really make a point at Bug Scrub of giving “The Weekly Update” in any sense as a standard or regular part of Flux Bug Scrub; it just isn’t in the format.

If you read the blog, and the release notes, and if you’ve read the docs, and know how to search the docs, then you can keep informed about all the big things in Flux as they are happening. Test new features on the day they came out!

On the other hand, if you go to Bug Scrub and volunteer some of your time with us, it’s as though you’ve got one very large ear on the ground. A low rumble is building in the distance. You can triangulate and listen closely to help us find out where Flux is going to land next, through the lens of user issues and community feedback that can come from anywhere.

And I get it, maybe staying for the whole ceremony and spreadsheet thing wasn’t likely to be your jam. Say you’ve been at one or more Flux Bug Scrub meetings before and you quickly realized it’s a form of torture as watching someone fast reading every issue and discussion in open mic format, just isn’t really your groove.

We can dig it. We’d still like to hear from you, in the discussion forums or in person (at the Zoom meeting, or at KubeCon, or whatever).

Bug Scrub is a much lower traffic meetup than you probably think, really. I bet one day soon when we’re, like, super popular and everyone keeps coming up to meet us, that we’re just so busy with meeting new people we can’t even do Bug Scrub stuff, but right now it’s definitely not an issue.

You can just show up. Just stop in and say hi! You don’t need an agenda item or anything else to bring first.

Chopping Wood and Carrying Water

We can take a break from carrying water to say hello to everyone that joins the meeting. We don’t know if you’re lurking or if your microphone just doesn’t work. I’m not going to make it awkward. You can listen in, participation is encouraged but not required. Just know you’re welcome to stop us and say hello, introduce yourself, tell us what brings you to Bug Scrub if it’s your first time.

And if we don’t see you arrive whilst we’re caught up in what we’re doing, maybe it’s the work of Bug Scrub, or some side chat, please know that it’s not an intentional snub. I’ll definitely try to stop and say hi, I like to acknowledge everybody that comes, but I’m not always very good at noticing subtle things, especially like a new person on the Zoom call when I’m in full-screen sharing spreadsheet or a GitHub Issue page.

We’re all in this together though, even if I’m the one sharing my screen, it’s totally fine for someone else to do the work of greeting our newest guest, you can be the greeter too, even if you yourself are new here. Everyone can hopefully volunteer and do at least a small job like this, and many hands together make the work light! Thanks for your attention, and thanks for reading this far.

I hope that I’ll see you again next time!

Team Hephy uses Flux

We’re not a company so we didn’t feel it made sense to post ourselves on the Flux adopters list, but as a Flux maintainer and Team Hephy maintainer, it could be important for some people to be able to know that I have both interests. We use Flux to support Hephy Workflow development!