16:29:47 <jlebon> #startmeeting fedora_coreos_meeting
16:29:47 <zodbot> Meeting started Wed Jan 19 16:29:47 2022 UTC.
16:29:47 <zodbot> This meeting is logged and archived in a public location.
16:29:47 <zodbot> The chair is jlebon. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:29:47 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:29:47 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:29:56 <jlebon> #topic roll call
16:30:25 <aaradhak[m]> .hi aaradhak
16:30:26 <zodbot> aaradhak[m]: Sorry, but user 'aaradhak [m]' does not exist
16:30:52 <saqali> .hello saqali
16:30:53 <zodbot> saqali: saqali 'Saqib Ali' <saqali@redhat.com>
16:30:54 <jaimelm> .hello2
16:30:56 <zodbot> jaimelm: jaimelm 'Jaime Magiera' <jaimelm@umich.edu>
16:31:02 <dustymabe> .hi
16:31:03 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:31:04 <lsm540> .hello lsm540
16:31:06 <aaradhak[m]> .hello aaradhak
16:31:06 <zodbot> lsm540: Sorry, but user 'lsm540' does not exist
16:31:09 <zodbot> aaradhak[m]: aaradhak 'Aashish Radhakrishnan' <aaradhak@redhat.com>
16:31:16 <lsm540> i'm lsm5 btw .. my irc id looks weird
16:31:26 <jlebon> #chair aaradhak[m] saqali jaimelm dustymabe lsm540
16:31:26 <zodbot> Current chairs: aaradhak[m] dustymabe jaimelm jlebon lsm540 saqali
16:31:26 <bgilbert> .hi
16:31:26 <lsm540> .hello lsm5
16:31:26 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:31:29 <zodbot> lsm540: lsm5 'Lokesh Mandvekar' <lsm5@redhat.com>
16:31:31 <jmarrero> .hi
16:31:32 <baude> .hello baude
16:31:32 <zodbot> jmarrero: jmarrero 'Joseph Marrero' <jmarrero@redhat.com>
16:31:35 <zodbot> baude: baude 'Brent Baude' <bbaude@redhat.com>
16:31:43 <mheon> .hello mheon
16:31:44 <zodbot> mheon: mheon 'Matthew Heon' <mheon@redhat.com>
16:31:45 <aaradhak> .hi
16:31:46 <zodbot> aaradhak: aaradhak 'Aashish Radhakrishnan' <aaradhak@redhat.com>
16:32:00 <jlebon> #chair bgilbert jmarrero baude mheon
16:32:00 <zodbot> Current chairs: aaradhak[m] baude bgilbert dustymabe jaimelm jlebon jmarrero lsm540 mheon saqali
16:32:03 <fifofonix> .hi
16:32:04 <travier> .hello siosm
16:32:04 <zodbot> fifofonix: fifofonix 'Fifo Phonics' <fifofonix@gmail.com>
16:32:07 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:32:09 <jlebon> #chair fifofonix travier
16:32:09 <zodbot> Current chairs: aaradhak[m] baude bgilbert dustymabe fifofonix jaimelm jlebon jmarrero lsm540 mheon saqali travier
16:32:39 <jlebon> lsm540: need to /nick ?
16:33:03 <miabbott> .hello miabbott
16:33:04 <zodbot> miabbott: miabbott 'Micah Abbott' <miabbott@redhat.com>
16:33:09 <lsm540> dunno, i'd rather keep it as-is if it's ok
16:33:15 <jlebon> cool ok, let's wait half a minute more
16:33:22 <dwalsh_> Hello
16:33:29 <jlebon> #chaiir miabbott dwalsh_
16:33:36 <miabbott> off by one
16:33:38 <lsm540> jlebon i'm already logged in from matrix, that's causing conflicts with this session
16:33:48 <davdunc[m> .hi
16:33:48 <zodbot> davdunc[m: Error: Missing "]".  You may want to quote your arguments with double quotes in order to prevent extra brackets from being evaluated as nested commands.
16:33:50 <jlebon> lsm540: ahh gotcha
16:33:53 <jlebon> #chair miabbott dwalsh_
16:33:53 <zodbot> Current chairs: aaradhak[m] baude bgilbert dustymabe dwalsh_ fifofonix jaimelm jlebon jmarrero lsm540 mheon miabbott saqali travier
16:33:58 <jlebon> #chair davdunc[m
16:33:58 <zodbot> Current chairs: aaradhak[m] baude bgilbert davdunc[m dustymabe dwalsh_ fifofonix jaimelm jlebon jmarrero lsm540 mheon miabbott saqali travier
16:34:00 <jlebon> miabbott: :)
16:34:06 <jlebon> ok let's start!
16:34:09 <jlebon> #topic Action items from last meeting
16:34:15 <jlebon> -ENOENT
16:34:30 <jlebon> great!
16:34:47 <travier> /!\ Matrix users: There might be lag with the bridge so consider joining via IRC just for this meeting
16:34:49 <jlebon> let's move to the first labeled topic
16:35:06 <jlebon> #topic podman v4.0, Fedora 35 and CoreOS
16:35:10 <jlebon> #link https://github.com/coreos/fedora-coreos-tracker/issues/1070
16:35:19 <jlebon> lsm540: do you want to introduce this one?
16:35:31 <lsm540> i'll let mheon go
16:35:42 <ravanelli> .hi
16:35:42 <lsm540> RE: breaking changes
16:35:42 <zodbot> ravanelli: ravanelli 'Renata Renata Andrade Matos Ravanelii' <renata.ravanelli@gmail.com>
16:35:59 <dwalsh> I will take a stab.
16:36:06 <mheon> in summary: we're about to release a new major release of podman, v4.0.0
16:36:07 <lsm540> go ahead dan
16:36:13 <dwalsh> mheon, go
16:36:29 <lorbus> .hi
16:36:30 <mheon> this release includes some breaking changes to our REST API, hence the major version bump
16:36:30 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com>
16:36:51 <jlebon> #chair lorbus
16:36:51 <zodbot> Current chairs: aaradhak[m] baude bgilbert davdunc[m dustymabe dwalsh_ fifofonix jaimelm jlebon jmarrero lorbus lsm540 mheon miabbott saqali travier
16:36:51 <mheon> as such, we won't be including this in fedora 35 and older, only rawhide and eventually 36 once that lands
16:37:17 <mheon> however, this introduces a problem with podman-machine, which we use for running podman on OS X, by means of an FCOS VM
16:37:36 <lucab> .hi
16:37:37 <zodbot> lucab: lucab 'Luca BRUNO' <lucab@redhat.com>
16:37:52 <mheon> as noted, there were breaking changes in the remote API, so Podman v3.x.x cannot be used with Podman v4.x.x, either as a client or server.
16:38:13 <travier> Does anything else apart from the REST API has breaking changes?
16:38:25 <mheon> OS X podman is mostly distributed via homebrew, and we have very little control of the update pace, so it will update rapidly to v4.0.0 once that releases
16:38:42 <mheon> travier: no. there were extensive changes to the network stack but they should not be user-facing.
16:38:44 <jlebon> hmm ok. and remind me again. does podman machine hardcode a version or just brings up the latest stable and let's it auto-update?
16:38:53 <dwalsh> We are changing the entire Network Stack. Some potential changes in command line json fields as well.
16:39:04 <mheon> jlebon: we allow stream selection, but default is latest stable
16:39:07 <mheon> no hardcoding afaik
16:39:16 <mheon> though i'll defer to baude for technical details on machine
16:39:26 <miabbott> #chair lucab
16:39:26 <zodbot> Current chairs: aaradhak[m] baude bgilbert davdunc[m dustymabe dwalsh_ fifofonix jaimelm jlebon jmarrero lorbus lsm540 lucab mheon miabbott saqali travier
16:39:31 <jlebon> #chair ravanelli
16:39:31 <zodbot> Current chairs: aaradhak[m] baude bgilbert davdunc[m dustymabe dwalsh_ fifofonix jaimelm jlebon jmarrero lorbus lsm540 lucab mheon miabbott ravanelli saqali travier
16:39:57 <dwalsh> Bottom line is we don't want to wait for fedora 36 to update package on Fedora CoreOS.
16:40:03 <travier> Can we ship a version that has both APIs for a while?
16:40:10 <baude> there is no hard coding, but there is an API comparison
16:40:20 <baude> travier, no
16:40:41 <dustymabe> hmm. I see two problems here we need to consider
16:40:46 <baude> that version would still have to be "in" the fcos vm anyway
16:40:48 <dwalsh> We will make packages available for Fedora 35 but they will not be released.  So if people want them, they will have to manually download them.
16:41:01 <mheon> we'd like to improve API compatibility in the future, try and prevent this situation, but with the current architecture we don't have time before v4.0 release
16:41:05 <dustymabe> 1. how do we help the podman team achieve their goals with podman-machine/FCOS
16:41:18 <dustymabe> 2. how to we handle the breaking changes when FCOS stable moves to podman 4.0
16:41:25 <lsm540> can coreos consume koji builds that are not in bodhi updates?
16:41:34 <dustymabe> lsm540: we can
16:41:46 <baude> i believe we handle the breaking changes in podman itself
16:41:46 <travier> but I don't think we should
16:41:57 <dustymabe> we'll want to consider all options and choose the best one
16:42:15 <travier> Does the docker API compat has any breaking change? that's the one most likely to be used today
16:42:29 <baude> for #2, when thhe version inside and outside are 4.x, we handle the breaking changes, upgrades, etc
16:42:39 <mheon> negative, the docker API continues to support docker v1.40
16:42:53 <mheon> the changes are to the podman-specific side of the API
16:43:08 <baude> and travier that also is not true ... when you call podman images on the mac, that uses the podman api ... so all cli from the mac is podman api based
16:43:09 <dwalsh> The issue from our point of view, is MAC Users are chafing at the bit, for new version.  Docker starts charging at end of January.  So lots of people are looking for an improved Podman experience
16:43:29 <travier> We generally try to have a 3 months warning period for major changes in FCOS
16:43:45 <baude> fwiw, this issue isnt just for major versions, we have the same issue with .Y releases, takes weeks to sync versions
16:44:00 <dustymabe> travier: yes, for stable.. I think `next` is there for more experimental changes
16:44:07 <mheon> less serious for minor releases, since the APIs generally match
16:44:08 <travier> baude: sorry, I meant "used in FCOS" outside of the macOS use case
16:44:19 <baude> ok, yeah
16:44:21 <jlebon> dustymabe: yeah, that's what I was thinking too, re. `next`
16:44:21 <dustymabe> so the question is, would we want to be able to ship podman 4.0 earlier in `next`?
16:44:37 <baude> dustymabe, that would be somewhat ideal
16:45:15 <travier> How far are we from F36 beta?
16:45:32 <dustymabe> travier: I think branching is happening sometime soon (like next week or something)
16:45:39 <dustymabe> let me check the date for beta
16:45:45 <travier> Beta release Tue 2022-03-15
16:45:46 <dwalsh> BTW, this is probably not just Podman, some of the packages it relies on will get upgraded as well, although those can be released to F35 (crun, runc, slirp4netns ...)
16:45:48 <jlebon> beta freeze is feb 22
16:46:03 <baude> dwalsh, netavark and aardvark too
16:46:10 <jlebon> beta release is mar 15
16:46:17 <dwalsh> Yup, we could release those to f35
16:46:34 <travier> We generally rebase next when the beta is released right?
16:46:50 <dustymabe> travier: my proposal isn't to rebase next now
16:47:01 <dustymabe> it would be to pick up a podman 4.0 built against f35
16:47:13 <dwalsh> Sure, what we want is f35 bits with newer Podman 4.0
16:47:13 <baude> dustymabe, ++
16:47:15 <dustymabe> i.e. built in koji but not released to the rest of Fedora
16:47:24 <dwalsh> Agreed
16:47:27 <lsm540> dustymabe that sounds great to me
16:47:44 <dustymabe> to be clear, i'm not voting for or against it right now, just playing with the idea
16:47:47 <travier> Sure, but this is making us clearly not Fedora anymore as we would be shipping something not ever released in Fedora
16:48:10 <dustymabe> ehh, i'm not sure it's that black/white
16:48:22 <dustymabe> we shipped a systemd package in f34 that was never released in bodhi
16:48:31 <dustymabe> did that make us not Fedora anymore?
16:48:32 <dwalsh> Think of it as a Rawhide release of Podman, so it will be "released" in Fedora.
16:48:34 <bgilbert> if podman on mac used FCOS `next` for the next couple months, it would be getting builds that haven't baked in public the way `stable` has
16:48:34 <jlebon> will the side podman build be fully supported?
16:48:47 <dwalsh> jlebon, yes
16:48:49 <jlebon> i.e. if our users hit issues, can we count on the podman team to fix things?
16:49:01 <lsm540> yes
16:49:06 <bgilbert> and then if we rebase `next` to 36 in March, it'd be getting relatively bleeding-edge stuff
16:49:12 <dustymabe> bgilbert: I think they use `testing` today, so it's probably already true
16:49:37 <dwalsh> The Container team will be more concentrating and fixing issues on Podman 4.0 then we will on 3.4
16:49:41 <dustymabe> at least for the "getting builds that haven't baked in public the way `stable` has"
16:50:11 <jlebon> there's also walters' idea of having podman machine do an override replace
16:50:13 <travier> baude said they default to stable right now
16:50:21 <dustymabe> travier: oh nice
16:50:35 <dustymabe> maybe I was operating off of old information
16:51:10 <travier> > <mheon> jlebon: we allow stream selection, but default is latest stable
16:51:13 <dwalsh> We will change the default in podman 4.0 release.
16:51:18 <baude> dustymabe, travier i would need to double check .. at one point we did default to testing, and then it was just for aarch64
16:51:29 <baude> dwalsh, exactly
16:51:39 <baude> dwalsh, and then down the line, we can flip back
16:51:51 <dustymabe> ok so let's wrangle this discussion back to proposals
16:51:53 <jlebon> override would also allow users to keep selecting the stream they want
16:51:55 <mheon> i don't think swapping to a more recent stream is a blocker for us, it's not that difficult to swap back and forth
16:51:58 <dwalsh> Sure once podman 4.* stablizes.
16:51:59 <jlebon> instead of being forced to use next
16:52:13 <jlebon> not sure how much they care about the actual stream used though
16:52:15 <travier> I don't think the comparison with the systemd cse stands as this is a much bigger change. I'm more in favor of override
16:52:31 <dustymabe> jlebon: so your proposal is to have rpm-ostree override replace on first boot?
16:52:36 <baude> dwalsh, we could put a little logic in there too
16:52:50 <bgilbert> travier: the original design for `next` had it shipping -rc kernels.  I'm not too concerned about the "not Fedora" part for next
16:52:53 <dwalsh> dustymabe, The problem with that is it is slow...
16:52:58 <baude> very slow
16:52:59 <jlebon> dustymabe: right yeah, it seems like the simplest releng wise. (though it's walters' proposal to be clear :) )
16:53:18 <baude> and it delays usage of the vm, while users wonder what is going on
16:53:35 <dustymabe> jlebon: but then we'd need a repo somewhere maintained (couldn't use fedora repos because it's not released in Fedora)
16:53:49 <lsm540> how about a copr ?
16:53:56 <jlebon> dustymabe: that shouldn't be hard to resolve
16:53:58 <dustymabe> right something like that
16:54:13 <lsm540> copr would be doable for podman team
16:54:19 <bgilbert> baking podman 4 in `next` seems like a good use of `next` IMO
16:54:33 <jlebon> baude: re. very slow, are you talking about the override replace approach?
16:54:41 <walters> Yeah I'm good with `next`
16:55:05 <jlebon> also, what's the idea with stream selection? do you expect a lot of users to select a different stream than the default?
16:56:09 <jlebon> bgilbert: my only minor concern is the beta rebase part of it
16:56:14 * dustymabe has to run go pick someone up - I'm +1 for using next, but will note it means podman users are going to get f36 at beta time, so better be ready for that.
16:56:19 <bgilbert> jlebon: agreed
16:56:19 <mheon> to handle situations like this, i believe - we faced something similar early on with a lack of aarch64 mainline builds
16:56:37 <dustymabe> i.e. podman team needs to make sure they are happy with rawhide, otherwise they won't be happy with f36
16:56:47 <dwalsh> jlebon, When we default to stable, users might want to take advantage of new features, so they would change to next.
16:57:19 <jlebon> dwalsh: so what the override approach provides is the stability of stable with just podman swapped out, which you control
16:57:51 <jlebon> it doesn't even have to be a either/or thing. i could imagine doing both (i.e. also ship in next first)
16:57:54 <dwalsh> But with the penalty of every user everywhere has to wait for a yum update to complete, with 0 feedback.
16:58:15 <dustymabe> jlebon: +1 for "both"
16:58:22 <travier> Could it be a rpm-ostree usroverlay and rpm -i?
16:58:52 <dustymabe> basically they can use `next` but if something breaks in `next` they'll need to be able to handle it by selecting a different stream and doing an override
16:59:12 <bgilbert> dwalsh: are you willing to accept a stability tradeoff to avoid that yum update?
16:59:14 <jlebon> dwalsh: i think we can optimize that. e.g. podman machine could only enable the single repo it needs
16:59:17 <travier> If next breaks and they have podman 4 on the host, they have no other options
16:59:38 <dustymabe> travier: see my last comment, does that help?
16:59:44 <jlebon> and then `apply-live` to avoid rebooting
17:00:01 <dustymabe> jlebon: does apply live work with override replace?
17:00:11 <jlebon> it does, you have to pass in a switch
17:00:19 <dustymabe> hmm, that seems dangerous :)
17:00:43 <jlebon> it's been pretty good honestly :)
17:00:49 * dustymabe leaves now
17:01:07 <dwalsh> How unstable has next been in the past.  If next == Rawhide, then probably no.  If next == updates-testing then probably yes.
17:01:33 <dwalsh> We really only rely on systemd, kernel and Podman on the fcos.
17:01:34 <jlebon> next is testing right now, but will be f36 beta at beta release
17:01:41 <dustymabe> next is going to be equal to f36 beta in 2 months
17:01:45 <dustymabe> that's about as unstable as it gets
17:03:17 <jlebon> ok so, a few options here so far:
17:03:18 <baude> i suppose a one off with f35 and podman4 is out of question
17:03:53 <jlebon> 1. ship v4 in next now
17:04:05 <jlebon> 2. ship v4 in a repo, and have podman machine do an override
17:04:12 <jlebon> 3. do both of those things
17:04:40 <travier> (Maybe we should write a Fedora 36 change too for all of that. But the deadline is just passed.)
17:04:49 <jlebon> baude: and not update?
17:05:00 <baude> and then update when things sync up
17:05:15 <jlebon> side FCOS builds doesn't sound great
17:05:24 <bgilbert> baude: => 3+ months without security updates
17:05:41 <travier> I'd prefer we ship in next instead of something custom
17:06:12 <jlebon> baude: do you want to investigate the override approach? we can help out to see how to optimize it
17:06:16 <dwalsh> We can ship in next and then have until march 15 to figure out what to do...
17:06:25 <bgilbert> dwalsh: true
17:06:55 <jlebon> yup, that sounds ok with me too
17:07:12 <dwalsh> How bad was it when f35 branched.  Were there kernel and systemd issues?
17:07:16 <bgilbert> the next scheduled FCOS release is Feb 1
17:07:39 <baude> jlebon, i would like to know more about that regardless of this issue
17:08:16 <jlebon> dwalsh: i don't recall too well tbh, but I don't think we had anything major break
17:08:48 <travier> Fedora releases are getting more stable so I don't remember anything major breaking with the latest beta
17:09:02 <travier> There was a btrfs bug but we don't use that
17:09:16 <jlebon> baude: ack sounds good
17:09:28 <jlebon> ok, let me try a proposal
17:10:00 <travier> The override approach as several benefits: We can actually test it in CI, you can ship whatever podman you want on all streams.
17:10:19 <jlebon> #proposed we will ship podman v4 based on f35 in next for now and meanwhile investigate other approaches before the beta rebase
17:10:34 <baude> seems reasonable
17:10:53 <bgilbert> it sounds as though they'd like podman 4 in the image before Feb 1 though
17:11:02 <bgilbert> s/in the image/available/
17:11:09 <jlebon> is that so?
17:11:49 <mheon> my expected final release date was the 1st exactly, assuming no slips
17:11:50 <dwalsh> No Podman 4 is only rc1.
17:12:08 <jlebon> ok cool
17:12:21 <jlebon> in that case, we'll need to coordinate if we want to catch that week's release
17:12:24 <bgilbert> yup
17:12:44 <bgilbert> in a pinch, maybe we could do a one-off build and then you could switch back to `next` after the following release?
17:13:31 <bgilbert> if the schedules badly misalign
17:13:35 <jlebon> we could also do an ad-hoc next release but not roll it out
17:13:52 <bgilbert> jlebon: it'd still affect new launches though, fwiw
17:14:02 <bgilbert> oh, you mean not update stream metadata
17:14:26 <jlebon> at least the release index would still need updating though
17:14:30 <jlebon> anyway, circling back
17:14:38 <jlebon> acks for the proposal above?
17:14:50 <bgilbert> +1
17:14:55 <travier> +1
17:15:01 <miabbott> +1
17:15:12 <nemric> I give my vote
17:15:42 <jlebon> ok, i'll assume dwalsh is +1 since he suggested it :)
17:15:46 <jlebon> #agreed we will ship podman v4 based on f35 in next for now and meanwhile investigate other approaches before the beta rebase
17:15:47 <lsm540> +1 from me too
17:16:14 <jlebon> cool. anything else to discuss on this topic today?
17:16:16 <bgilbert> good discussion, thanks all
17:16:25 <nemric> Podman 4 or 4 rc1 .
17:16:27 <nemric> ?
17:16:52 <jlebon> nemric: 4 ideally :)
17:17:00 <nemric> right
17:17:10 <jlebon> ok, let's move on to the next topic
17:17:17 <jlebon> thanks podman team for joining!
17:17:18 <lsm540> thanks all
17:17:31 <jlebon> #topic Certificate expired for https://updates.coreos.fedoraproject.org
17:17:34 <jlebon> #link https://github.com/coreos/fedora-coreos-tracker/issues/1072
17:17:34 <travier> thanks for all the info!
17:17:51 <jlebon> bgilbert: you tagged this. want to take it?
17:18:10 <bgilbert> I don't know more than what I've read, but sure
17:18:11 <travier> I think this is to raise awareness
17:18:39 <bgilbert> the TLS cert expired for the production Cincinnati update server
17:18:43 <lucab_> there is movement going on on -infra
17:19:05 <bgilbert> it should be fixed now, but AIUI there are other staging certs etc. which are still being fixed
17:19:20 <baude> jlebon, dustymabe thanks for the consideration
17:19:24 <jlebon> ok cool. wasn't sure if it was more than just for awareness
17:19:55 <bgilbert> jlebon: also I'll second what you said in the ticket that we should RCA this
17:19:58 <jlebon> baude: of course!  and feel free to discuss in #fedora-coreos re. the other approaches
17:20:15 <nemric> bgilbert: My grafana don't show me cert warnings anymore....
17:20:36 <nemric> It should be fixed
17:20:39 <jlebon> ok cool. so let's all stay tuned for developments.
17:20:44 <lucab_> it still isn't completely fixed
17:21:04 <jlebon> travier: do you think we have time for https://github.com/coreos/fedora-coreos-tracker/issues/1062 or should we do it next week?
17:21:26 <travier> jlebon: next week will be good
17:21:57 <jlebon> ok cool. there's a few other meaty tickets, but I don't think we have enough time
17:22:17 <jlebon> so i'd suggest we go to open floor unless someone has an topic they want to bring up
17:22:20 <jlebon> a*
17:22:21 <davdunc> jlebon: still working on the kernel bug.
17:22:34 <fifofonix> observation: impressive how the rollback worked seamlessly last week (i think that was last week).  thanks.
17:22:45 <jlebon> davdunc: ack, thanks for the update
17:23:21 <jlebon> fifofonix: cool :)
17:23:21 <bgilbert> fifofonix: +1
17:23:25 <jlebon> #topic Open Floor
17:23:49 <lucab_> fifofonix: a local rollback or the stream tweak?
17:23:57 <fifofonix> the stream 'tweak'.
17:25:06 <jlebon> it's cool when we get to exercise the flexibility we have with the release process
17:25:07 <lucab_> ack (sorry just making sure there was no unintended rollback)
17:25:50 <jlebon> 99% of the time it's always the same thing (release rollouts), but that 1% is fun
17:26:28 <fifofonix> it is good to test these things once in a while!  local rollbacks also awesome (used once before, forget why).
17:26:53 <jlebon> +1 sweet
17:27:03 <jlebon> hopefully one day we'll get around to automatic rollback :)
17:27:37 <jlebon> ok, closing the meeting in 60s unless anyone has anything else
17:28:30 <nemric> I have to learn git more deeply :D but that's my own personal bug
17:28:37 <jlebon> #endmeeting