16:30:04 <miabbott> #startmeeting fedora_coreos_meeting
16:30:04 <zodbot> Meeting started Wed Oct 10 16:30:04 2018 UTC.
16:30:04 <zodbot> This meeting is logged and archived in a public location.
16:30:04 <zodbot> The chair is miabbott. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:04 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:04 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:10 <miabbott> #topic roll call
16:30:10 <lorbus> .hello2
16:30:11 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com>
16:30:13 <dustymabe> .hello2
16:30:14 <ajeddeloh> .hello2
16:30:14 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:30:15 <slowrie> .hello2
16:30:17 <zodbot> ajeddeloh: ajeddeloh 'Andrew Jeddeloh' <andrew.jeddeloh@redhat.com>
16:30:20 <zodbot> slowrie: slowrie 'Stephen Lowrie' <slowrie@redhat.com>
16:30:20 <geoff-> geoff-: Geoff Levand  <geoff@infradead.org>
16:30:22 <yzhang> .hello2
16:30:26 <zodbot> yzhang: yzhang 'Yu Qi Zhang' <jzehrarnyg@gmail.com>
16:30:32 <mskarbek> .hello2
16:30:33 <zodbot> mskarbek: mskarbek 'None' <redhat@skarbek.name>
16:30:40 <miabbott> #chair lorbus dustymabe ajeddeloh slowrie geoff- yzhang mskarbek
16:30:40 <zodbot> Current chairs: ajeddeloh dustymabe geoff- lorbus miabbott mskarbek slowrie yzhang
16:30:43 <jbrooks> .fas jasonbrooks
16:30:44 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com>
16:30:45 <ksinny> .hello sinnykumari
16:30:47 <zodbot> ksinny: sinnykumari 'Sinny Kumari' <ksinny@gmail.com>
16:31:06 <miabbott> #chair jbrooks ksinny
16:31:06 <zodbot> Current chairs: ajeddeloh dustymabe geoff- jbrooks ksinny lorbus miabbott mskarbek slowrie yzhang
16:31:23 <jlebon> .hello2
16:31:24 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:31:45 <miabbott> #chair jlebon
16:31:45 <zodbot> Current chairs: ajeddeloh dustymabe geoff- jbrooks jlebon ksinny lorbus miabbott mskarbek slowrie yzhang
16:31:50 <bgilbert> .hello2
16:31:51 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:31:57 <miabbott> #chair bgilbert
16:31:57 <zodbot> Current chairs: ajeddeloh bgilbert dustymabe geoff- jbrooks jlebon ksinny lorbus miabbott mskarbek slowrie yzhang
16:32:09 <bhavin192> .hello2
16:32:10 <zodbot> bhavin192: bhavin192 'Bhavin Gandhi' <bhavin7392@gmail.com>
16:32:15 <miabbott> #chair bhavin192
16:32:15 <zodbot> Current chairs: ajeddeloh bgilbert bhavin192 dustymabe geoff- jbrooks jlebon ksinny lorbus miabbott mskarbek slowrie yzhang
16:32:20 <rfairley> .hello rfairleyredhat
16:32:21 <zodbot> rfairley: rfairleyredhat 'Robert Fairley' <rfairley@redhat.com>
16:32:28 <miabbott> #chair rfairley
16:32:28 <zodbot> Current chairs: ajeddeloh bgilbert bhavin192 dustymabe geoff- jbrooks jlebon ksinny lorbus miabbott mskarbek rfairley slowrie yzhang
16:32:46 <mskarbek> where I can fix that "None" in the name? I have my full name in FAS. :/
16:32:51 <kaeso> .hello2 lucab
16:32:52 <zodbot> kaeso: Sorry, but you don't exist
16:33:05 <kaeso> .hello lucab
16:33:06 <zodbot> kaeso: lucab 'Luca Bruno' <lucab@redhat.com>
16:33:11 <miabbott> #chair kaeso
16:33:11 <zodbot> Current chairs: ajeddeloh bgilbert bhavin192 dustymabe geoff- jbrooks jlebon kaeso ksinny lorbus miabbott mskarbek rfairley slowrie yzhang
16:33:20 <miabbott> we'll get started in about 1m
16:33:36 <kaeso> (I'm briefly afk)
16:34:18 <dustymabe> mskarbek: i'm not actually sure
16:34:30 <dustymabe> :(
16:34:30 <miabbott> #topic Action items from last meeting
16:34:41 <miabbott> bgilbert to write up a proposal in the ticket (i.e. your description + modifications based on our conversations) and we can all read it and vote on it in the ticket #22
16:35:14 <jligon> .hello jligon
16:35:15 <zodbot> jligon: jligon 'Jeff Ligon' <jligon@redhat.com>
16:35:26 <miabbott> #chair jligon
16:35:26 <zodbot> Current chairs: ajeddeloh bgilbert bhavin192 dustymabe geoff- jbrooks jlebon jligon kaeso ksinny lorbus miabbott mskarbek rfairley slowrie yzhang
16:35:50 <miabbott> bgilbert: it looks like that action got completed with ticket #22?
16:36:50 <bgilbert> yup
16:36:51 <miabbott> dustymabe: do you know? ^^
16:36:55 <miabbott> ah, ok
16:37:01 <miabbott> moving on then
16:37:02 <bgilbert> #info https://github.com/coreos/fedora-coreos-tracker/issues/22#issuecomment-427218716
16:37:08 <miabbott> #topic set up logging for #fedora-coreos IRC channel #11
16:37:13 <miabbott> #link https://github.com/coreos/fedora-coreos-tracker/issues/11
16:37:28 <miabbott> dustymabe: assigned to you ^^
16:37:43 <dustymabe> I just added a comment to the ticket
16:37:59 <dustymabe> https://github.com/coreos/fedora-coreos-tracker/issues/11#issuecomment-428639803
16:38:07 <bgilbert> dustymabe: seems a bit unclear?
16:38:29 <bgilbert> dustymabe: legal says that avoiding long-term storage would help, but we should get someone else to do it?
16:38:29 <dustymabe> bgilbert: mostly: 'steering clear of hosting our own instance as part of the Fedora project is still the recommendation from legal'
16:38:58 <dustymabe> mostly that it "might help". I think they'd have to do some more research
16:39:07 <mnguyen_> .hello2 mnguyen
16:39:08 <zodbot> mnguyen_: Sorry, but you don't exist
16:39:15 <mnguyen_> .hello mnguyen
16:39:16 <zodbot> mnguyen_: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:39:23 <ajeddeloh> ^ love that error message
16:39:25 <miabbott> #chair mnguyen_
16:39:25 <zodbot> Current chairs: ajeddeloh bgilbert bhavin192 dustymabe geoff- jbrooks jlebon jligon kaeso ksinny lorbus miabbott mnguyen_ mskarbek rfairley slowrie yzhang
16:39:35 <bgilbert> what's the next step then?
16:39:46 <dustymabe> I think there is a decent level of 'we're scared of what we don't know and this can only bring us pain' involved
16:39:50 <ajeddeloh> dustymabe: is this something where we can ask "but seriously, can we do this"
16:40:14 <dustymabe> yep. we can start more formal conversations around the topic
16:40:28 <dustymabe> there are a few things we need to figure out
16:40:32 <dustymabe> 1. can we do it legally
16:40:40 <ajeddeloh> because to my knowledge there aren't any other 3rd party ones that I know of
16:40:42 <dustymabe> 2. would fedora infra be willing to host an instance
16:40:49 <dustymabe> 3. if not are we willing to host our own?
16:41:03 <ajeddeloh> hosting our own is worth it imo
16:41:13 <ajeddeloh> if fedora infra wont
16:41:30 <bgilbert> for the uninitiated, what does "hosting our own" mean in the context of Fedora?  some individual going and doing it?
16:42:00 <jbrooks> dustymabe, my team could provide a vm for hosting, if fedora infra won't
16:42:02 <ajeddeloh> long running cloud instance that reboots at some time for updates?
16:42:23 <ajeddeloh> it's not like it needs a particularly beefy instance
16:42:26 <dustymabe> bgilbert: I think host our own would mean we "coreos group" hosts own infra/server for this purpose
16:42:42 <dustymabe> another option is an individual chooses to implement it themselves
16:43:20 <dustymabe> i don't think the hosting/implementation is really the hard part at all
16:43:25 <dustymabe> it's mostly the legal aspects
16:43:27 <lorbus> if Fedora Infra were to host tho, wouldnt that be preferable?
16:43:42 <dustymabe> lorbus: yeah, i mean anyone to manage something we don't have to :)
16:43:45 <ajeddeloh> i think so, yes
16:43:49 <lorbus> and with Fed Infra, we'd probably get the best legal scrutiny
16:44:24 <lorbus> otherwise (personal instance) someone would personally be responsible for it...which I wouldnt do
16:44:25 <miabbott> i wonder if you could setup a cron job to just start/end a meeting in the channel every day...
16:45:15 <ajeddeloh> lorbus: if we can get a container that does it, just set up a cl instance and once we have a stable fcos switch to that
16:45:21 <ajeddeloh> test of the "set and forget"
16:45:32 <dustymabe> maybe. I don't know if that would actually be a good UI though
16:45:35 <bgilbert> yeah, I'm still wondering why recording meetings is okay
16:45:55 <dustymabe> I'll start an email thread with bgilbert + ajeddeloh and legal
16:46:05 <ajeddeloh> +1
16:46:08 <lorbus> sure, but legally, someone would still be responsible for any wrongdoings under GDPR
16:46:31 <miabbott> #action dustymabe  start an email thread with bgilbert + ajeddeloh and legal about logging #fedora-coreos in fedora infra or otherwise
16:46:49 <miabbott> #topic Major release and update cycle for Fedora CoreOS #22
16:46:55 <miabbott> #link https://github.com/coreos/fedora-coreos-tracker/issues/22
16:47:21 <miabbott> nobody assigned, but bgilbert made a detailed comment most recently
16:47:31 <miabbott> s/comment/proposal/
16:48:57 <miabbott> i guess we need folks to read it and leave feedback?
16:48:59 <mskarbek> bgilbert: one could argue that meeting is a "formalized" activity you are choosing to participate and by that agree to the conditions - also recording, normal irc conversation dosn't have any formal requirements to participate
16:49:42 <bgilbert> mskarbek: maybe.  we do warn people in the topic that the channel is logged though
16:49:58 <ajeddeloh> Something to note: we change the details of what the CL release channels mean occasionally. I think we ought to reserve the right to do the same with FCOS
16:49:58 <bgilbert> miabbott: sure, or we can discuss here
16:50:15 <ajeddeloh> because we probably won't get it perfect from the start
16:50:29 <miabbott> let's disucss
16:50:33 <miabbott> discuss*
16:50:40 <bgilbert> ajeddeloh: yeah.  I mentioned that the cadences are not contractual but I should broaden that language
16:50:54 <dustymabe> so one bikeshed - on stable/testing/next names
16:51:19 <dustymabe> i wonder if we should rename testing to something else so as to not confuse with bodhi updates-testing
16:51:48 <bgilbert> I'm fine with that
16:51:50 <kaeso> bgilbert: re. deprecation, weren't we planning to have chokepoint and dead-leaves via cincinnati? (to simplify the graph and avoid year-long deprecations)
16:51:55 <ajeddeloh> bgilbert: +1 I think just maybe draw a distinction between cadence and channel meaning
16:51:59 <kaeso> *chokepoints
16:52:18 <bgilbert> ajeddeloh: ?
16:52:28 <dustymabe> i also think it might be beneficial to rename the "development refs" from "updates" to "bodhi-updates" and "updates-testing" to "bodhi-updates-testing"
16:52:47 <dustymabe> just to try to further remove any confusion there
16:52:48 <lorbus> dustymabe: candidate?
16:52:50 <ajeddeloh> bgilbert: cadence refers to frequency, but not necessarily the meaning of the channels
16:53:24 <bgilbert> ajeddeloh: not sure if I'm understanding.  I was thinking of saying that both cadence and definition are non-contractual
16:53:47 <ajeddeloh> sgtm, we're in agreement
16:54:13 <bgilbert> kaeso: I realized after I wrote the comment that it doesn't take Cincinnati into account at all
16:54:17 <dustymabe> if we can't think of another really good name to replace 'testing' then let's just leave it at 'testing' :)
16:54:31 <bgilbert> kaeso: I have a mostly-written draft issue for update rollouts
16:54:57 <bgilbert> kaeso: I'd say let's only consider stream structure for now and think about rollouts separately
16:55:07 <kaeso> bgilbert: ack, I'll check that later on then
16:55:20 <lorbus> dustymabe: not sure if it's a good alternative,  but it could be renamed to "candidate"
16:55:38 <lorbus> (or something similar)
16:56:04 <dustymabe> lorbus: that could work, but a little long :)
16:56:57 <lorbus> rc..but that term could be confusing ppl
16:57:21 <dustymabe> rc ?
16:57:22 <bgilbert> I'll think on it a bit, even if we don't come up with a name now
16:58:01 <lorbus> it kinda is a release-candidate stream isnt it?
16:58:16 <ajeddeloh> I (don
16:58:46 <ajeddeloh> 't want to get into name bikeshedding now) but also thing we should avoid names that have other connotations like alpha, beta, rc, etc
16:58:59 <dustymabe> oh oh rc == release candidate
16:59:02 <dustymabe> got it :)
16:59:07 <lorbus> I dont feel strongly about this, tho. I'd be fine with leaving it as is
16:59:17 <lorbus> ajedeloh +1
16:59:18 <bgilbert> other non-naming comments?
16:59:23 <lorbus> ajeddeloh +1
16:59:36 <dustymabe> yeah. testing is good - especially if we rename the dev ref to 'bodhi-updates-testing'
16:59:53 * ksinny is +1 for bodhi-updates and bodhi-updates-testing
17:00:03 * dustymabe done with naming comments :)
17:00:31 <ksinny> but wonders which Fedora version we are talking, at a time there can be bodhi updates for multiple Fedora stream
17:00:42 <jbrooks> latest stable
17:01:10 <ksinny> okay, that's reasonable
17:01:33 <bgilbert> the proposal says the Fedora version currently tracked by `testing`
17:02:02 <ksinny> +1
17:02:16 <bgilbert> on the theory that if users have a problem with stable, they try testing, and if they still have a problem, they might try bodhi-*
17:02:48 <miabbott> sounds like we have good agreement on the proposal.  do we want to keep it open (on GH) for further discussion?
17:03:02 <lorbus> +1 for the bodhi- prefixes
17:03:17 <bgilbert> there are a couple tweaks from this discussion
17:03:20 <jbrooks> But bodhi-* could be tracking, today f28 or f27, but I'm assuming we care only about the latest bodhi-stable release
17:03:25 <dustymabe> I'll add the comment about the bodhi prefixes to the ticket
17:03:34 <bgilbert> after that, I can go directly to design doc PR, or we can do another round here
17:03:53 <bgilbert> dustymabe: no need, I'll update the proposal from this discussion and add a comment
17:04:32 <miabbott> #action bgilbert update ticket #22 with naming choices for development refs (bodhi-*) etc.
17:04:44 <bgilbert> jbrooks: for clarity, were you responding to my comment?
17:05:02 <miabbott> #topic  Determine how to handle automatic rollback #47
17:05:08 <miabbott> #link https://github.com/coreos/fedora-coreos-tracker/issues/47
17:05:36 <jbrooks> bgilbert, Yes, I think ksinny was pointing out that there are two stable fedoras out at any one time
17:05:55 <jbrooks> I'm assuming we're always talking about the more recent one
17:05:57 <ksinny> yeah
17:06:01 <bgilbert> jbrooks: right, and I was saying we'd track the one referenced by `testing`
17:06:10 <bgilbert> which may or may not be the current one
17:06:15 <jbrooks> oh, interesting
17:07:35 <ajeddeloh> so rollback: lorbus you worked on some boot-counting stuff right? Do you think there's a reason to give an install more than 1 try?
17:09:01 <dustymabe> install == an upgraded deployment ?
17:09:04 <lorbus> ajeddeloh: I don't know honestly. It was a feature that had been desired by different people for some time
17:09:13 <ajeddeloh> dustymabe: yeah
17:09:33 <ajeddeloh> bgilbert: keep me honest, but with CL we only have one shot
17:09:43 <dustymabe> i think that was configurable based on the work lorbus was doing (i.e. boot counting)
17:09:52 <dustymabe> the number of tries is just a variable
17:09:54 <kaeso> ajeddeloh: don't we have 3 retries?
17:09:58 <ajeddeloh> we do?
17:10:00 <dustymabe> can/should be able to be configured
17:10:37 <jlebon> yeah, esp. if we allow user-configured health checks
17:10:41 <lorbus> It is intended to eliminate any flakes/things that would usually go right, but just do not in that boot attempt
17:10:47 <kaeso> ajeddeloh: I was somehow arguing for a "more than one" on the basis that a node reboot / power failure / cosmic ray can happen
17:10:47 <bgilbert> pretty sure CL is one shot, yeah
17:11:11 <lorbus> kaeso: exactly that
17:11:58 <ajeddeloh> ok, that's a good reason, what happens if there's a race though?
17:12:02 <lorbus> using grub2, it'll be possible to configure how many tiems to try
17:12:11 <lorbus> times*
17:12:14 <ajeddeloh> like an install comes in, loses, wins, then loses
17:12:42 <lorbus> as implemented in grub2/greenboot it is not an atomic op, though
17:13:15 <kaeso> ajeddeloh: a win would be sticky I'd say
17:13:17 <jlebon> on the first win, then we stop retrying, right?
17:13:44 <lorbus> yes, sticky win
17:14:21 <jlebon> lorbus: have you been able to use the grub2 patches and see boot counting in action?
17:14:28 <ajeddeloh> sgtm, just wanted to be clear.
17:14:30 <lorbus> essentially, to use the grub2 boot counting feature, one must set boot_success=0 and boot_counter=x grubenv vars after rpm-ostree upgrade, before rebooting
17:14:31 <lorbus> https://github.com/projectatomic/rpm-ostree/issues/1598#issuecomment-428218699
17:14:55 <ajeddeloh> lorbus: those changes are all in grub configs and not the grub executables, yes?
17:15:22 <walters> .hello2
17:15:23 <zodbot> walters: walters 'Colin Walters' <walters@redhat.com>
17:15:47 <lorbus> jlebon: no. arithmetic ops weren't actually supported in grub2 until last week, when pjones implemented it. My code was merged a bit prematurely and was actually ignored by grub2-mkconfig
17:16:04 <miabbott> #chair walters
17:16:04 <zodbot> Current chairs: ajeddeloh bgilbert bhavin192 dustymabe geoff- jbrooks jlebon jligon kaeso ksinny lorbus miabbott mnguyen_ mskarbek rfairley slowrie walters yzhang
17:16:14 <jlebon> lorbus: ahh heh, fun
17:16:26 <lorbus> there is a PR open, that is supposed to finally get it working: https://github.com/rhboot/grub2/pull/42
17:16:47 <lorbus> but for now the script has been removed from grub in the current package build
17:17:17 <ajeddeloh> one of the things we've discussed previously is having a static, handwritten grub config that knows how to pick which install to use
17:17:31 <lorbus> ajeddeloh: yes. its all grubenv vars that are being changed
17:17:48 <walters> somewhat related, does anyone disagree that we don't want the "skip the bootloader unless failure" part?
17:17:50 <jlebon> hmm, i'm wondering if there's a way to use more grubenv vars to not have to manually set boot_success=0 after `upgrade`... e.g. maybe also store some deployment marker for which boot_success=1 was set
17:17:56 <walters> (for servers)
17:18:16 <dustymabe> walters: i don't want to ever skip the bootloader for servers
17:18:22 <dustymabe> IMHO
17:18:26 <jlebon> and the in the grub config if the marker doesn't match, we know it's a new deploymenht
17:18:27 <lorbus> walters +1
17:19:16 <jlebon> walters: "skip the bootloader" meaning the menu?
17:19:23 <walters> right
17:19:32 <lorbus> TBH I think grub is a monster and it's a good thing they want to replace it in the long term
17:19:33 <jlebon> if so, then yeah we should just never hide it
17:20:17 <lorbus> that doesn't help us right now, tho. and with grub not being able to edit the BLS filenames, grubenv is really the only thing I can see solving this right now
17:20:48 <bgilbert> flag files, potentially
17:21:22 <lorbus> boot_success was actually introduced with the menu_hide feature. they want to use a 2min timer unit to set it on the Workstation
17:22:01 <lorbus> but the counting feature doesnt rely in that code
17:22:06 <lorbus> on*
17:22:18 <walters> yeah that's the primary driver of the logic today, which is why i lean towards backing it all out until we have a tested design
17:22:40 <walters> (backing it out for FCOS/FAH, not Workstation obviously)
17:22:52 <ajeddeloh> sorry, network is dying
17:23:58 <lorbus> I believe it is also going to be used in Fedora IoT, which could be nice testing ground for us as well
17:24:10 <dustymabe> yes that will be nice
17:24:12 <lorbus> being rpm-ostree based and all
17:24:34 <ajeddeloh> not sure if this message made it through: lorbus thoughts on having a static, handwritten grub config then storing all state in the grubenv
17:24:53 <walters> +1 to that from me
17:25:22 <ajeddeloh> and what bits of state does an install need?
17:25:22 <lorbus> ajeddeloh: I'm not opposed. You dont actually need boot_counter before using it, too.
17:25:37 <walters> (more importantly IMO, removing os-prober)
17:25:44 <miabbott> ...we've got about 5m left for this meeting...
17:26:03 <lorbus> jlebon: how would we do this atomically, i.e. without explicitly setting boot_counter?
17:26:22 <ajeddeloh> are writes to grub-env atomic?
17:27:01 * walters thinks about sector sizes here
17:27:05 <lorbus> ajeddeloh: not sure.
17:27:20 <ajeddeloh> i.e. do we need to worry about what happens if we lose power in the middle of trying to write to it
17:27:22 <walters> btw there was a fsdevel thread about this
17:27:32 <lorbus> walters: that's important, yes. Grub2 cant increase the size
17:27:55 <walters> https://marc.info/?l=linux-fsdevel&m=153741350128439&w=2
17:28:12 <jlebon> lorbus: yeah, that's an issue. e.g. if grub is interrupted between updating boot_success and some e.g. boot_success_marker
17:28:36 <jlebon> we can noodle on it some more in issues
17:28:49 <ajeddeloh> +1 on noodling in issues
17:28:59 <lorbus> +1, also next cabal
17:29:07 <ajeddeloh> I think we ought to tack down what state we need for each install
17:29:14 <lorbus> or maybe we actually have some way forward by then
17:29:16 <ajeddeloh> and how that should be represented in the grubenv
17:30:03 <lorbus> ajeddeloh: yes. during GSoC we made a state machine diagram
17:30:21 <lorbus> I'll update that to the current implementation asap
17:30:26 <miabbott> folks, we're at scheduled time for the meeting
17:30:47 <ajeddeloh> lorbus: keep in mind we also need to have some state that can differentiate two successful installs
17:31:30 * lorbus will think about this some more
17:31:31 <miabbott> do we want to move to open floor?  or just close the meeting?
17:31:51 * ajeddeloh has nothing for OF
17:32:07 <miabbott> #topic open floor
17:32:13 * lorbus neither
17:32:15 <miabbott> if you have anything, now's the time  :)
17:32:19 <dustymabe> I have something
17:32:28 <dustymabe> there is a meeting happening now to discuss coreos-assembler
17:32:45 <dustymabe> https://lists.fedoraproject.org/archives/list/coreos@lists.fedoraproject.org/thread/T44VRBNIB7GN3ESQDR2DYLQEOEOFS2MS/
17:34:38 <miabbott> last call for open floor...closing meeting in 1m
17:35:34 <miabbott> #endmeeting