16:30:27 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:27 <zodbot> Meeting started Wed Dec 11 16:30:27 2019 UTC.
16:30:27 <zodbot> This meeting is logged and archived in a public location.
16:30:27 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:27 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:27 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:33 <dustymabe> #topic roll call
16:30:55 <jlebon> .hello2
16:30:56 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:31:13 <kaeso[m]> .hello lucab
16:31:14 <zodbot> kaeso[m]: lucab 'Luca Bruno' <lucab@redhat.com>
16:32:05 <dustymabe> .hello2
16:32:06 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:32:25 <ajeddeloh> .hello2
16:32:25 <zodbot> ajeddeloh: ajeddeloh 'Andrew Jeddeloh' <andrew.jeddeloh@redhat.com>
16:33:09 <mnguyen_> .hello mnguyen
16:33:10 <zodbot> mnguyen_: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:33:26 <ksinny> .hello sinnykumari
16:33:27 <zodbot> ksinny: sinnykumari 'Sinny Kumari' <ksinny@gmail.com>
16:33:36 <miabbott> .hello miabbott
16:33:37 <zodbot> miabbott: miabbott 'Micah Abbott' <miabbott@redhat.com>
16:34:08 <dustymabe> welcome all!
16:34:46 <dustymabe> #chair jlebon kaeso[m] ajeddeloh mnguyen_ ksinny miabbott
16:34:46 <zodbot> Current chairs: ajeddeloh dustymabe jlebon kaeso[m] ksinny miabbott mnguyen_
16:34:54 <dustymabe> #topic Action items from last meeting
16:35:06 <dustymabe> * ajeddeloh to create an ignition-validate container
16:35:36 <dustymabe> that's the only item we have from last meeting.. and I think from the meeting before we had something about sending a patch to crypto policies to remove python requirement from the rpm
16:35:44 <ajeddeloh> Still need to do that, but it's on my list
16:36:52 <dustymabe> +1
16:36:54 <dustymabe> thanks ajeddeloh
16:37:22 <dustymabe> We have one meeting topic and then a few misc items I want to bring up
16:37:31 <dustymabe> #topic behavior if no Ignition is provided
16:37:36 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/279
16:38:08 <dustymabe> so basically it's not obvious to users right now if an ignition config didn't get detected or used
16:38:52 <dustymabe> there was some discussion in the ticket about trying to do something so users don't end up in a confusing situation
16:39:22 <dustymabe> it ranged from halting the boot, to trying to let the user know of the possible error condition in the console-login-helper-messages
16:39:59 <ajeddeloh> I like the console-login-helper-messages idea
16:40:08 <ajeddeloh> failing the boot is a bad idea
16:40:31 <ajeddeloh> we can't fail in the initramfs since ssh keys may get fetched/added later
16:40:35 <jlebon> yeah, i like that too
16:40:59 <jlebon> basically say if (1) an Ignition config was provided and (2) if any provider SSH keys were found
16:41:00 <ajeddeloh> if Ignition ran and got a config it should still have the cached one in /run
16:41:26 <ajeddeloh> so [ -f /run/ignition.json ] should be enough to detect if it ran
16:42:04 <kaeso[m]> but even if it runs and gets a config it doesn't mean it picked the user one, right?
16:42:20 <ajeddeloh> hmm oh right
16:42:24 <ajeddeloh> base configs and all
16:42:38 <kaeso[m]> those indeed
16:43:13 <kaeso[m]> the hash we log right now, is that over the user-provided config only? Perhaps we can re-use that
16:43:38 <jlebon> could it write somewhere the sources it merged for the final config? e.g. /run/ignition.json.sources
16:43:42 <ajeddeloh> parsing logs is not _great_ but we could
16:44:18 <jlebon> you can log structured entries into the journal and it's not bad querying it back
16:44:30 <jlebon> we use that a bunch in rpm-ostree
16:44:33 <kaeso[m]> oh sorry, I don't mean parsing logs, I just mean persisting the hash (if that can be manually matched by a user)
16:44:50 <ajeddeloh> wait, the base config is on the machine
16:45:00 <ajeddeloh> just compare ignition.json to the base config
16:45:14 <dustymabe> sounds good to me.. assuming it doesn't get mangled in some way
16:45:27 <ajeddeloh> it does, since it gets rerendered
16:45:49 <ajeddeloh> but piping both through jq or similar would work
16:45:53 <jlebon> if we didn't have a base config, this wouldn't work though
16:46:14 <ajeddeloh> jlebon: how so
16:46:17 <ajeddeloh> "" == ""
16:47:05 <jlebon> ok, second counter:  :)
16:47:21 <jlebon> if the user config is a subset of the base config, it'll just get merged into the same config, right?
16:47:32 <kaeso[m]> comparing JSON has the assumption that JSON can be sanely canonicalized, which is not true
16:47:33 <ajeddeloh> yes
16:48:03 <jlebon> feels cleaner to have a `/run/ignition-sources.json`, which would be useful in its own right
16:48:29 <jlebon> structured journal entries work too though
16:48:32 <ajeddeloh> would we include the base config in it?
16:48:38 <kaeso[m]> I'd rather just augment the logic in Ignition which is fetching the user config, in order to record the hash before merging
16:48:58 <jlebon> ajeddeloh: yeah, everything ignition used to derive the final config
16:51:36 <ajeddeloh> I dont have a strong opinion on ignition.sources vs logging the user hash
16:52:19 <kaeso[m]> jlebon: what did you plan to log, all config fragments?
16:52:29 <jlebon> i think it's mostly similar. basically have ignition record the sources and hashes from each source somewhere
16:52:55 <jlebon> kaeso[m]: oh no, just the source URIs and hashes
16:53:19 <jlebon> i think cloud-init had a similar thing
16:54:09 <dustymabe> ok so is there an RFE for ignition somewhere in here?
16:54:11 <jlebon> no strong opinions either where it gets recorded, as long as it's easily parseable too
16:54:33 <kaeso[m]> I guess we'd need something like a `kind` to know if a fragment is a user or a base one, too
16:55:08 <ajeddeloh> kaeso[m]: so the base doesn't have a uri
16:55:31 <ajeddeloh> we could just write "base" instead of a uri
16:55:40 <jlebon> ajeddeloh: file:///usr/lib/ignition/base.ign  ?
16:55:57 <kaeso[m]> ajeddeloh: the ISO embedded one too is coming from a file
16:56:52 <ajeddeloh> thats not a base config
16:56:56 <ajeddeloh> thats the user config
16:57:15 <ajeddeloh> unless I'm misunderstanding
16:57:35 <jlebon> i think we can discuss this in an ignition ticket
16:57:48 <kaeso[m]> ack
16:58:03 <ajeddeloh> +1
16:58:21 <dustymabe> who wants to open the ticket?
16:58:25 <kaeso[m]> ajeddeloh: I think I'm agreeing with your last proposal, I just worded my massage badly :)
16:58:59 <ajeddeloh> oh yeah
16:59:13 <ajeddeloh> we can just write file:/// uris since it always fetches the same ones
16:59:55 <dustymabe> #action jlebon to open an RFE for ignition to make logic for #279 easier
17:00:16 <jlebon> ack
17:00:24 <dustymabe> can you all voice your opinion in https://github.com/coreos/fedora-coreos-tracker/issues/279 about preference for console-login-helper messages approach?
17:02:34 <ajeddeloh> +1
17:02:48 <dustymabe> #topic OKD community asking for azure images
17:03:00 <dustymabe> #link https://github.com/openshift/community/projects/1#card-30394302
17:03:18 <dustymabe> A member of the OKD community was asking if there were azure FCOS images that could be used
17:03:48 <dustymabe> Since our images built by `cosa buildextend-azure` should be working, I added it as an output of the pipeline
17:04:00 <dustymabe> https://github.com/coreos/fedora-coreos-pipeline/pull/175
17:04:37 <dustymabe> #info The release that we're doing today should have azure images in it that people can try to upload and use
17:05:10 <dustymabe> anybody here want to volunteer to verify one of the images when we build it? /me really should try to dig up some azure creds
17:06:16 <miabbott> we have access to Azure creds (and knowledge) in the RHCOS/OCP program.  we could find a volunteer to help out from that side of the fence
17:06:32 <dustymabe> cool. thanks miabbott
17:06:58 <miabbott> you can action me if you like and i'll figure it out
17:07:16 <dustymabe> #action miabbott to help us get the azure image upload/boot tested
17:07:19 <dustymabe> thanks miabbott
17:07:24 <dustymabe> #topic open floor
17:07:32 <dustymabe> any other topics to discuss today?
17:07:56 <dustymabe> should we mention the issue that the last update could have caused to peoples systems?
17:08:38 <jlebon> yeah, that's probably a good idea
17:08:40 <dustymabe> #info the last update caused issues if you started from a moderately old FCOS. See https://github.com/coreos/fedora-coreos-tracker/issues/322
17:09:04 <dustymabe> you can workaround by running `sudo mount /dev/disk/by-label/boot /boot`
17:09:55 <jlebon> once the release kaeso[m] is doing is out, we should probably email that out too
17:10:43 <dustymabe> thanks kaeso[m] for doing this week's release!
17:11:04 <jlebon> kaeso++
17:11:17 <ksinny> kaeso++
17:11:17 <zodbot> ksinny: Karma for lucab changed to 2 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:11:56 <kaeso[m]> once release is out I'll try to unstuck my canary and send the step by email, sounds good?
17:13:22 <jlebon> SGTM
17:13:54 <dustymabe> anything else before we close out?
17:14:29 <kaeso[m]> #action lucab to send upgrade email with manual recovery steps
17:14:47 <dustymabe> +1
17:14:52 <dustymabe> will end meeting in 20
17:19:40 <dustymabe> #endmeeting