16:29:42 <travier> #startmeeting fedora_coreos_meeting
16:29:42 <zodbot> Meeting started Wed Jun 22 16:29:42 2022 UTC.
16:29:42 <zodbot> This meeting is logged and archived in a public location.
16:29:42 <zodbot> The chair is travier. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:29:42 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:29:42 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:29:51 <travier> #topic roll call
16:29:59 <travier> .hello siosm
16:30:00 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:30:41 <bgilbert> .hi
16:30:42 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:30:50 <Gris> .hi
16:30:51 <zodbot> Gris: Sorry, but user 'Gris' does not exist
16:30:56 <jlebon> .hello2
16:30:57 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:31:17 <ffmancera> .hi
16:31:18 <zodbot> ffmancera: ffmancera 'Fernando F. Mancera' <ferferna@redhat.com>
16:31:32 <travier> #chair bgilbert jlebon ffmancera Gris
16:31:32 <zodbot> Current chairs: Gris bgilbert ffmancera jlebon travier
16:33:29 <travier> low turn out :/
16:34:32 <qinqon> ./ I am from kubevirt network team and kubernetes-nmstate maintainer
16:34:40 <travier> #chari qinqon
16:34:44 <travier> Welcome!
16:34:51 <travier> #chair qinqon
16:34:51 <zodbot> Current chairs: Gris bgilbert ffmancera jlebon qinqon travier
16:34:57 <dustymabe> .hi
16:34:58 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:35:00 <jlebon> hi qinqon!
16:35:03 <travier> #chair dustymabe
16:35:03 <zodbot> Current chairs: Gris bgilbert dustymabe ffmancera jlebon qinqon travier
16:35:04 * bnemec lurks
16:35:12 <travier> #chair bnemec
16:35:12 <zodbot> Current chairs: Gris bgilbert bnemec dustymabe ffmancera jlebon qinqon travier
16:35:48 <travier> Let's get started. How about we start with nmstate as folks are there?
16:35:49 <mnguyen> .hi
16:35:50 <zodbot> mnguyen: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:35:54 <travier> #chair mnguyen
16:35:54 <zodbot> Current chairs: Gris bgilbert bnemec dustymabe ffmancera jlebon mnguyen qinqon travier
16:36:04 <travier> #topic Action items from last meeting
16:36:07 <nemric> o/
16:36:07 <jaime> hi, I am on the openshift networking team
16:36:18 <travier> #chair nemric jaime
16:36:18 <zodbot> Current chairs: Gris bgilbert bnemec dustymabe ffmancera jaime jlebon mnguyen nemric qinqon travier
16:36:51 <travier> Action items
16:36:51 <travier> cverna to open ticket for FCOS as an edition and update
16:36:51 <travier> jlebon to open investigation tickets for the IMA/FIDO changes
16:37:26 <travier> #action cverna to open ticket for FCOS as an edition and update
16:37:39 <travier> Re-actionning this one given that cverna what out for a while
16:37:46 <travier> jlebon: any update on yours?
16:38:00 <jlebon> travier: i did not do that yet, so we can re-action it
16:38:08 <travier> +1
16:38:17 <travier> #action jlebon to open investigation tickets for the IMA/FIDO changes
16:38:38 <travier> #topic New Package Request: nmstate-libs and nmstate
16:38:45 <travier> #link https://github.com/coreos/fedora-coreos-tracker/issues/1175
16:39:05 <travier> bgilbert maybe for the summary?
16:39:32 <bgilbert> sure
16:39:55 <bgilbert> there have been a couple changes since we last discussed this
16:40:42 <bgilbert> first, qinqon has contributed coreos-installer support for nmstate configs: https://github.com/coreos/coreos-installer/pull/864
16:40:59 <travier> 🎉
16:41:05 <travier> Awesome!
16:41:05 <bgilbert> this will allow coreos-installer iso/pxe customize commands to take a --network-nmstate option similar to --network-keyfile
16:41:19 <bgilbert> it'll pre-render the nmstate configs to keyfiles and embed them in the ISO/PXE image
16:41:46 <bgilbert> this allows nmstate configs to be applied to the live system (including the Ignition run) and the installed system (including the Ignition run).
16:42:03 <bgilbert> the downside is: because nmstate is being run off-node, it doesn't have an opportunity to validate the config on the running system.
16:42:19 <bgilbert> but the upside is: the resulting config affects the Ignition run.
16:42:29 <bgilbert> (other downside: bare-metal only)
16:42:58 <travier> So trade offs, but looks cool
16:43:20 <bgilbert> second: a separate use case is putting nmstate configs in Ignition and having them applied at runtime.
16:43:48 <bgilbert> this allows more validation, but by design, Ignition itself can't affect the Ignition run, so it'd only affect the real root.
16:44:28 <bgilbert> if we supported this, both features could be used together: use --network-nmstate to bootstrap just enough network config to get Ignition talking to internal servers, then put the rest of the nmstate config inside the Ignition config and allow it to be validated at runtime.
16:44:58 <jlebon> validation in that case happen post-pivot, right?
16:45:08 <bgilbert> yes
16:45:41 <bgilbert> the nmstate team has agreed to support an /etc/nmstate directory, and add a systemd unit to take its contents and apply them to the running system: https://github.com/nmstate/nmstate/pull/1936
16:45:48 <bgilbert> #link https://github.com/coreos/coreos-installer/pull/864
16:45:56 <bgilbert> #link https://github.com/nmstate/nmstate/pull/1936
16:46:21 <travier> Anything specific blocking at Ignition run time validation of nmstate files?
16:47:11 <bgilbert> travier: hmm, I don't know whether nmstate can validate nmstate configs without also applying them
16:47:11 <dustymabe> I see the validation as slightly less useful if it happens post pivot.. i.e. we make it to the real root anyway and the system could "limp along" in a half configured state.. that being said, it's really no different than someone offering keyfiles today
16:47:28 <jlebon> we could in theory chroot run nmstate to validate before pivoting
16:47:29 <bgilbert> dustymabe: that's the thing.  Ignition doesn't guarantee that the configs it's writing out are valid
16:47:47 <bgilbert> in general
16:47:56 <jlebon> bgilbert: good point
16:48:00 <dustymabe> yeah, just making notes
16:48:03 <bgilbert> +1
16:48:13 <travier> sure, but what if we could? (not saying we must)
16:48:25 <bgilbert> travier: if we could, then we can :-)
16:48:38 <travier> qinqon jaime WDYT?
16:48:41 <bgilbert> travier: that can also be added as a retrofit without a compat break IIUC
16:48:49 <bgilbert> since the config is already invalid, we're just catching it earlier
16:49:20 <bgilbert> one other note: --copy-network
16:49:29 <travier> Is it possible to validate nmstate config files from the initrd whitout applying them?
16:49:50 <travier> (question above for the nmstate folks)
16:49:56 <qinqon> travier: validation is done checking state after apply on the final system state
16:50:07 <bgilbert> AIUI (nmstate people, please correct me if I'm wrong), nmstate could be used at the live ISO prompt to configure the network, which would cause NM keyfiles to be written into /etc, and then the user could use coreoos-installer install --copy-network to propagate them to the installed system
16:50:10 <Gris> nmstate can do memory only apply and verification if NetworkManager provide dbus interface in intrd
16:50:10 <bgilbert> similar to the `nmtui` flow
16:50:38 <dustymabe> bgilbert: +1
16:50:58 <dustymabe> Gris: what does "memory only apply" mean here?
16:51:35 <Gris> do not write persistent network configuration, only ask NetworkManaget to set the network and nmstate do verification afterwords. Nmstate will rollback if verfifcation failed.
16:51:42 <bnemec> I think validating in initrd would be problematic because NetworkManager isn't actually running and neither are some other services like openvswitch that might be required by the config.
16:52:13 <dustymabe> Gris: right, but that does actually affect the NICs on the running system, it just doesn't write out the config to disk
16:52:19 <dustymabe> correct?
16:52:22 <Gris> dustymabe: correct
16:52:28 <dustymabe> +1
16:52:45 <qinqon> bgilbert: the --nmstate-network options would be the same `nmstatectl gc` + `--copy-network` I think
16:52:46 <dustymabe> ok I think we should separate the "validation in Ignition" thread into a later discussion
16:53:02 <dustymabe> or rather "validation in initramfs"
16:53:07 <bgilbert> +1 to separating out validation
16:53:08 <jlebon> Gris: re. https://github.com/nmstate/nmstate/pull/1936, i think that's the case but to confirm: after application, the .applied nmstate files are no longer used in any way and the generated NM keyfiles become source of truth?
16:53:08 <jaime> Can the live system run with a different set of network kernel arguments than the real root? Can this influence any validation in one place or the other?
16:53:11 <bgilbert> qinqon: +1
16:53:13 <travier> yes, looks like this is more complex
16:53:30 <ffmancera> Well, I think (please Gris correct me if I am wrong), that nmstate is doing early validation. So it will report failures if the nmstate config is wrong.. that should detect not compatible options, etc..
16:53:52 <bgilbert> jaime: network _kernel_ arguments are sorta out of scope here, I think.  we're generally trying to discourage their use
16:53:56 <bgilbert> jaime: but yes, they're independent
16:54:01 <Gris> jlebon: Correct. once applied(verified), it is NetworkManager's work/config to persistent the network config on reboot.
16:55:05 <bgilbert> so, since we now have a path to applying nmstate configs from the Ignition config, I propose that we consider shipping nmstate in FCOS
16:55:06 <Gris> ffmancera: Nmstate has two stage of checking. Pre-applied check for know limitations and typo, after-applied verification for kernel or hardware limitations.
16:55:51 <travier> 👍
16:56:00 <ffmancera> Gris: so in this case the pre-applied check for known limitations could be useful.. what do you think?
16:56:35 <travier> Maybe we should run that in Butane / Ignition as part of config validation?
16:57:06 <bnemec> That would already happen as part of the processing for --network-nmstate, right?
16:57:32 <Gris> ffmancera: `nmstatectl gc` already has pre-applied check, but of course without accessing host network current status, these check are limited.
16:57:33 <travier> via coreos-installer yes, but not for independently written config
16:57:34 <bgilbert> travier: that's relatively easy in coreos-installer because nmstate has a Rust crate.  in Go it'd be harder
16:57:35 <travier> s*
16:57:56 <qinqon> bnemec: yes same as `nmstatectl gc` does
16:57:59 <travier> bgilbert: yes, that's what I'm worried about too
16:58:17 <jlebon> Gris: +1 nice that simplifies the story
16:58:29 <bgilbert> I don't think we should necessarily expect a higher bar than already exists for deploying NM keyfiles
16:58:37 <Gris> travier: nmstate has go binding(wrapping from c library), not sure that helps or not.
16:58:46 <bgilbert> we can always document "here's how you validate nmstate configs"
16:59:09 <bgilbert> Gris: yeah, the issue is that the C library may not be installed on the system running Butane
16:59:26 <bgilbert> we support downloading the Butane bare binary and running it
16:59:34 <travier> Unfortunately it would need to be pure Go :/
16:59:42 <bgilbert> shipping nmstate would also allow it to be used by installed software to modify the network config on day 2, but I think that's less relevant to us
16:59:52 <travier> but recommending user check their config beforehand is also good eough
16:59:53 <bgilbert> (presumably useful to someone though)
16:59:57 <qinqon> bgilbert, travier: We can zip nmstate at butane ignition same that is done a terraform on openshift installer (I think)
17:00:19 <bgilbert> qinqon: the nmstate binary, you mean?
17:01:00 <dustymabe> bikeshed for later time: /etc/nmstate/ -> /etc/NetworkManager/nmstate/ ?
17:01:31 <qinqon> bgilbert: I understand this is to do pre-validation at ignition before apply it with nmstatectl apply at fcos right ?
17:01:54 <jlebon> we'll need to adjust our initrd teardown logic to also check for nmstate files before propagating initrd-generated NM keyfiles
17:01:58 <bgilbert> qinqon: it'd probably be at Butane time, not Ignition time
17:02:02 <travier> It's to do pre-verification before generating the Ignition configuration
17:02:13 <bgilbert> jlebon: ah, yes
17:02:23 <jaime> implementation detail note: do we need to think about selinux policies?
17:02:40 <bgilbert> Ignition itself is pretty low-level; it just writes the files it's told to
17:02:45 <jlebon> the messy thing here though is that it's an all or nothing thing which might make the bootstrap scenario you mentioned earlier more awkward bgilbert
17:02:46 <bgilbert> Butane is where higher-level validation happens
17:03:00 <travier> jaime: SELinux is enabled at initrd time in FCOS & RHCOS
17:03:21 <travier> but yes, probably not a concern
17:03:33 <travier> would be the same policy work as system runtime
17:03:46 <jlebon> travier: not exactly. labeling is done at initrd time. the policy is still loaded by systemd at pivot time
17:03:52 <qinqon> travier: the generated ignition from butane after validation will have the nmstate yamls that will be later applied right ?
17:04:00 <bgilbert> jlebon: hmm.  would it help for the coreos-installer sugar to record that it's injected nmstate-derived configs?
17:04:12 <travier> whoops then, my bad
17:04:24 <bgilbert> qinqon: right
17:05:06 <travier> qinqon: yes
17:05:16 <qinqon> bgilbert, travier: we have also static linking go->clib->rust if necessary so we end up with one butane binary
17:05:40 <bgilbert> qinqon: hmm, okay, we could consider that
17:06:00 <jlebon> bgilbert: on second thought, i think the same awkward issue exists today with NM keyfiles
17:06:02 <bgilbert> overall, more validation is always useful but I don't think it should be a blocker here
17:06:02 <travier> alright, it generaly feels like we don't have blockers here but are looking at the detalis to make things right. Should we make a proposal?
17:06:17 <bgilbert> jlebon: to what extent is that a blocker?
17:06:45 <jlebon> bgilbert: not a blocker
17:06:48 <bgilbert> I think it's still useful to support Ignition-delivered nmstate via, say, an autoconfigured management interface
17:07:02 <bgilbert> without any coreos-installer-delivered injection
17:07:31 <jlebon> the issue i'm thinking of is that if a keyfile was injected in e.g. the ISO, you'll need those to describe the full network state instead of partially because we can't easily tell whether what's in the rootfs vs what's in the initrd is conflicting or independent
17:08:01 <jlebon> which is why today we basically just don't propagate anything if keyfiles were written by Ignition
17:08:15 <bgilbert> hmm, couldn't it describe a partial network state, and then the nmstate config in the real root overwrites the entire thing?
17:08:18 <dustymabe> my rule has always been if any configuration exists in the real root we don't propagate what was in the initrd
17:08:43 <dustymabe> we should extend that to include nmstate configs
17:08:50 <jlebon> bgilbert: yes, in that case the nmstate config will have to repeat what was injected
17:08:59 <bgilbert> that doesn't seem objectionable
17:09:13 <bgilbert> #proposed We will add nmstate in Fedora CoreOS.
17:09:14 <bgilbert> :-)
17:09:33 <bgilbert> there are some details to work out, but we don't necessarily need to commit to them here (unless we want to)
17:09:34 <travier> +1
17:09:40 <bgilbert> s/in/to/
17:09:40 <jlebon> yup, just wasn't sure with your boostrap scenario if you were thinking each piece would bring just the parts it needs :)
17:10:04 <dustymabe> I'm +1 - but we do need to consider how we present this to users (for example, our documentation)
17:10:08 <bgilbert> interested in feedback from the nmstate team here, but I think it's probably cleaner to have the runtime config clobber the bootstrap one
17:10:33 <jlebon> +1 also from me (though... do we have a final tally on deps and sizes?)
17:11:29 <Gris> jlebon: only depend on glibc and size is 1.6M in Fedora 35.
17:12:09 <jlebon> bgilbert: but as i mentioned it's a consideration that applies today to NM keyfiles too
17:12:11 <Gris> jlebon: The nmstate-libs is 1.6M and nmstate is 6.3M.
17:12:27 <jlebon> Gris: ack, thanks
17:12:37 <bgilbert> jlebon: right
17:13:05 <dustymabe> ooft 1.6M was nice.. +6.3M not as much
17:13:17 <qinqon> bgilbert: feedback about how to present to users ?
17:13:32 <qinqon> dustymabe: we may be able to use some rust features to reduce size
17:13:37 <dustymabe> +1
17:13:38 <jlebon> dustymabe: about average for Rust applications
17:13:40 <bgilbert> qinqon: we should explicitly document the nmstate flows in the FCOS networking docs
17:13:50 <bgilbert> coreos-installer flow, Ignition flow, and the hybrid one
17:14:00 <bgilbert> maybe the --copy-network one
17:14:10 <qinqon> bgilbert: +1
17:14:12 <bgilbert> jlebon: +1
17:14:18 <bgilbert> any other votes?
17:14:43 <Gris> +1 not sure my vote counts.
17:14:46 <bgilbert> :-)
17:14:51 <travier> every vote counts :)
17:15:12 <qinqon> :)
17:15:27 <bgilbert> #agreed We will add nmstate to Fedora CoreOS.
17:15:36 <bgilbert> thanks for the discussion all, this was productive
17:15:43 <travier> Thanks for the productive discussion!
17:15:44 <jlebon> thanks a lot Gris and qinqon for all your work on this!
17:15:55 <ffmancera> Thank you!
17:16:10 <travier> Should we do another small one or move to open floor?
17:16:10 <qinqon> thanks to all the team is being nice to work things out here
17:16:15 <Gris> jlebon: Thank you! I got new stuff to learning after this meeting.
17:16:22 <travier> :)
17:16:48 <bgilbert> yup, and thanks Gris and qinqon for walking us through this
17:16:53 <bgilbert> travier: might be nice to discuss the pinger one
17:17:02 <travier> ok, let's do that
17:17:15 <travier> #topic Consider removing fedora-coreos-pinger
17:17:21 <travier> #link https://github.com/coreos/fedora-coreos-tracker/issues/770
17:17:28 <travier> Any taker for the summary?
17:17:30 <bgilbert> me
17:17:39 <bgilbert> now I get to argue in the other direction :-)
17:18:01 <bgilbert> fedora-coreos-pinger is the component that was supposed to handle metrics gathering for FCOS.
17:18:20 <bgilbert> this was before countme existed, and was intended to collect a broader set of metrics.
17:18:53 <bgilbert> this wasn't a priority before FCOS was generally released, but we were conscious of privacy concerns re collection of metrics, so the plan was:
17:19:25 <bgilbert> make sure the FCOS install documentation all included instructions on disabling metrics, for a future in which metrics collection existed.
17:20:11 <bgilbert> ship a component that only validated the config file, to prevent any surprises from showing up when the rest of the metrics collection tool was deployed.
17:20:40 <bgilbert> that's pinger.  there is currently no metrics backend, and the frontend only validates a config file describing whether the user wants metrics to be collected.
17:20:54 <bgilbert> ever since FCOS was first released, empirically it hasn't been a priority for us to build the rest of the tool.
17:21:20 <bgilbert> also, importantly, we never actually documented how to disable metrics, except in one place in the Container Linux migration docs.
17:21:28 <travier> #link https://docs.fedoraproject.org/en-US/fedora-coreos/migrate-cl/
17:21:37 <dustymabe> +1 remove
17:21:42 <travier> :)
17:21:42 <bgilbert> because of the latter, we haven't actually achieved our goal of giving advance warning that metrics collection may exist.
17:22:02 <bgilbert> meanwhile, pinger is unmaintained.  it needs to be ported from Travis to GH Actions, needs Dependabot, etc.
17:22:16 <bgilbert> instead, I'd like to archive the upstream repo, retire the Fedora package, and drop it from the distro.
17:22:35 <travier> +1
17:22:48 <bgilbert> none of that precludes changing our mind later, and it doesn't further increase the user notice requirements if we did so.
17:23:24 <bgilbert> #proposed We will remove fedora-coreos-pinger from FCOS and archive the repo upstream.
17:23:31 <travier> +1
17:23:33 <jlebon> i'm kinda sad about this, but understand. i think if someone showed up to drive this forward, it'd be of great benefit to FCOS
17:24:09 <jlebon> i do want those metrics, and i think removing pinger will make it even less likely that we ever do this
17:24:30 <jlebon> but if no one sees a glimmer of hope of that direction changing in the future, i agree we should kill it
17:24:43 <bgilbert> jlebon: honestly, the work it would take to build the backend and collection code >> the work to re-introduce a package
17:25:00 <travier> I think it would get more trackion as a distro independent project
17:25:38 <bgilbert> I agree it'd be nice to have this someday.  I think a major reason it's difficult is navigating the privacy issues
17:25:43 <jlebon> bgilbert: given the amount of discussion and debating each package addition gets, i'd say just > instead of >> :)
17:25:47 <bgilbert> hah
17:25:51 <bgilbert> fair
17:25:53 <travier> :D
17:26:25 <bgilbert> any other votes/discussion?
17:26:47 <travier> Once you add RGPD into the mix, the backend work alone is a major project
17:27:00 <travier> and then there is hosting it
17:27:52 <travier> anyone else for the vote?
17:28:01 <jlebon> anyway, in the end +1 on this but make it clear that we do want it and that anyone interested to step up should do so.   i'll admit i don't know much about how GDPR would affect implementation
17:28:14 <jlebon> s/we/some of us/ :)
17:28:22 <travier> :)
17:29:24 <bgilbert> #proposed We will remove fedora-coreos-pinger from FCOS and archive the repo upstream.  We still believe this functionality would be useful and are open to re-adding the package if the functionality is built out.
17:29:36 <travier> +1
17:29:44 <jlebon> +1 thanks!
17:29:45 <dustymabe> +1
17:30:04 <bgilbert> #agreed We will remove fedora-coreos-pinger from FCOS and archive the repo upstream.  We still believe this functionality would be useful and are open to re-adding the package if the functionality is built out.
17:30:14 <travier> #topic Open Floor
17:30:24 <travier> We're at time, thus let's keep it short
17:30:40 <travier> I'll likely close in 5 min
17:30:57 <jlebon> pinger is 2.15M, so net change with the nmstate addition is +5.8M :)
17:31:02 <travier> :D
17:31:04 <bgilbert> :-P
17:31:24 <dustymabe> since we didn't get to it today is someone open to creating a new ticket for the additional requests in https://github.com/coreos/fedora-coreos-tracker/issues/1088 ?
17:31:34 <dustymabe> so we can ship that boat that the river
17:31:58 <jlebon> a message came up on the CentOS CI users list re. testing rawhide, and I took the opportunity to plug FCOS :)  https://lists.centos.org/pipermail/ci-users/2022-June/004556.html
17:32:04 <bgilbert> I still don't think it needs a separate ticket, but I'm okay with that if desired
17:33:07 <dustymabe> jlebon: :) - yeah, we have to be careful to ID what's opinion and what isn't
17:33:09 <jlebon> i'm OK with including it as part of the PR too
17:33:23 <dustymabe> i think FCOS is awesome for many reasons
17:33:50 <dustymabe> but often people compare FCOS to other fedora offerings and I try to be real careful when doing that
17:34:02 <bgilbert> +1 dustymabe
17:34:32 <jlebon> dustymabe: it's funny because i wasn't even trying to go down that road
17:34:32 <dustymabe> i.e. I risk essentially "bad-mouthing" another edition/variant
17:34:52 <dustymabe> yeah, I know you weren't (I know who you are :))
17:35:07 <dustymabe> it's a fine line to walk for sure
17:35:12 <bgilbert> fwiw it reads a bit strong to me too
17:35:18 <travier> What's this CI we're talking about here?
17:35:53 <jlebon> ack, thanks for the feedback!
17:36:00 <jlebon> travier: CentOS CI is migrating to AWS
17:36:10 <jlebon> it's used by a lot of upstream projects
17:36:27 <dustymabe> https://lists.centos.org/pipermail/ci-users/2022-June/004556.html
17:36:33 <dustymabe> that's the email we're discussing
17:36:58 <dustymabe> now.. I *do* think we should start doing exactly what jlebon did, which is promote FCOS - it's ready!!!
17:36:59 <jlebon> i guess i should've s/*the*/a/
17:37:07 <dustymabe> jlebon: +1
17:37:13 <dustymabe> jlebon: thank you for promoting FCOS
17:37:46 <bgilbert> jlebon: +1 <3
17:38:27 <travier> +1 for more FCOS
17:38:41 <travier> (I've no idea how this other CI works however)
17:38:46 <travier> ETOOMANYCI
17:39:16 <dustymabe> FYI - I probably won't be around next week for this meeting.. the following week is the 1st meeting of the month (usually video meeting) anyone want to volunteer to organize a video meeting?
17:39:33 <dustymabe> oh also - This Month in FCOS ticket:
17:39:49 <dustymabe> https://github.com/coreos/fedora-coreos-tracker/issues/1234
17:40:46 <travier> we can skip video if we don't have anything specific
17:40:53 <travier> Alright, closing in 30 secs.
17:42:01 <travier> #endmeeting