16:30:48 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:48 <zodbot> Meeting started Wed Mar 20 16:30:48 2019 UTC.
16:30:48 <zodbot> This meeting is logged and archived in a public location.
16:30:48 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:48 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:48 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:52 <dustymabe> #topic roll call
16:30:54 <geoff-> geoff-: Geoff Levand <geoff@infradead.org>
16:30:56 <dustymabe> .hello2
16:30:57 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:31:08 <yzhang> .hello2
16:31:08 <rfairley> .hello2
16:31:08 <zodbot> yzhang: yzhang 'Yu Qi Zhang' <jzehrarnyg@gmail.com>
16:31:10 <jbrooks> .fas jasonbrooks
16:31:11 <ajeddeloh> .hello2
16:31:11 <zodbot> rfairley: rfairley 'None' <rfairley@redhat.com>
16:31:14 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com>
16:31:17 <zodbot> ajeddeloh: ajeddeloh 'Andrew Jeddeloh' <andrew.jeddeloh@redhat.com>
16:31:28 <slowrie> .hello2
16:31:29 <zodbot> slowrie: slowrie 'Stephen Lowrie' <slowrie@redhat.com>
16:31:53 <jlebon> .hello2
16:31:54 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:32:26 <dustymabe> #chair geoff- jbrooks yzhang ajeddeloh rfairley ajeddeloh jlebon slowrie
16:32:26 <zodbot> Current chairs: ajeddeloh dustymabe geoff- jbrooks jlebon rfairley slowrie yzhang
16:32:51 <bgilbert> .hello2
16:32:52 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:32:57 <dustymabe> #chair bgilbert
16:32:57 <zodbot> Current chairs: ajeddeloh bgilbert dustymabe geoff- jbrooks jlebon rfairley slowrie yzhang
16:33:05 <dustymabe> #topic Action items from last meeting
16:33:26 <dustymabe> Here were the action items from last meeting:
16:33:39 <dustymabe> * kaeso to open a ticket for node stream introspection, manual stream
16:33:41 <dustymabe> switching, and forced upgrades
16:34:16 <dustymabe> #info kaeso opened #163 to discuss node stream introspection, manual stream switching, and forced upgrades
16:34:21 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/163
16:34:49 <dustymabe> and that was all for action items - any comments before we move on?
16:35:29 <walters> .hello2
16:35:31 <zodbot> walters: walters 'Colin Walters' <walters@redhat.com>
16:35:35 <dustymabe> #chair walters
16:35:35 <zodbot> Current chairs: ajeddeloh bgilbert dustymabe geoff- jbrooks jlebon rfairley slowrie walters yzhang
16:35:37 <dustymabe> welcome walters
16:35:46 <dustymabe> #topic agenda for this meeting
16:35:59 <dustymabe> The agenda for this meeting is in https://public.etherpad-mozilla.org/p/20190320-FCOS-Meeting
16:36:06 <dustymabe> please add agenda items to the open floor section!
16:36:27 <mnguyen_> .hello mnguyen
16:36:28 <zodbot> mnguyen_: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:36:41 <dustymabe> We don't have any tickets with the meeting label today so we'll move straight to open floor items
16:36:44 <dustymabe> #chair mnguyen_
16:36:44 <zodbot> Current chairs: ajeddeloh bgilbert dustymabe geoff- jbrooks jlebon mnguyen_ rfairley slowrie walters yzhang
16:36:52 <dustymabe> #topic Building packages continuously in Fedora
16:36:59 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/84
16:37:14 <dustymabe> I have worked with releng to get a proper setup for this
16:37:36 <dustymabe> we have tested it out in stage koji and they will implement it in prod koji as soon as freeze for F30 beta is over
16:38:00 <dustymabe> the discussion happened over in https://pagure.io/releng/issue/8165
16:38:16 <dustymabe> a summary is: we will have a fXX_coreos_continuous tag
16:38:38 <dustymabe> we can build things in koji often (using tooling to automate builds) and then tag them into this tag (using bots)
16:38:55 <dustymabe> then a yum repo will get created with items from the tag
16:39:09 <dustymabe> then we can consume the yum repo as input to our builds
16:39:24 <jlebon> that's awesome!
16:39:28 <dustymabe> this has a nice advantage of working across all architectures that koji supports
16:39:48 * ajeddeloh is looking forward to it
16:40:01 <walters> very nice
16:40:08 <jlebon> dustymabe: so is the remaining work here mostly about automating this?
16:40:19 <dustymabe> an example repo that I created is here: https://kojipkgs.stg.fedoraproject.org/repos-dist/f29-coreos-continuous/
16:40:38 <dustymabe> jlebon: the remaining work is 1. get it into prod 2. automate
16:40:58 <dustymabe> some other teams within fedora have been working on a tool called `packit` that I think will help us build from upstream sources easier
16:41:11 <dustymabe> #link https://fedoraproject.org/wiki/Packit
16:41:20 <yzhang> curious: how long would those repos last
16:41:45 <dustymabe> yzhang: the current iteration of the repo should not get garbage collected
16:42:00 <dustymabe> so as long as the rpm that you want is tagged in the tag, then it should not get cleaned up
16:42:06 <yzhang> cool
16:42:39 <dustymabe> This is particularly exciting to me and I can't wait until we get the remaining pieces in place
16:42:44 <dustymabe> :)
16:43:00 <mnguyen_> its pretty awesome!
16:43:08 * dustymabe will add my summary from above to the ticket
16:43:52 <dustymabe> any other comments before we move on to the next item?  (also, please add more items to the agenda)
16:44:32 <dustymabe> #topic open floor topic: CI system (bare metal req)
16:44:42 * dustymabe yields the floor
16:45:17 <walters> this one is related to the previous
16:45:24 <walters> today in rpm-ostree's CI we rely on rdgo building ostree
16:45:41 <walters> it's common for us to add an API to ostree git master and then start using it in rpm-ostree
16:46:03 <walters> clearly rpm-ostree's upstream CI could consume this (rdgo-successor-that-needs-a-short-name)
16:46:32 <walters> but we really want bare metal resources hooked up to CI to use coreos-assembler
16:46:39 <walters> i want to rework rpm-ostree's CI to use cosa
16:46:50 <walters> and ostree
16:46:53 <walters> and for that matter ignition
16:47:26 <dustymabe> walters: the bare metal req is because nested virt isn't sufficient ?
16:47:28 <walters> we have a few options for this; use centos ci, add resources to fedora, or do something custom
16:47:37 <walters> it's pretty bad
16:47:38 <ajeddeloh> walters: can you elaborate on that a bit? I'm not sure I follow
16:48:17 <walters> new PR to ignition, build fcos using it and run through kola qemu
16:48:58 <walters> and of course for cosa itself
16:49:03 <ajeddeloh> oh I see, use the CI for testing ostree/cosa/ignition/etc
16:49:21 <jlebon> walters: are you suggesting dropping all instances of the `ostree commit -b vmcheck --tree=ref=$(pending_csum)` pattern with cosa invocations?
16:49:30 <walters> maybe
16:49:42 <jlebon> that'll multiple the testsuite time by a crazy amount
16:49:43 <walters> i think for a lot of things a "fast overlay" pattern like that is sufficient
16:49:58 <walters> well we can make a pristine build and clone it
16:50:13 <walters> i.e. avoid importing pkgs again etc.  but that detail is more getting into rpm-ostree territory
16:50:27 <jlebon> i agree overall though with easier integration with cosa in CI
16:50:49 <dustymabe> i guess the original ask still remains - we need bare metal hardware
16:50:59 <dustymabe> walters: how are you suggesting we manage the hardware if someone gave it to us ?
16:51:03 <walters> is there any longer term vision for where this fcos-in-centos-ci vs fedora infra goes?
16:51:57 <dustymabe> walters: I don't have any more info for you on that right now - fedora infra is OK with us using fcos-pipeline in centos CI for the time being. I think we'll import some of the artifacts back into fedora before we do releeases though
16:52:24 <dustymabe> but really we just need an openshift instance
16:52:32 <dustymabe> so if we move it to fedora's openshift that would work too
16:53:35 <jlebon> dustymabe: is that an option? that'd be sweet if it's more up to date than the cci instance
16:53:57 <walters> is it on metal?
16:55:33 <dustymabe> i don't know the answer to that question
16:56:00 <misc> (unlikely, but let me check)
16:56:34 <misc> nope
16:57:08 <walters> anyways I don't think we're going to solve this today, but the current situation of not having any real CI for cosa or ignition needs to be fixed, and the papr situation with ostree/rpm-ostree works-ish but makes it much harder to use cosa
16:57:16 <misc> see https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/playbooks/groups/os-cluster.yml
16:57:55 <ajeddeloh> on the topic of "real CI for Ignition" the bb tests will need to find a new home eventually too. They're still running on old CoreOS Inc infra
16:57:57 <walters> one option is to hook up OpenShift's Prow on these repos but have it delegate to scheduling tests on a separate cluster; shouldn't be too hard
16:58:02 <dustymabe> walters: yeah. if someone gave us hardware would you want to set up a cluster on it ?
16:58:15 <ajeddeloh> but they can't run in a container and need root, so finding a home would be a giant pain
16:59:14 <dustymabe> maybe CNV/kubvirt will solve all of our problems :)
16:59:40 <ajeddeloh> the other option would be to rewrite the test runner to use VMs but that's a giant project
17:00:14 <walters> CNV isn't required for the pattern of "single container puppeting a test VM" which is what cosa does today
17:00:53 <walters> but it will be useful to set up clusters of test machines without requiring an external cloud
17:01:18 <dustymabe> +1
17:01:25 <dustymabe> any more progress we can make on this topic today
17:02:01 <walters> probably not =)
17:03:06 <dustymabe> #topic open floor topic: support for vmware in fcos pipeline
17:03:13 <dustymabe> mnguyen_: want to give us an update on this?
17:03:44 <mnguyen_> sure
17:04:01 <mnguyen_> we have a PR in coreos-assembler to support producing OVA files for vmware
17:05:31 <dustymabe> nice work.. i think this was building on a previous PR from imcleod
17:05:34 <mnguyen_> the latest FCOS rpm is building from source that doesn't have support for a guestinfo.ignition.config.data so i'm going to patch the rpm with a backport
17:06:03 <mnguyen_> yes, this is building on top of imcleod's great work
17:06:33 * dustymabe also thanks mnguyen_ for showing him how to use vsphere yesterday
17:07:14 <dustymabe> any comments before we move to the next topic?
17:07:55 <dustymabe> #topic open floor
17:08:04 <dustymabe> anything from the multi-arch side of the house ?
17:08:09 <geoff-> For Etherpad users, please add your name.  It makes it easier to see who added what.
17:08:32 <dustymabe> geoff-: done :)
17:09:23 <dustymabe> FYI rafael opened an issue regarding using libvirt to better enable multi-arch support - please add input: https://github.com/coreos/coreos-assembler/issues/419
17:10:25 <dustymabe> igntion 3.x spec work is moving along nicely - nice work andrew and everyone else involved
17:10:34 * ajeddeloh is still wary of using libvirt for everything, since it hide what's actually going on
17:10:52 <dustymabe> and jlebon landed the libdnf rebase in rpm-ostree - woot!
17:11:32 <ajeddeloh> I'd rather see explicitly what needs to happen to spin up a VM on a specific arch than have things happen automatically
17:11:49 <ajeddeloh> plus for mantle we'll hit the same issues and I don't think we want libvirt as a dep for mantle
17:12:07 <bgilbert> I'd tend to agree with ajeddeloh
17:12:56 <walters> yeah
17:12:57 <dustymabe> i guess it depends on who you are, but in general I think abstractions are useful here, especially when there is a team/community that is managing that abstraction for us
17:13:03 <geoff-> I'd say we shouldn't try to support old stuff (that doesn't have GICv3 for example).
17:13:13 <walters> deduping the mantle code with cosa is a subtopic
17:14:12 <walters> probably needs to happen first really; doesn't help anything to change cosa to use libvirt if we still need to have all qemu/arch stuff in kola
17:15:01 <dustymabe> walters: so you're saying the real topic is: should mantle use libvirt ?
17:15:24 <bgilbert> kola does a _lot_ of work to drive qemu directly
17:15:34 <bgilbert> off the top of my head, I don't know how much of that could be dropped
17:15:38 <dustymabe> wouldn't it be nice if it didn't have to?
17:15:42 <dustymabe> :)
17:15:48 <walters> perhaps one incremental step would be sharing qemu shell script wrappers between mantle and cosa
17:15:55 <bgilbert> I think some of it may be impossible under libvirt, e.g. passing deleted disk images by file handle
17:16:03 <bgilbert> to avoid leaking disk images
17:16:21 <dustymabe> bgilbert: luckily libvirt has an escape hatch for that I think
17:16:34 <bgilbert> if we did want to switch (and take on libvirt as a dependency, which doesn't thrill me), I don't think it's in the critical path
17:16:37 <walters> or perhaps the more nuclear option...folding mantle and cosa into a single git repo
17:16:41 <dustymabe> you can arbitrarily pass qemu command line args via another field in the libvirt xml
17:16:54 <ajeddeloh> walters: uhhhhhhhhhhhhh
17:17:06 <walters> ajeddeloh: just throwing it out there =)
17:17:28 <dustymabe> :) - brainstorming crazy ideas can be useful at times
17:17:42 <slowrie> I mean mantle does have a history of carrying sdk's inside of it...
17:17:49 <jlebon> dustymabe: yeah, virt-install has --qemu-commandline at least
17:18:04 <jlebon> which i use for passing -fw_cfg
17:18:19 <walters> right, the openshift-installer's libvirt backend does that too
17:18:33 <walters> tactically I find the daemon thing most annoying
17:18:50 <dustymabe> maybe we need to have a separate meeting/issue where we discuss the merits of libvirt and qemu ?
17:18:51 <walters> but I also agree with andrew about the "what is my qemu doing" thing
17:19:17 <slowrie> being able to explicitly control what pieces are spun up is very relevant for a testing framework
17:19:22 <dustymabe> walters: but we can always find out what qemu is doing by inspecting the actual CLI that gets run?
17:19:49 <dustymabe> slowrie: i don't disagree, but can we still get that with libvirt?
17:20:31 <dustymabe> another thing to note: if we have feature requests there is a team that can help us get them in place
17:21:08 <slowrie> dustymabe: potentially, but you'd have to start requiring (for validation purposes) specific versions of libvirt to ensure that the underlying magic doesn't change underneath you
17:21:10 <jlebon> how hard is it actually to support multi-arch qemu though? if it's just a few snags to figure out once, might not be worth the engineering work required to switch over to libvirt
17:21:28 <bgilbert> jlebon: CL did it with ARM64.  wasn't a huge deal.
17:21:47 <dustymabe> jlebon: to me it's more about the effort over time
17:21:50 <jlebon> there's maintenance associated of course, but i suspect there'd be maintenance with libvirt too (e.g. the recent f29 virt-install upgrade that broke cosa)
17:22:19 <bgilbert> I'd suggest that this is, at least, a down-the-road thing.  the current setup works, switching would be nontrivial effort, and we're trying to ship a distro :-)
17:22:30 <jlebon> yeah, i
17:22:43 <jlebon> 'd agree with that :)
17:22:46 <walters> yeah
17:23:07 <dustymabe> yeah I think the only thing rafael would change is the one call to qemu that we have inside cmdlib.sh
17:23:07 <jlebon> let's feel that pain and see how bad it is
17:23:24 <dustymabe> to be a virt-install command
17:23:36 <dustymabe> they're trying to get this working for aarch64 and ppc64le
17:23:37 <walters> but andrew's work to drop anaconda surely uses qemu directly right?
17:23:48 <walters> and for that matter we're also using `env LIBGUESTFS_BACKEND=direct`
17:23:51 <dustymabe> it would be another case, I believe
17:24:16 <dustymabe> but I think his works uses `runvm` which is the function rafael would change
17:24:34 <walters> (on this topic of course libguestfs also tried really hard to not use libvirt I think for much of the same daemon reasons but eventually caved)
17:25:05 <ajeddeloh> walters: its using qemu directly ye
17:25:14 <ajeddeloh> dustymabe: yeah
17:26:09 * dustymabe has to run downstairs - bgilbert can you finish up the meeting ?
17:27:10 <bgilbert> sure
17:27:24 <bgilbert> anything else on this topic?
17:27:27 <bgilbert> or for open floor generally?
17:29:06 <bgilbert> #endmeeting