16:00:29 <nirik> #startmeeting Redhat Community Platform Engineering (CPE) SysAdmin Weekly Meeting
16:00:29 <zodbot> Meeting started Fri Feb 14 16:00:29 2020 UTC.
16:00:29 <zodbot> This meeting is logged and archived in a public location.
16:00:29 <zodbot> The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:29 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:29 <zodbot> The meeting name has been set to 'redhat_community_platform_engineering_(cpe)_sysadmin_weekly_meeting'
16:00:29 <nirik> #meetingname CPE_SysAdmin_Weekly_Meeting
16:00:29 <nirik> #chair arrfab bstinson cverna mboddu relrod smooge
16:00:29 <nirik> #info meeting is 30 minutes MAX. At the end of 30, its stops
16:00:29 <zodbot> The meeting name has been set to 'cpe_sysadmin_weekly_meeting'
16:00:29 <zodbot> Current chairs: arrfab bstinson cverna mboddu nirik relrod smooge
16:00:39 * mboddu is here
16:00:46 <nirik> (likely a short meeting today :)
16:01:28 <mboddu> Friday Friday Friday....
16:01:33 <nirik> I was going to ask: what should we use for centos8 image in aws... I know it's complicated, but we should try and share the same image if we can
16:01:58 <nirik> and the fedora image story is also horrible for the record.
16:02:17 <nirik> If you search for 'Fedora' you get a bunch of random rawhide and branched images
16:02:24 <mboddu> What do you mean by "try and share the same image"
16:02:37 <nirik> if you search for 'Fedora 31' you get a bunch of random rawhide and branched images pre-release.
16:02:48 <mboddu> Ohhhh
16:02:55 <mboddu> fedimg?
16:02:56 <nirik> if you search for 'Fedora 31 1.9' you get the actual final release image
16:03:07 <nirik> (because it was rc9 I guess)
16:03:22 <nirik> mboddu: between centos and fedora infra in our aws account, for our instances. ;)
16:03:44 <mboddu> .releng 9240
16:03:45 <zodbot> mboddu: Issue #9240: Update published Fedora 31 cloud images - releng - Pagure.io - https://pagure.io/releng/issue/9240
16:04:00 <mboddu> Oh okay
16:04:17 <Arrfab> nirik: sorry, yes I'm here but also discussing with MikeM about koji dist-repo
16:04:49 <nirik> mboddu: sure, but that doesn't help. we should have a way to 'hide' all those developer ones unless someone wants a rawhide or prerelease...
16:05:01 <mboddu> Right
16:05:02 <nirik> Arrfab: no worries. we can just end today too if you are busy.
16:07:00 <Arrfab> nirik: working on a signing process, as we don't have news about sigul, so was asked to not use it and work on something new
16:07:54 <Arrfab> nirik: for your question about AWS : it's all blocked because process isn't in our hands : only one guy can speak with AWS for marketplace and process is stuck, without any progress in one year
16:08:12 <Arrfab> our centos 7 AMI is still a 7.6 from january 2019 :-(
16:08:18 <nirik> sure, but for our personal/account use, should we upload a image we can use?
16:08:44 <Arrfab> nirik: that seems to be the only workaround, as it's currently broken for all centos users
16:09:03 <Arrfab> nirik: when I tried, my account was lacking some rights to import image
16:09:26 <nirik> bstinson pointed me to a qcow2, but it needs you to download that, convert to raw and re-upload... so I held off until we could talk. ;)
16:09:35 <Arrfab> you can convert qcow2 to raw, upload to public s3 and then call import , so it would create ebs snapshot and you can deploy from it
16:09:54 <nirik> ok. I can see about doing that... not sure when I will get to it. ;)
16:09:58 <Arrfab> yeah, that's the process, but when I tried with my account, I was blocked :)
16:10:20 <Arrfab> also, it seems that one has to do it in all regions too
16:10:30 <Arrfab> but if you can have it working, that would be great
16:10:38 <nirik> yeah, anoyiing that. ;(
16:10:59 <nirik> I'll let you know if I get it and you do likewise if you get it before me. ;)
16:10:59 <Arrfab> nirik: keep asking Evolution ;-)
16:12:01 <nirik> yeah, I noticed if you search for 'centos 8' or for 'fedora 31' for that matter, the top hits are marketplace people providing those images "with support!" for xyz amount. which seems... poor
16:12:26 <Arrfab> yeah, it's really bad and also hurts the project : there are even people providing centos stream
16:12:42 <Arrfab> normally Evolution reported that to RH legal as we don't know what that is
16:12:51 <nirik> yeah. ;(
16:13:08 <nirik> ok. I didnt have too much more today...
16:13:19 <Arrfab> and because we have no official image (due to process stuck on one guy and nothing we can do) that would give RH bad press if those images would have been modified
16:13:42 <nirik> yep. oh well, I hope it can all be sorted someday....
16:13:47 <Arrfab> people are stupid enough to just consume what they can find, without verifying first :)
16:13:59 * nirik nods. ;(
16:14:44 <Arrfab> nirik: talking about koji : are you using dist-repo to generate repositories with signed pkgs that would go out, or just use pungi for this ?
16:15:02 <nirik> we are using dist repos for some things yeah.
16:15:20 <nirik> we use them for our infrastructure packages (if we have to have something different from the base os)
16:15:27 <nirik> and coreos is using them
16:15:36 <Arrfab> nirik: here is the question : as dist-repo would need to know the tag and with key to use, how can you be sure that you have indeed all the signed rpms in place first in koji ?
16:16:12 <nirik> so, we have robosign... it listens for tag events on the message bus then calls sigul to sign the package and move it to another tag...
16:16:19 <nirik> so we know if it's in that tag it's been signed.
16:16:44 <nirik> we also have a koji plugin called tag2repo that makes a dist-repo anytime a tag changes...
16:16:59 <nirik> https://pagure.io/releng/tag2distrepo
16:16:59 <Arrfab> in our case, for cbs.centos.org, same pkg can be tagged multiple times, as different SIG
16:17:05 <Arrfab> so signed with different keys too
16:17:24 <Arrfab> oh, tag2repo seems nice
16:17:47 <Arrfab> but the problem is that , per board decision, only -release pkgs are signed,
16:17:48 <nirik> yeah, our flow has been around tag1 -> sign/operate on it -> tag2 -> etc.
16:17:56 <Arrfab> no -candidate nor -testing
16:18:25 <nirik> that makes things much more difficult.
16:18:27 <Arrfab> and SIGs users can't access keys either, so the process had to be initiated from the signing box (through multiple layers)
16:18:35 <Arrfab> yeah
16:19:13 <Arrfab> was evaluating dist-repo, as the current workflow has been : mash generates repositories (all unsigned)
16:19:43 <Arrfab> signing process verifies if new pkgs landed in -testing or -release (-candidate pkgs stay in koji)
16:19:54 <Arrfab> if tagged for -testing : they go out as-is
16:20:10 <Arrfab> if -release : pkgs are signed + repodatat signed too => mirror network
16:20:14 <Evolution> nirik: yeah. that's something matthew and I bring up periodically. (others providing 'official' centos/fedora images). I don't know why legal doesn't seem to care.
16:20:20 <Evolution> but we keep raising it.
16:20:44 <nirik> dist-repo is pretty nice. I suppose you could do lots of dist-repos (some signed, some not) and combine them for what you need?
16:20:52 <nirik> Evolution: yeah, it's pretty weird.
16:21:06 <Arrfab> nirik: well, before dist-repo, pkgs would need to be signed
16:21:27 <Arrfab> so that would mean a koji client somewhere, with a message bus, and reacting to tag events
16:21:30 <nirik> yeah, it just can use them if they are signed, it has no signing ability itself
16:21:46 <Arrfab> and based on tags, signing + koji import-sig and then trigger dist-repo
16:22:03 <nirik> actually I think yoou can ask it to use signed packages by key X and also ignore errors of unsigned... so you might be able to mix them that way if you needed
16:22:35 <Arrfab> nirik: yes : --allow-missing-signatures
16:22:45 <nirik> yeah, or skip if you don't want them in there.
16:23:05 <Arrfab> but problem is that it would generate a repo with unsigned pkgs
16:23:16 <Arrfab> maybe this would work :
16:23:34 <Arrfab> - asking dist-repo <tag>-release
16:23:38 <Arrfab> - waiting
16:24:02 <Arrfab> - signing box (through nfs) verify each pkg that landed and not yet signed
16:24:11 <Arrfab> - if there are some, sign them, koji import-sig
16:24:20 <Arrfab> go to step1
16:24:40 <nirik> yeah, could work
16:24:43 <Arrfab> hopefully that would produce repo with all signed pkgs, but that would be a lot of work on koji to chain that
16:25:21 <nirik> I would think it wouldn't be too many loops? it should get them all on the first run usually no?
16:25:30 <nirik> I guess if it's being added to...
16:25:33 <bstinson> we should carefully consider what we need those repos for in the first place
16:25:47 * nirik nods
16:26:16 <Arrfab> bstinson: do you know if there is a way through koji to verify which tagged pkgs in a tag aren't already signed ?
16:26:33 <Arrfab> that would speed up the whole thing and then only calling dist-repo after
16:26:46 <Arrfab> when I asked mikem he had no opinion/idea
16:26:54 <bstinson> we have a script for that
16:27:01 <Arrfab> oh !
16:27:15 <bstinson> but pulling unsigned packages and sticking them on buildlogs.centos.org doesn't need to involve this at all
16:27:31 <Arrfab> bstinson: true
16:27:38 <Arrfab> so that would be for -release tags
16:27:55 <bstinson> long-term i'd like pungi to do layered variants for this stuff, and we can have it do unsigned for testing, signed for release and not require dist-repos at all
16:28:05 * nirik would only think of the kludgy way to know that... koji write-signed-rpm and if it errors... no signed. ;)
16:28:27 <Arrfab> bstinson: that would probably be the goal yes
16:28:27 <nirik> or I suppose just check the url for a signed copy.
16:28:46 <Arrfab> so wondering if I should just reimplement what we have so far with mash, and revisit later for pungi ?
16:30:13 <nirik> good ol mash. At first a comedy and then a drama. :)
16:30:47 * nirik sees we are at 30min for this meeting anything else to discuss, or shall we continue out of meeting...
16:31:15 <Arrfab> nirik: at least "koji dist-repo <tag> <keyID>" exit 1 if there are unsigned pkgs
16:31:26 <nirik> true, that also
16:31:48 <nirik> #endmeeting