16:00:29 #startmeeting Redhat Community Platform Engineering (CPE) SysAdmin Weekly Meeting 16:00:29 Meeting started Fri Feb 14 16:00:29 2020 UTC. 16:00:29 This meeting is logged and archived in a public location. 16:00:29 The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:29 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:00:29 The meeting name has been set to 'redhat_community_platform_engineering_(cpe)_sysadmin_weekly_meeting' 16:00:29 #meetingname CPE_SysAdmin_Weekly_Meeting 16:00:29 #chair arrfab bstinson cverna mboddu relrod smooge 16:00:29 #info meeting is 30 minutes MAX. At the end of 30, its stops 16:00:29 The meeting name has been set to 'cpe_sysadmin_weekly_meeting' 16:00:29 Current chairs: arrfab bstinson cverna mboddu nirik relrod smooge 16:00:39 * mboddu is here 16:00:46 (likely a short meeting today :) 16:01:28 Friday Friday Friday.... 16:01:33 I was going to ask: what should we use for centos8 image in aws... I know it's complicated, but we should try and share the same image if we can 16:01:58 and the fedora image story is also horrible for the record. 16:02:17 If you search for 'Fedora' you get a bunch of random rawhide and branched images 16:02:24 What do you mean by "try and share the same image" 16:02:37 if you search for 'Fedora 31' you get a bunch of random rawhide and branched images pre-release. 16:02:48 Ohhhh 16:02:55 fedimg? 16:02:56 if you search for 'Fedora 31 1.9' you get the actual final release image 16:03:07 (because it was rc9 I guess) 16:03:22 mboddu: between centos and fedora infra in our aws account, for our instances. ;) 16:03:44 .releng 9240 16:03:45 mboddu: Issue #9240: Update published Fedora 31 cloud images - releng - Pagure.io - https://pagure.io/releng/issue/9240 16:04:00 Oh okay 16:04:17 nirik: sorry, yes I'm here but also discussing with MikeM about koji dist-repo 16:04:49 mboddu: sure, but that doesn't help. we should have a way to 'hide' all those developer ones unless someone wants a rawhide or prerelease... 16:05:01 Right 16:05:02 Arrfab: no worries. we can just end today too if you are busy. 16:07:00 nirik: working on a signing process, as we don't have news about sigul, so was asked to not use it and work on something new 16:07:54 nirik: for your question about AWS : it's all blocked because process isn't in our hands : only one guy can speak with AWS for marketplace and process is stuck, without any progress in one year 16:08:12 our centos 7 AMI is still a 7.6 from january 2019 :-( 16:08:18 sure, but for our personal/account use, should we upload a image we can use? 16:08:44 nirik: that seems to be the only workaround, as it's currently broken for all centos users 16:09:03 nirik: when I tried, my account was lacking some rights to import image 16:09:26 bstinson pointed me to a qcow2, but it needs you to download that, convert to raw and re-upload... so I held off until we could talk. ;) 16:09:35 you can convert qcow2 to raw, upload to public s3 and then call import , so it would create ebs snapshot and you can deploy from it 16:09:54 ok. I can see about doing that... not sure when I will get to it. ;) 16:09:58 yeah, that's the process, but when I tried with my account, I was blocked :) 16:10:20 also, it seems that one has to do it in all regions too 16:10:30 but if you can have it working, that would be great 16:10:38 yeah, anoyiing that. ;( 16:10:59 I'll let you know if I get it and you do likewise if you get it before me. ;) 16:10:59 nirik: keep asking Evolution ;-) 16:12:01 yeah, I noticed if you search for 'centos 8' or for 'fedora 31' for that matter, the top hits are marketplace people providing those images "with support!" for xyz amount. which seems... poor 16:12:26 yeah, it's really bad and also hurts the project : there are even people providing centos stream 16:12:42 normally Evolution reported that to RH legal as we don't know what that is 16:12:51 yeah. ;( 16:13:08 ok. I didnt have too much more today... 16:13:19 and because we have no official image (due to process stuck on one guy and nothing we can do) that would give RH bad press if those images would have been modified 16:13:42 yep. oh well, I hope it can all be sorted someday.... 16:13:47 people are stupid enough to just consume what they can find, without verifying first :) 16:13:59 * nirik nods. ;( 16:14:44 nirik: talking about koji : are you using dist-repo to generate repositories with signed pkgs that would go out, or just use pungi for this ? 16:15:02 we are using dist repos for some things yeah. 16:15:20 we use them for our infrastructure packages (if we have to have something different from the base os) 16:15:27 and coreos is using them 16:15:36 nirik: here is the question : as dist-repo would need to know the tag and with key to use, how can you be sure that you have indeed all the signed rpms in place first in koji ? 16:16:12 so, we have robosign... it listens for tag events on the message bus then calls sigul to sign the package and move it to another tag... 16:16:19 so we know if it's in that tag it's been signed. 16:16:44 we also have a koji plugin called tag2repo that makes a dist-repo anytime a tag changes... 16:16:59 https://pagure.io/releng/tag2distrepo 16:16:59 in our case, for cbs.centos.org, same pkg can be tagged multiple times, as different SIG 16:17:05 so signed with different keys too 16:17:24 oh, tag2repo seems nice 16:17:47 but the problem is that , per board decision, only -release pkgs are signed, 16:17:48 yeah, our flow has been around tag1 -> sign/operate on it -> tag2 -> etc. 16:17:56 no -candidate nor -testing 16:18:25 that makes things much more difficult. 16:18:27 and SIGs users can't access keys either, so the process had to be initiated from the signing box (through multiple layers) 16:18:35 yeah 16:19:13 was evaluating dist-repo, as the current workflow has been : mash generates repositories (all unsigned) 16:19:43 signing process verifies if new pkgs landed in -testing or -release (-candidate pkgs stay in koji) 16:19:54 if tagged for -testing : they go out as-is 16:20:10 if -release : pkgs are signed + repodatat signed too => mirror network 16:20:14 nirik: yeah. that's something matthew and I bring up periodically. (others providing 'official' centos/fedora images). I don't know why legal doesn't seem to care. 16:20:20 but we keep raising it. 16:20:44 dist-repo is pretty nice. I suppose you could do lots of dist-repos (some signed, some not) and combine them for what you need? 16:20:52 Evolution: yeah, it's pretty weird. 16:21:06 nirik: well, before dist-repo, pkgs would need to be signed 16:21:27 so that would mean a koji client somewhere, with a message bus, and reacting to tag events 16:21:30 yeah, it just can use them if they are signed, it has no signing ability itself 16:21:46 and based on tags, signing + koji import-sig and then trigger dist-repo 16:22:03 actually I think yoou can ask it to use signed packages by key X and also ignore errors of unsigned... so you might be able to mix them that way if you needed 16:22:35 nirik: yes : --allow-missing-signatures 16:22:45 yeah, or skip if you don't want them in there. 16:23:05 but problem is that it would generate a repo with unsigned pkgs 16:23:16 maybe this would work : 16:23:34 - asking dist-repo -release 16:23:38 - waiting 16:24:02 - signing box (through nfs) verify each pkg that landed and not yet signed 16:24:11 - if there are some, sign them, koji import-sig 16:24:20 go to step1 16:24:40 yeah, could work 16:24:43 hopefully that would produce repo with all signed pkgs, but that would be a lot of work on koji to chain that 16:25:21 I would think it wouldn't be too many loops? it should get them all on the first run usually no? 16:25:30 I guess if it's being added to... 16:25:33 we should carefully consider what we need those repos for in the first place 16:25:47 * nirik nods 16:26:16 bstinson: do you know if there is a way through koji to verify which tagged pkgs in a tag aren't already signed ? 16:26:33 that would speed up the whole thing and then only calling dist-repo after 16:26:46 when I asked mikem he had no opinion/idea 16:26:54 we have a script for that 16:27:01 oh ! 16:27:15 but pulling unsigned packages and sticking them on buildlogs.centos.org doesn't need to involve this at all 16:27:31 bstinson: true 16:27:38 so that would be for -release tags 16:27:55 long-term i'd like pungi to do layered variants for this stuff, and we can have it do unsigned for testing, signed for release and not require dist-repos at all 16:28:05 * nirik would only think of the kludgy way to know that... koji write-signed-rpm and if it errors... no signed. ;) 16:28:27 bstinson: that would probably be the goal yes 16:28:27 or I suppose just check the url for a signed copy. 16:28:46 so wondering if I should just reimplement what we have so far with mash, and revisit later for pungi ? 16:30:13 good ol mash. At first a comedy and then a drama. :) 16:30:47 * nirik sees we are at 30min for this meeting anything else to discuss, or shall we continue out of meeting... 16:31:15 nirik: at least "koji dist-repo " exit 1 if there are unsigned pkgs 16:31:26 true, that also 16:31:48 #endmeeting