16:02:32 #startmeeting Redhat Community Platform Engineering (CPE) Weekly Standup Meeting 16:02:32 Meeting started Fri Feb 7 16:02:32 2020 UTC. 16:02:32 This meeting is logged and archived in a public location. 16:02:32 The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:02:32 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:02:32 The meeting name has been set to 'redhat_community_platform_engineering_(cpe)_weekly_standup_meeting' 16:02:32 #chair arrfab bstinson cverna mboddu relrod smoge 16:02:32 Current chairs: arrfab bstinson cverna mboddu relrod smoge smooge 16:02:32 #info meeting is 30 minutes MAX. At the end of 30, its stops 16:02:32 #info agenda is at tbd 16:02:33 smooge: You got your nick wrong in "#chair" 16:02:33 #topic Tickets needing review 16:02:34 #info ticket place tbd 16:03:05 smoooooooge. :) 16:03:07 mboddu, thanks. my dashing evil twin brother seems to want to chair this week 16:03:08 * mboddu is here, but be back in 2 min, need to grab some tea 16:03:29 Haha :) 16:03:30 ok so this meeting is a new event we are working on trying to get syncronization between teams 16:03:47 We will be working on clearing up agendas, tickets, tools and such over the next couple of meetings 16:04:05 nirik: in the vein of 'let's try to do things the same way' .. maybe we can start with AWS ? 16:04:16 and using https://pagure.io/fedora-infrastructure/issue/8625 as starter 16:04:19 but mostly I think it's a good chance to communicate so we don't each do something different ways or reinvent things 16:04:31 that ^ :-) 16:04:46 #meetingname CPE_SysAdmin_Weekly_Meeting 16:04:46 The meeting name has been set to 'cpe_sysadmin_weekly_meeting' 16:04:57 Arrfab: right. So, I thought we found a workaround for that, but I cannot seem to find it. 16:05:05 I mean, you have a workaround now... but it's not great 16:05:37 nirik: yes, I looked at the Fedora distribution for cloudfront, and on the S3 bucket but can't find anything 16:06:11 nothing that I could find in IAM policies either, so was wondering which kind of "black magic" was working for fedora mirror 16:06:21 as clearly you have also gcc-c++ as package name :) 16:06:42 the one I remember being discussed was some sort of ugly sed and such in the files and metadata 16:07:05 we are currently doing the same thing you are... it gives a 404 and goes to a non cloudfront mirror. 16:07:14 we found a workaround, but never implemented it as far as I know 16:07:15 smooge: you mean touching/modifying repodata before pushing to s3 ? that would be ugly 16:07:23 nirik: ah ! 16:07:35 it seems that https://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-page-redirect.html#advanced-conditional-redirects would probably help 16:07:36 it was something with a lambda function that would re-write things... 16:07:44 but I couldn't find it. 16:07:44 yep. Arrfab that was only one I remember 16:08:01 that seems like a kind of RewriteRule for aws :) 16:08:20 there may have been a lambda but I don't know if like any of the lambdas we have tried worked :) 16:08:24 so basically if we can find a way with a regex to say "replace + with space" 16:08:28 but is that really any better than just 404ing and going to another mirror? I guess it's a little less ugly 16:09:28 https://stackoverflow.com/questions/44779042/aws-how-to-fix-s3-event-replacing-space-with-sign-in-object-key-names-in-js 16:09:31 perhaps that one ? 16:09:48 nirik: my quick workaround was just to add mirror.centos.org as a second mirror, so if that fails, it goes to that one 16:10:00 *but* interestingly myself I couldn't reproduce the error 16:10:20 I could download "yumdownloader" gcc-c++ fine, and through cloudfront 16:10:23 yeah, it's the same with fedora... 16:10:42 dnf download gcc-c++ 16:10:42 Last metadata expiration check: 0:00:11 ago on Fri 07 Feb 2020 04:06:10 PM UTC. 16:10:42 [MIRROR] gcc-c++-9.2.1-1.fc31.x86_64.rpm: Status code: 404 for https://d2lzkl7pfhq30w.cloudfront.net/pub/fedora/linux/rel 16:10:42 eases/31/Everything/x86_64/os/Packages/g/gcc-c++-9.2.1-1.fc31.x86_64.rpm (IP: 52.84.225.124) 16:10:42 [MIRROR] gcc-c++-9.2.1-1.fc31.i686.rpm: Status code: 404 for https://d2lzkl7pfhq30w.cloudfront.net/pub/fedora/linux/relea 16:10:44 ses/31/Everything/x86_64/os/Packages/g/gcc-c++-9.2.1-1.fc31.i686.rpm (IP: 52.84.225.124) 16:10:46 (1/2): gcc-c++-9.2.1-1.fc31.i686.rpm 6.3 MB/s | 11 MB 00:01 16:10:49 (2/2): gcc-c++-9.2.1-1.fc31.x86_64.rpm 5.8 MB/s | 12 MB 00:02 16:11:14 its maaaaaagic 16:11:28 * nirik shrugs. I guess we could also ask davdd if there's anyworkarounds 16:11:49 nirik: already asked, and he wasn't even aware of such limitation ;-) 16:11:54 * smooge also says its magic that i386 is bigger than x86_64 by 0.5 mb 16:12:19 * smooge reads the wrong size 16:12:25 * smooge gets his glasses 16:12:41 ok sorry so we are basically doing the same thing here 16:12:47 yes 16:12:58 do we have much more else on it? or should we go to somehting else? 16:13:01 it's not nice in that users get that 404... but meh 16:13:38 nirik: amusingly as said, without any modification, I can download internally through cloudfront that package 16:14:04 huh... 16:14:40 I was testing from a instance in aws too... 16:14:53 gcc-c++-4.8.5-39.el7.x86_64.rpm | 7.2 MB 00:00:00 16:14:54 I don't understand why it would work for some and not others 16:15:22 so spin a c7 instance, you get cloudfront as mirror, and it works .. so wondering if it doesn't depend on the edge server one can hit 16:15:45 and if , depending on their own infra, if they support retrying automatically by converting "on the fly" if they get 404 16:16:47 Arrfab, nirik what command did you use? On el7 it would be yum based and on f31 it would be dnf based 16:17:03 those act differently on status codes 16:17:21 yeah, I was using dnf on proxy30... 16:17:59 anyway.. we are nearing the end of this meeting. I would like to get a couple of side things dealt with 16:18:15 1. for the agenda is it ok for it to be housed on board.net or somewhere else? 16:18:19 perhaps yum is silently retrying? can you see with URLGRABBER_DEBUG=1 16:18:44 2. tickets to be reviewed for fedora are ok but not sure about centos+fedora related tickets 16:18:56 s/are ok/are easy for me to link to/ 16:18:59 agenda is fine with me anywhere. 16:19:30 me too 16:19:42 I don't think we want to review tickets... more perhaps 'high level summary of what we were all working on that might be of interest to the entire group' ? not sure how to tell that 16:20:01 I think aws stuff is one place where we overlap and should keep in close contact. 16:20:09 nirik: when can we review also policies in AWS ? as I saw that maybe some permissions are probably not relevant anymore 16:20:10 we still should setup a repo and docs and such... 16:20:16 +1 16:20:30 Agenda is fine for me anywhere it is 16:20:39 I have gitforge paralysis tho... so hard to choose one 16:21:04 well .. 16:21:04 * mboddu dont want to get into that 16:21:28 I'd probibly just make a pagure project with no other needs... 16:21:34 +1 16:21:44 ok cool 16:22:30 nirik, I am good with that also. 16:22:52 Arrfab: you care where? or want to make it? or want me to? 16:22:57 nirik, Arrfab I will update the agenda doc and when a repo is done will put it there 16:23:26 make project, pull in current policy stuff, go over it and update it next meeting? 16:23:29 for "internal" repo for centos we use our own gitea instance .. but I don't mind elsewhere 16:23:38 as long as it's not public 16:23:41 and encrypted 16:23:53 "not public"? 16:23:56 don't know what Fedora uses, but we use git-crypt 16:24:11 mboddu: yes, we don't want to have our security policies being public? :) 16:24:13 I wasn't thinking of putting any secrets in it... 16:24:18 incuding with ARNs, etc 16:24:37 ^^ yes, thats my assumption as well 16:24:39 hum, not sure how much it matters, but I guess thats a point. 16:25:29 the guideline for centos infra is (doesn't mean that we have to enforce it but ..) : ansible roles are public 16:25:58 but inventory isn't (as it includes users/credentials) , so in a private git repo, that is gpg encrypted automatically 16:26:03 well, I like default to open, but I am not sure of the implications of public policy. 16:26:30 our inventory is public also... we have a seperate private repo with passwords/etc. 16:28:00 Pagure doesn't support private repos (well it does, but its not enabled on pagure.io) 16:28:01 to be discussed 16:28:21 in semi related news, I created a restic role for centos 16:28:50 cool! 16:28:53 so "archiving" local backup through encrypted remote backup to s3, with rotation and also deduplication (if you don't know the tool) 16:29:26 we should look at it... we currently use rdiff-backup, but it's got a lot of limitations... 16:29:26 ok going to end the meeting in a short bit 16:30:04 and when creating that, I thought about simple policy for S3 bucket : each service would require its own account, with policy with minimum, identified through source ip, and which bucket (only one) 16:30:09 #endmeeting