16:02:32 <smooge> #startmeeting Redhat Community Platform Engineering (CPE) Weekly Standup Meeting
16:02:32 <zodbot> Meeting started Fri Feb  7 16:02:32 2020 UTC.
16:02:32 <zodbot> This meeting is logged and archived in a public location.
16:02:32 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:02:32 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:02:32 <zodbot> The meeting name has been set to 'redhat_community_platform_engineering_(cpe)_weekly_standup_meeting'
16:02:32 <smooge> #chair arrfab bstinson cverna mboddu relrod smoge
16:02:32 <zodbot> Current chairs: arrfab bstinson cverna mboddu relrod smoge smooge
16:02:32 <smooge> #info meeting is 30 minutes MAX. At the end of 30, its stops
16:02:32 <smooge> #info agenda is at tbd
16:02:33 <mboddu> smooge: You got your nick wrong in "#chair"
16:02:33 <smooge> #topic Tickets needing review
16:02:34 <smooge> #info ticket place tbd
16:03:05 <nirik> smoooooooge. :)
16:03:07 <smooge> mboddu, thanks. my dashing evil twin brother seems to want to chair this week
16:03:08 * mboddu is here, but be back in 2 min, need to grab some tea
16:03:29 <mboddu> Haha :)
16:03:30 <smooge> ok so this meeting is a new event we are working on trying to get syncronization between teams
16:03:47 <smooge> We will be working on clearing up agendas, tickets, tools and such over the next couple of meetings
16:04:05 <Arrfab> nirik: in the vein of 'let's try to do things the same way' .. maybe we can start with AWS ?
16:04:16 <Arrfab> and using https://pagure.io/fedora-infrastructure/issue/8625 as starter
16:04:19 <nirik> but mostly I think it's a good chance to communicate so we don't each do something different ways or reinvent things
16:04:31 <Arrfab> that ^ :-)
16:04:46 <smooge> #meetingname CPE_SysAdmin_Weekly_Meeting
16:04:46 <zodbot> The meeting name has been set to 'cpe_sysadmin_weekly_meeting'
16:04:57 <nirik> Arrfab: right. So, I thought we found a workaround for that, but I cannot seem to find it.
16:05:05 <nirik> I mean, you have a workaround now... but it's not great
16:05:37 <Arrfab> nirik: yes, I looked at the Fedora distribution for cloudfront, and on the S3 bucket but can't find anything
16:06:11 <Arrfab> nothing that I could find in IAM policies either, so was wondering which kind of "black magic" was working for fedora mirror
16:06:21 <Arrfab> as clearly you have also gcc-c++ as package name :)
16:06:42 <smooge> the one I remember being discussed was some sort of ugly sed and such in the files and metadata
16:07:05 <nirik> we are currently doing the same thing you are... it gives a 404 and goes to a non cloudfront mirror.
16:07:14 <nirik> we found a workaround, but never implemented it as far as I know
16:07:15 <Arrfab> smooge: you mean touching/modifying repodata before pushing to s3 ? that would be ugly
16:07:23 <Arrfab> nirik: ah !
16:07:35 <Arrfab> it seems that https://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-page-redirect.html#advanced-conditional-redirects would probably help
16:07:36 <nirik> it was something with a lambda function that would re-write things...
16:07:44 <nirik> but I couldn't find it.
16:07:44 <smooge> yep. Arrfab that was only one I remember
16:08:01 <Arrfab> that seems like a kind of RewriteRule for aws :)
16:08:20 <smooge> there may have been a lambda but I don't know if like any of the lambdas we have tried worked :)
16:08:24 <Arrfab> so basically if we can find a way with a regex to say "replace + with space"
16:08:28 <nirik> but is that really any better than just 404ing and going to another mirror? I guess it's a little less ugly
16:09:28 <nirik> https://stackoverflow.com/questions/44779042/aws-how-to-fix-s3-event-replacing-space-with-sign-in-object-key-names-in-js
16:09:31 <nirik> perhaps that one ?
16:09:48 <Arrfab> nirik: my quick workaround was just to add mirror.centos.org as a second mirror, so if that fails, it goes to that one
16:10:00 <Arrfab> *but* interestingly myself I couldn't reproduce the error
16:10:20 <Arrfab> I could download "yumdownloader" gcc-c++ fine, and through cloudfront
16:10:23 <nirik> yeah, it's the same with fedora...
16:10:42 <nirik> dnf download gcc-c++
16:10:42 <nirik> Last metadata expiration check: 0:00:11 ago on Fri 07 Feb 2020 04:06:10 PM UTC.
16:10:42 <nirik> [MIRROR] gcc-c++-9.2.1-1.fc31.x86_64.rpm: Status code: 404 for https://d2lzkl7pfhq30w.cloudfront.net/pub/fedora/linux/rel
16:10:42 <nirik> eases/31/Everything/x86_64/os/Packages/g/gcc-c++-9.2.1-1.fc31.x86_64.rpm (IP: 52.84.225.124)
16:10:42 <nirik> [MIRROR] gcc-c++-9.2.1-1.fc31.i686.rpm: Status code: 404 for https://d2lzkl7pfhq30w.cloudfront.net/pub/fedora/linux/relea
16:10:44 <nirik> ses/31/Everything/x86_64/os/Packages/g/gcc-c++-9.2.1-1.fc31.i686.rpm (IP: 52.84.225.124)
16:10:46 <nirik> (1/2): gcc-c++-9.2.1-1.fc31.i686.rpm                                                     6.3 MB/s |  11 MB     00:01
16:10:49 <nirik> (2/2): gcc-c++-9.2.1-1.fc31.x86_64.rpm                                                   5.8 MB/s |  12 MB     00:02
16:11:14 <smooge> its maaaaaagic
16:11:28 * nirik shrugs. I guess we could also ask davdd if there's anyworkarounds
16:11:49 <Arrfab> nirik: already asked, and he wasn't even aware of such limitation ;-)
16:11:54 * smooge also says its magic that i386 is bigger than x86_64 by 0.5 mb
16:12:19 * smooge reads the wrong size
16:12:25 * smooge gets his glasses
16:12:41 <smooge> ok sorry so we are basically doing the same thing here
16:12:47 <nirik> yes
16:12:58 <smooge> do we have much more else on it? or should we go to somehting else?
16:13:01 <nirik> it's not nice in that users get that 404... but meh
16:13:38 <Arrfab> nirik: amusingly as said, without any modification, I can download internally through cloudfront that package
16:14:04 <nirik> huh...
16:14:40 <nirik> I was testing from a instance in aws too...
16:14:53 <Arrfab> gcc-c++-4.8.5-39.el7.x86_64.rpm                                                                                          | 7.2 MB  00:00:00
16:14:54 <nirik> I don't understand why it would work for some and not others
16:15:22 <Arrfab> so spin a c7 instance, you get cloudfront as mirror, and it works .. so wondering if it doesn't depend on the edge server one can hit
16:15:45 <Arrfab> and if , depending on their own infra, if they support retrying automatically by converting "on the fly" if they get 404
16:16:47 <smooge> Arrfab, nirik what command did you use? On el7 it would be yum based and on f31 it would be dnf based
16:17:03 <smooge> those act differently on status codes
16:17:21 <nirik> yeah, I was using dnf on proxy30...
16:17:59 <smooge> anyway.. we are nearing the end of this meeting. I would like to get a couple of side things dealt with
16:18:15 <smooge> 1. for the agenda is it ok for it to be housed on board.net or somewhere else?
16:18:19 <nirik> perhaps yum is silently retrying? can you see with URLGRABBER_DEBUG=1
16:18:44 <smooge> 2. tickets to be reviewed for fedora are ok but not sure about centos+fedora related tickets
16:18:56 <smooge> s/are ok/are easy for me to link to/
16:18:59 <nirik> agenda is fine with me anywhere.
16:19:30 <Arrfab> me too
16:19:42 <nirik> I don't think we want to review tickets... more perhaps 'high level summary of what we were all working on that might be of interest to the entire group' ? not sure how to tell that
16:20:01 <nirik> I think aws stuff is one place where we overlap and should keep in close contact.
16:20:09 <Arrfab> nirik: when can we review also policies in AWS ? as I saw that maybe some permissions are probably not relevant anymore
16:20:10 <nirik> we still should setup a repo and docs and such...
16:20:16 <Arrfab> +1
16:20:30 <mboddu> Agenda is fine for me anywhere it is
16:20:39 <nirik> I have gitforge paralysis tho... so hard to choose one
16:21:04 <Arrfab> well ..
16:21:04 * mboddu dont want to get into that
16:21:28 <nirik> I'd probibly just make a pagure project with no other needs...
16:21:34 <mboddu> +1
16:21:44 <smooge> ok cool
16:22:30 <smooge> nirik, I am good with that also.
16:22:52 <nirik> Arrfab: you care where? or want to make it? or want me to?
16:22:57 <smooge> nirik, Arrfab I will update the agenda doc and when a repo is done will put it there
16:23:26 <nirik> make project, pull in current policy stuff, go over it and update it next meeting?
16:23:29 <Arrfab> for "internal" repo for centos we use our own gitea instance .. but I don't mind elsewhere
16:23:38 <Arrfab> as long as it's not public
16:23:41 <Arrfab> and encrypted
16:23:53 <mboddu> "not public"?
16:23:56 <Arrfab> don't know what Fedora uses, but we use git-crypt
16:24:11 <Arrfab> mboddu: yes, we don't want to have our security policies being public? :)
16:24:13 <nirik> I wasn't thinking of putting any secrets in it...
16:24:18 <Arrfab> incuding with ARNs, etc
16:24:37 <mboddu> ^^ yes, thats my assumption as well
16:24:39 <nirik> hum, not sure how much it matters, but I guess thats a point.
16:25:29 <Arrfab> the guideline for centos infra is (doesn't mean that we have to enforce it but ..) : ansible roles are public
16:25:58 <Arrfab> but inventory isn't (as it includes users/credentials) , so in a private git repo, that is gpg encrypted automatically
16:26:03 <nirik> well, I like default to open, but I am not sure of the implications of public policy.
16:26:30 <nirik> our inventory is public also... we have a seperate private repo with passwords/etc.
16:28:00 <mboddu> Pagure doesn't support private repos (well it does, but its not enabled on pagure.io)
16:28:01 <Arrfab> to be discussed
16:28:21 <Arrfab> in semi related news, I created a restic role for centos
16:28:50 <nirik> cool!
16:28:53 <Arrfab> so "archiving" local backup through encrypted remote backup to s3, with rotation and also deduplication (if you don't know the tool)
16:29:26 <nirik> we should look at it... we currently use rdiff-backup, but it's got a lot of limitations...
16:29:26 <smooge> ok going to end the meeting in a short bit
16:30:04 <Arrfab> and when creating that, I thought about simple policy for S3 bucket : each service would require its own account, with policy with minimum, identified through source ip, and which bucket (only one)
16:30:09 <smooge> #endmeeting