18:00:00 <nirik> #startmeeting Infrastructure (2017-01-19) 18:00:00 <zodbot> Meeting started Thu Jan 19 18:00:00 2017 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:00 <zodbot> The meeting name has been set to 'infrastructure_(2017-01-19)' 18:00:01 <nirik> #meetingname infrastructure 18:00:01 <zodbot> The meeting name has been set to 'infrastructure' 18:00:01 <nirik> #topic aloha 18:00:01 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore threebean pingou puiterwijk pbrobinson 18:00:01 <zodbot> Current chairs: abadger1999 dgilmore lmacken nirik pbrobinson pingou puiterwijk relrod smooge threebean 18:00:01 <nirik> #topic New folks introductions 18:00:10 <clime> hello 18:00:12 * pingou around but in and out for the first few minutes 18:00:24 <nirik> welcome everyone to another exciting Fedora Infrastructure meeting. ☄ 18:00:38 <puiterwijk> Hello 18:01:32 <linuxmodder> .fas linuxmodder 18:01:36 <zodbot> linuxmodder: linuxmodder 'Corey W Sheldon' <sheldon.corey@openmailbox.org> 18:02:22 <roshi> .fas roshi 18:02:23 <zodbot> roshi: roshitha33 'Roshitha Perera' <roshitha33@yahoo.com> - hiroshi 'hiro' <hirotakarazuka@nifty.cpm> - oqto 'Hiroshi Yamauchi' <ambrella@mac.com> - szkh9fed 'Hiroshi Suzuki' <suzuki@radio.ce.titech.ac.jp> - hi64g3 'Hiroshi H. Higashijima' <hiro4@hiroritan.com> - roshinitaj 'roshinitaj' <roshinitaj@incognitomail.org> - roshini 'roshini ravikumar' <mailmeroshini@gmail.com> - porochan 'hiroshi hisano' (2 more messages) 18:02:35 <nirik> ok, lets go ahead and dive in... 18:02:46 <nirik> any new folks like to give a short one line introduction ? 18:02:53 <nirik> 🎤 18:02:58 <roshi> o/ 18:03:04 * nirik is emjoi happy today. Probibly will break meetbot 18:03:34 <roshi> I'm roshi, and I'm recently back from hiatus, but plan to be more involved in infra now that I'm 100% around again 18:03:36 <athos> o/ 18:03:44 <linuxmodder> emojis not printing here so don't care much :) 18:03:46 <roshi> most of you probably know me from QA or the Cloud WG 18:03:46 <nirik> welcome back roshi 18:03:52 <roshi> :D 18:03:55 <tflink> .hello tflink 18:03:56 <zodbot> tflink: tflink 'Tim Flink' <tflink@redhat.com> 18:04:56 <nirik> alright then... lets go on to status/info dump 18:05:15 <nirik> #topic announcements and information 18:05:15 <nirik> #info problems with buildvms last week/weekend/this week, seem ok on 4.10rc4 - kevin 18:05:15 <nirik> #info problems with srpm download in koji, issue filed - kevin 18:05:15 <nirik> #info problems with dnf downloading packages for buildroots, bug filed - kevin 18:05:15 <nirik> #info bodhi-backend01 f25 instance ready for service - kevin 18:05:16 <nirik> #info bodhi-backend03 f24 instance can be retired as soon as 03 is tested - kevin 18:05:20 <nirik> #info some progress on s390 builder networking/firewalling - kevin 18:05:22 <nirik> #info taskotron production redeployed as f24/f25 - tflink 18:05:24 <nirik> #info nagios staging is still being worked on 18:05:26 <nirik> #info piwik moved from 2 to 3. -- thanks ricky 18:05:28 <nirik> #info found we had not gotten weblogs from several proxies which fixed stats drop 18:05:30 <nirik> #info found we had a problem with logging running out of file desc. fixing 18:05:53 <nirik> anything anyone wants to add or expand on there? 18:06:13 <linuxmodder> nfm 18:06:37 * relrod checks in 18:06:39 <nirik> #topic fedorahosted migration progress - kevin 18:06:49 <nirik> so, more things migrating this last week... 18:07:02 <bowlofeggs> .hello bowlofeggs 18:07:03 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <randy@electronsweatshop.com> 18:07:30 <jcline> .hello jcline 18:07:31 <zodbot> jcline: jcline 'Jeremy Cline' <jeremy@jcline.org> 18:07:52 <nirik> I made a wiki page... but it's still pretty out of date 18:07:54 <cep> .hello cep 18:07:55 <zodbot> cep: cep 'Pradeep CE' <breathingcode@gmail.com> 18:07:57 <nirik> https://fedoraproject.org/wiki/Infrastructure/Fedorahostedmigrations 18:08:10 <smooge> hello 18:08:11 <nirik> I also can't seem to get it to sort if anyone wants to try and fix that. ;) 18:08:38 <linuxmodder> can take a few today and tommorrow to birng up to date nirik 18:09:15 * doteast here and late 18:09:19 <nirik> sure. Mostly the problem is projects that have moved and we don't know it. 18:09:39 <nirik> so, go to the project page and check and see if they point people somewhere else, etc 18:09:58 <sayan> .hello sayanchowdhury 18:09:59 <zodbot> sayan: sayanchowdhury 'Sayan Chowdhury' <sayan.chowdhury2012@gmail.com> 18:10:20 <linuxmodder> nirik, I can reach out and ask/cehck for you 18:10:32 <nirik> sure, any updating welcome. :) 18:10:55 <nirik> also if there's projects anyone here is involved with, do migrate them somewhere or ask whoever is in charge of the project to do that 18:11:04 <nirik> #info migrations are continuing. 18:11:08 <linuxmodder> puiterwijk, the freeipaotp one was what you were talking about the other day in -security yes? on that wiki 18:11:14 <nirik> #info https://fedoraproject.org/wiki/Infrastructure/Fedorahostedmigrations needs updating 18:11:38 <puiterwijk> linuxmodder: no. My script is not anywhere there. I think that that was the upstream development of the code 18:11:52 <linuxmodder> puiterwijk, noted 18:12:06 <linuxmodder> #info linuxmodder to attempt updating as much as possible 18:12:12 <nirik> anyhow, do remind anyone you know to migrate. ;) 18:12:20 <linuxmodder> ack 18:12:26 * pingou reminds pingou to migrate! 18:12:32 <nirik> we still should try and schedule a meeting with freemedia folks and badges folks to talk about those migrations. 18:12:36 * pingou goes kicking pingou 18:12:46 <nirik> pingou: don't do that. You might make pingou mad. ;) 18:12:49 <pingou> nirik: the next release might help the badge workflow 18:12:58 <nirik> oh? cool. 18:13:03 <pingou> mreynolds added 'list' for custom fields 18:13:13 <nirik> nice 18:13:13 <pingou> so you can pick the values in a specified list 18:13:19 <pingou> + colored tags 18:13:24 <pingou> which is cool :) 18:13:26 <nirik> ooh... colors. 18:13:37 <nirik> next step: add emojies 18:13:43 <sayan> pingou++ for colored tags 18:13:46 <zodbot> sayan: Karma for pingou changed to 19 (for the f25 release cycle): https://badges.fedoraproject.org/tags/cookie/any 18:13:49 <pingou> it's all mreynolds :) 18:13:52 <pingou> so mreynolds++ 18:14:11 <sayan> mreynolds++ 18:14:11 <zodbot> sayan: Karma for mreynolds changed to 2 (for the f25 release cycle): https://badges.fedoraproject.org/tags/cookie/any 18:14:23 <nirik> excellent. 18:14:24 <pingou> mreynolds++ 18:14:28 <linuxmodder> nirik, I am in freemedia we plan to next week last i saw 18:14:55 <nirik> linuxmodder: ok, will take some work... freemedia is a weird setup. ☹ 18:15:02 <linuxmodder> no kidding 18:15:11 <nirik> but we will get it. 18:15:18 <nirik> ok, anything else on this or shall we move on? 18:15:31 <linuxmodder> ~> 18:15:59 <nirik> #topic fas3 status? 18:16:10 <nirik> so, where are we on fas3? anything folks can help out with? 18:16:32 <pingou> I heard SmootherFrOgZ and puiterwijk synced up over the week-end 18:16:46 <pingou> puiterwijk: could you mention the next steps? 18:16:50 <linuxmodder> I'm woefully out of loop on fas3 but willign to help 18:16:59 * cverna is here 18:17:21 <puiterwijk> pingou: I have synced up with what is needed from FAS from Ipsilon's point of view, so next up is SmootherFrOgZ working on what I pointed out 18:17:40 <pingou> ok cool :) 18:17:49 <SmootherFrOgZ> I will have something by next week 18:17:57 <nirik> cool. ;) 18:18:01 <nirik> whats the next steps after that? 18:18:06 <SmootherFrOgZ> will prepare a release so we can update stg 18:18:10 <puiterwijk> SmootherFrOgZ: I have been working on doing the fas-sync code though, so I'll be submitting a PR for that one of these days 18:18:45 <SmootherFrOgZ> puiterwijk: k 18:19:10 <nirik> ok. 18:19:34 <SmootherFrOgZ> Oh, we should have a new release ready for stg though. with a few fix and the fedoraproject's theme 18:19:54 <SmootherFrOgZ> so people can start play with it and give ryanlerch feedbacks and such 18:21:01 <nirik> so, are we going to try and move to sssd/kerberos or move to fas3 fasClient then kerberos? or unknown yet? 18:22:53 <SmootherFrOgZ> I'd say sssd/kerberos unless puiterwijk has another thought on this 18:22:55 <puiterwijk> I would be ready to move to sssd/kerberos whenever. So I think that that might be the ideal way forward 18:23:28 <puiterwijk> Now that we have all the group info synced to IPA, that is doable 18:23:33 <nirik> ok. That would include packagers/pkgs? 18:23:45 <puiterwijk> They can retain ssh key for now 18:23:57 <puiterwijk> (would still be sssd, so no fasClient) 18:24:06 <nirik> ah, ok. 18:24:33 <nirik> BTW, has anyone talked with centos/rpmfusion? are they going to move to 3? or ? 18:25:04 <SmootherFrOgZ> rpmfusion, for sure 18:25:21 <SmootherFrOgZ> centos, we need to talk to them about 18:25:21 <puiterwijk> I'll discuss with CentOS. I think they'll be following as well 18:25:31 <SmootherFrOgZ> puiterwijk: thanks 18:26:23 <nirik> and both may look at freeipa synced to it also? 18:26:29 <nirik> I guess thats up to them... 18:26:36 <SmootherFrOgZ> btw, we would like to officially move fas2 as support only thus make fas3 branch the new develop (or merge it into) and open a support branch on latest FAS2's release 18:26:40 <pingou> SmootherFrOgZ: going to fosdem? 18:26:41 <puiterwijk> I have a talk with CentOS people about that soon already 18:26:55 <SmootherFrOgZ> pingou: nope, I'll be in the US on February 18:27:11 <puiterwijk> nirik: ^ ("that" is the hybrid approach) 18:28:09 <nirik> ok, cool. 18:28:14 <nirik> SmootherFrOgZ: +1 18:28:25 <nirik> fas2 has kinda been support/security only for a long while. 18:28:56 <SmootherFrOgZ> yeah but when you go to the code page it doesn't look like it :) 18:29:07 <SmootherFrOgZ> alrighty then 18:29:30 <SmootherFrOgZ> puiterwijk: let me know when you can push the release branch to stg so we can move forward on that one 18:29:30 <nirik> yeah, true 18:29:38 <pingou> SmootherFrOgZ: +1 on making FAS3 default 18:29:46 <nirik> #info SmootherFrOgZ will make some more fas3 fixes and do a release soon 18:29:59 <nirik> #info fas3.stg will get new release and theme to test things out with 18:30:19 <nirik> #info infra will move to sssd soon to avoid having to move to fas3 fasClient. 18:30:23 <nirik> anything else here? 18:31:08 <smooge> fas came up on lwn.net today 18:31:21 <nirik> oh? 18:31:24 <smooge> as an alternative to persona 18:31:40 <pingou> fas or ipsilon? 18:31:49 <pingou> (or did they mix up?) 18:32:00 <puiterwijk> alternative to Persona?! That... is not FAS 18:32:07 <smooge> or something like that. I just put a comment about development and SmootherFrOgZ on it 18:32:39 <nirik> huh. 18:32:55 <nirik> yeah, I saw the persona story... not sure why they would mention fas, but ok. 18:33:10 <smooge> its not in the story mentioning fas.. someone in the comments 18:33:12 <SmootherFrOgZ> hah! 18:34:07 <nirik> ah, ok 18:34:23 <nirik> ok, shall we move on? 18:34:30 <nirik> #topic fedmsg policy - Patrick 18:34:43 <nirik> puiterwijk: ⁉ 18:34:55 <puiterwijk> So, as some of you might have realized, last week was another round of security issues in our infra. This time it was in fedmsg. 18:35:42 <puiterwijk> So, as a part of that, the fedmsg topic policy that describes who is allowed to send what, was never verified. As a result, nobody has ever noticed that our policies were... way off, and invalid in a lot of cases. 18:36:38 <puiterwijk> So, I will be sending an overview of all topics that were sent in the last 6 months to the infra list somewhere this week, and would like everyone to identify whether or not that topic was sent by the correct servers, as I've been working on getting this policy fixed based on the historical data 18:37:04 <adrianr> is this related to the missing messages for umdl? 18:37:08 <puiterwijk> So, when I send that, could anyone send me an off-list reply if anything is wrong? 18:37:10 <smooge> that is going to be a loooot to go through 18:37:23 <puiterwijk> adrianr: yes. The policy for umdl was totally wrong, and refused the boxes that were actually sending the messages 18:37:27 <linuxmodder> puiterwijk, ack 18:38:06 <puiterwijk> smooge: well, I've written some scripts that do most of the analysis and that collapsed it down to so far 400 pairs, most of which I can resolve myself. So it's not too bad 18:38:06 <adrianr> ok, the latest rawhide push has not been picked up by umdl and I think I have to run it manually 18:38:34 <puiterwijk> adrianr: huh, I thought I fixed that the day after you mentioned it. So if it's still an issue, please ping me and I'll verify again 18:39:03 <adrianr> no org.fedoraproject.prod.compose.rawhide.rsync.complete messages since 2017-01-11 18:39:11 <adrianr> accodring to datagrepper 18:39:16 <adrianr> according* 18:39:21 <puiterwijk> Huh, okay. I'll check the policy once more and make sure it's applied. Thanks for the info 18:39:39 <puiterwijk> If anyone else is missing messages, also ping me, but it will most likely be in the overview I'll be sending out 18:40:44 <puiterwijk> Anyway, that was it. If anyone has questions about this, please email me directly or send me a PM. 18:41:20 <nirik> puiterwijk: so you are going to fix up all of them you can easily and just mail the list about ones that are questionable? 18:41:39 <nirik> #info fedmsg policy was broken 18:41:42 <puiterwijk> nirik: yes. I will also send the list of things I fixed up, so other people can inform me if I made educated mistakes 18:41:57 <nirik> #info need to revalidate and correct what hosts are allowed to send what messages. 18:42:08 <puiterwijk> (but I'll try to make sure it's clear what I still need feedback on, and what is just "please check I didn't make stupid mistakes") 18:42:20 <nirik> #info puiterwijk is going to correct the ones he can, then mail the list the corrections plus any questionable ones 18:42:32 <nirik> ok, sounds good. 18:42:39 <nirik> anything else on this? 18:42:45 <puiterwijk> That was it from me 18:43:30 <nirik> #topic Apprentice Open office hours 18:43:41 <nirik> any apprentices have questions or comments or the like? 18:44:12 * roshi has nothing off hand 18:44:28 <roshi> but I'll be mostly pestering tflink about QA services and whatnot 18:44:45 <nirik> ok. As always feel free to ask questions on the list or in our various irc channels. :) 18:44:55 * tflink gets the feeling that it would be a good time to go afk ... 18:44:56 <tflink> :-P 18:44:58 <roshi> will do :) 18:45:11 <roshi> what's umdl? (that was one question I had during the meeting) 18:45:11 <nirik> ha 18:45:19 <nirik> Update mirror directory list 18:45:28 <roshi> ah, thanks 18:45:38 <nirik> it's the thing in mirrormanager that sees new content and updates the database with it. 18:46:22 <nirik> ok, I wanted to talk about kojipkgs a bit since I had to deal with it this week so much 18:46:34 <nirik> #topic Learn about:kojipkgs.fedoraproject.org - kevin 18:47:09 <nirik> So, kojipkgs is a config parameter that koji has... it's basically the server to download all packages it needs for buildroots etc. 18:47:29 <nirik> If you have a very simple/small koji config, you can just set this to the same server, ie, koji.whatever 18:47:42 <nirik> We have a more complex setup. :) 18:48:22 <nirik> Our kojipkgs has been a seperate server that runs squid to listen for requests, then apache to serve the orig content and nfs mounts to get the data from... 18:49:00 <nirik> so, if someone requests foo-1.0-1.noarch.rpm from it... it hits squid. If squid has it cached it just sends it back. If not, it asks apache, and apache serves it from the nfs filesystem. 18:49:17 <dgilmore> yippe serving from ram 18:49:18 <nirik> this reduces load on the nfs side 18:49:35 <puiterwijk> dgilmore: well, RAM and local disk 18:49:36 <nirik> and for things in the default buildroots they get download... a lot 18:50:15 <nirik> so, this setup has worked for us, but it's also a single point of failure. If that kojipgks01 server was down, all builds would fail. 18:50:25 <dgilmore> puiterwijk: sure, it was supposed to always be ram only :) 18:50:28 <nirik> So, I stupidly tried to get rid of this SPOF 18:50:45 <nirik> we now have a kojipgks01 and kojipkgs02 18:51:05 <smooge> dgilmore, we never got the money for the 2 TB memory needed 18:51:15 <nirik> The orig idea was to point all the builders/koji to our proxy servers, they would in turn use haproxy over the pair of squid servers. 18:51:44 <nirik> but sadly, koji and dnf didn't like that. 18:51:59 <smooge> yum on the other hand... 18:52:03 <nirik> so, currently, kojipgks.fedoraproject.org internally to phx2 is pointing to kojipkgs01 18:52:11 <dgilmore> yum loves everything 18:52:15 <nirik> (as it was before) 18:52:35 <nirik> and I guess it will stay this way until we fix the issues we hit 18:53:02 <puiterwijk> nirik: I think we did reconfigure koji to use kojipkgs01 and kojipkgs02 as fallback mirrors? Or did we revert that? 18:53:21 <nirik> I had to revert all that. 18:53:28 <puiterwijk> Ah, okay. Too bad 18:53:37 <nirik> koji's src.rpm download fails unless it's pointed directly at a single kojipkgs 18:53:42 <sayan> so kojipkgs02 is not used? 18:53:48 <linuxmodder> nirik, are those in same dc still tho ? seeing as we've had dc outages recently too iirc 18:54:12 <puiterwijk> sayan: not from internally. From outside PHX2, it is I think 18:54:15 <nirik> sayan: internally, no. externally it still is. 18:54:26 <nirik> linuxmodder: yes, everything is in the same datacenter 18:54:35 <nirik> well, all these things. 18:54:36 <linuxmodder> in phx2 i assume 18:54:41 * nirik nods 18:55:00 <sayan> Okay 18:55:24 <nirik> so I am going to try and isolate the koji thing more... hopefully we can figure out why it's failing in that case. 18:55:31 <linuxmodder> and do the apache serving operations go internally only if the initial fails? 18:55:33 <nirik> I think we worked around the dnf case so it doesn't matter anymore 18:56:05 <linuxmodder> the src.rpm bit I mean if that fails 18:56:54 <nirik> koji's src.rpm download is very very simple 18:57:13 <nirik> it uses pythons urllib2 and downloads a file 18:57:15 <nirik> it doesn't check it 18:57:21 <nirik> it doesn't retr 18:57:27 <nirik> it just downloads it and goes on 18:58:13 <linuxmodder> so then I'm lost what failure would cause it to call on apache and nfs 18:58:35 <nirik> thats the kojipgks setup. 18:58:39 <nirik> squid -> apache -> nfs 18:59:04 <nirik> when koji requests a src.rpm it just hits the top part of that. The squid->apache->nfs machine has the correct src.rpm. 18:59:12 <nirik> if koji hits it directly it works 18:59:28 <nirik> if it hits it via another proxy->haproxy or via round robin dns it fails. 18:59:40 <nirik> so, koji needs more smarts on downloading 18:59:57 <linuxmodder> ah 19:00:12 <linuxmodder> so if its not cached and it has to go thru a proxy then 19:00:15 <nirik> anyhow, I can try and test it more and isolate it 19:00:34 <nirik> we are running out of time. ;) 19:00:38 <nirik> #topic Open Floor 19:00:42 <adrianr> I have/had two short mirrormanager related topics 19:00:42 <nirik> anyone have anything for open floor? 19:01:21 <linuxmodder> adrianr, ^ 19:01:43 <nirik> adrianr: go ahead 19:01:46 <adrianr> one is the missing org.fedoraproject.prod.compose.rawhide.rsync.complete which is now kind of solved and not that bad 19:01:57 <adrianr> as we get messages for fedora-updates 19:02:12 <adrianr> the other is about http://download.fedoraproject.org/pub/epel and its redirects 19:02:28 <adrianr> I was asked if http://download.fedoraproject.org/pub/epel should redirect to https or not 19:02:40 <adrianr> right now it redirects to whatever mm returns first 19:03:00 <smooge> i thought that was a RHEL5 need to not do so 19:03:02 <nirik> hum, I guess I wonder what yum does there. 19:03:09 <adrianr> somehow it is correct that if asking for http, getting a https mirror is unexpected 19:03:34 <adrianr> should be easy to fix in the apache rewrites 19:03:41 <linuxmodder> but it shouldn't fail out id think 19:04:05 <linuxmodder> short of a cert error at least or so id think 19:04:39 <adrianr> I can try to prepare a fix and send it around for review 19:04:43 <nirik> yeah, I'm not sure if I feel strongly about this at all. 19:04:51 <nirik> since people shouldn't really be using that anyhow. ;) 19:05:45 <nirik> anyone have anything else? 19:05:52 <linuxmodder> using what http or the redirect? 19:06:09 <nirik> download.fedoraproject.org/pub/epel 19:06:20 <nirik> the default/right thing to do is use the metalink. 19:08:49 <nirik> ok, if nothing else will close in some time less than a minute 19:09:05 <nirik> #endmeeting