18:00:26 <smooge> #startmeeting Infrastructure (2017-09-14) 18:00:26 <zodbot> Meeting started Thu Sep 14 18:00:26 2017 UTC. The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:26 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:26 <zodbot> The meeting name has been set to 'infrastructure_(2017-09-14)' 18:00:26 <smooge> #meetingname infrastructure 18:00:26 <zodbot> The meeting name has been set to 'infrastructure' 18:00:26 <smooge> #topic aloha 18:00:26 <smooge> #chair smooge relrod nirik abadger1999 dgilmore threebean pingou puiterwijk pbrobinson 18:00:26 <zodbot> Current chairs: abadger1999 dgilmore nirik pbrobinson pingou puiterwijk relrod smooge threebean 18:00:39 <smooge> #topic New folks introductions 18:00:40 <smooge> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves 18:00:52 <marc84> hi everyone 18:00:54 <nirik> morning 18:01:02 <profnixer> Bonsoir ;) 18:01:14 <puiterwijk> nirik: evening 18:01:18 <netcronin> Hi all. 18:01:31 * ole88 waves 18:01:51 <lauratait> hello all 18:02:32 * cverna waves 18:02:58 <smooge> #topic announcements and information 18:02:58 <smooge> #info PHX2 Colo Trip, Dec 4th - 9th 18:02:58 <smooge> #info Infrastructure will be down during that trip. 18:02:58 <smooge> #info beta freeze still going on. 18:02:59 <smooge> #info Moving .stg. ips ongoing 18:03:00 <smooge> #info high NFS4.1 traffic solved - kevin 18:03:03 <smooge> #info Moving i2 download server ongoing. 18:04:09 <maxamillion> .hello maxamillion 18:04:10 <zodbot> maxamillion: maxamillion 'Adam Miller' <maxamillion@gmail.com> 18:04:19 <dustymabe> .hello dustymabe 18:04:20 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dustymabe@redhat.com> 18:04:26 <smooge> The Beta Go/NoGo was a NoGo I believe 18:04:44 <nirik> yep. moving out another week of freeze 18:05:30 <smooge> nirik, do you want to outline the NFS 4.1 issue as you found it? 18:05:31 * relrod is semi-here 18:06:09 <nirik> sure 18:06:33 <nirik> basically we were seeing tons of i/o with nfs 4.1... but we had switched everything to 4.0... so it was a bit of a puzzle 18:06:49 <nirik> the netapp folks did a packet capture and saw errors coming from our rawhide-composer vm. 18:07:14 <nirik> I did see on it that it looked like it was doing lots of io and load was kinda high... but it was idle. No processes really doing anything. 18:07:29 <nirik> I tried umounting and mounting the nfs mounts again, but it didn't do anything. 18:07:32 <nirik> so I rebooted it. 18:07:43 <nirik> and that cleared up the phrantom nfs 4.1 traffic entirely. 18:09:05 <smooge> did the IO go down? 18:09:44 <nirik> yes way down 18:09:56 <nirik> from about 900MB/sec to about 300. 18:10:28 <nirik> and 30,000 IOPS to 20,000ish 18:10:56 <smooge> wow 18:11:28 <maxamillion> damn 18:11:31 <smooge> Are there any other announcements for the week? 18:12:08 <maxamillion> just be aware that OSBS in stage is busted and is going to be that way for a couple weeks while we wait on some things to land upstream 18:12:29 <maxamillion> the impact is basically zero, it's been busted for a few weeks and nobody has noticed to the best of my knowledge 18:12:36 <maxamillion> but wanted folks to be aware 18:13:04 <smooge> what does OSBS tie into in staging? 18:13:21 <maxamillion> koji 18:13:25 <maxamillion> koji -> OSBS 18:13:52 <maxamillion> so basically right now koji will send builds to OSBS for layered container builds but nothing will happen, because things get hung up indefinitely on the OSBS side 18:14:06 <smooge> ah thanks that was the next q from me 18:14:27 <smooge> thanks for the update 18:14:29 <maxamillion> if this becomes a problem, I can change a few playbooks, destroy it all and reprovision the old system in it's place 18:14:39 <maxamillion> sure thing 18:16:04 <smooge> ok I am not sure who is using it but if it is we will go through a ticket and FBR process for it 18:16:35 <smooge> OK next up.. things we should discuss. No one changed my topic so here it is 18:16:38 <smooge> #topic Sleep and why you should get lots 18:16:46 <threebean> sleep++ 18:16:46 <zodbot> threebean: Karma for sleep changed to 1 (for the f26 release cycle): https://badges.fedoraproject.org/tags/cookie/any 18:16:54 <puiterwijk> jetlag-- 18:17:12 <smooge> Sleep is an important part of every one's life. You should sleep when your body says it needs it even if it is in a meeting 18:17:27 <nirik> yeah, completely agreed. 18:17:27 <smooge> which from the nodding off I can tell is happening right now 18:17:31 * nirik goes to take a nap 18:17:46 <maxamillion> smooge: oh, only stage is affected ... prod is in good working order and I have the playbooks sectioned out so they won't overlap accidentally 18:17:49 <puiterwijk> Shall we call it a day and jsut spend the rest of the meeting on naps? 18:17:53 <maxamillion> smooge: or shouldn't ;) 18:18:07 <smooge> puiterwijk, that is possible 18:18:24 <smooge> maxamillion, cool. better than mine 18:18:31 <smooge> OH I had something 18:18:51 <smooge> #topic AWX and the bright bold Future 18:19:16 <smooge> So last week Ansible was able to make the big bold opensourcing of Tower as AWX. 18:19:33 <smooge> puiterwijk, stickster and maxamillion ? were all there for different things 18:20:29 <puiterwijk> I've been looking at how we can deploy AWX in Infra, and was planning to write it down and send an overview to the list later today 18:20:37 <puiterwijk> Especially since packaging of AWX is... slightly tricky 18:20:54 <dgilmore> puiterwijk: it looks like a huge challenge to package 18:21:10 <puiterwijk> dgilmore: oh, it's a hell.... It's not supposed to be deployed outside of containers 18:21:25 <smooge> well we can run containers on batcontainer 18:21:33 <dgilmore> puiterwijk: thats eems to be the new world 18:21:37 <puiterwijk> smooge: something like that was my idea, yes 18:21:47 <dgilmore> sems to be even 18:21:52 <dgilmore> gahh 18:21:53 <dgilmore> seems 18:21:56 <maxamillion> smooge: sorry 18:22:15 <maxamillion> puiterwijk: +1 18:22:30 <smooge> puiterwijk, cool. 18:22:51 <smooge> I look forward to having signed containers of it from staging real soon :P 18:23:06 <smooge> ok next up 18:23:06 * puiterwijk notes we *have* signed containers 18:23:26 * dgilmore looks forward to deploying on arm at home 18:23:50 <puiterwijk> Do note that if I don't misremember, there hasn't been an AWX release since opensourcing yet 18:24:03 <smooge> I expect it will be a rawhide 18:24:04 <puiterwijk> And there's at least one patch that I submitted after opensourcing that we'll need 18:24:07 <nirik> I wonder if they are making any moves to make things more portable... since I expect every distro will have issues with old versions of things. 18:24:07 <nirik> anyhow, look forward to puiterwijk's post. 18:24:07 <nirik> AWX will be very handy for a number of our uses. 18:24:30 <puiterwijk> nirik: no, last I heard they didn't plan on any other release mechanism 18:24:40 <nirik> thats sad. 18:24:41 <nirik> ok 18:24:51 <puiterwijk> The community might, but that's going to be hard. 18:25:03 <puiterwijk> The Ansible folks didn't plan on working on it, I think 18:25:52 <smooge> anything else on AWX etc ? 18:25:57 <dustymabe> nope 18:26:05 <smooge> or ansiblefest or ... 18:26:09 <smooge> ok then 18:26:14 <dustymabe> i do have one request/question for open floor 18:26:17 <smooge> #topic Apprentice Office Hours 18:26:50 <netcronin> Apprentice here, any admins need help on projects? 18:27:04 <netcronin> from a sys-admin perspective. 18:27:11 <dustymabe> FYI - i'm also now apprentice level 18:27:24 <smooge> Understood. At the moment I am just tyring to deal with budget requests and such 18:27:38 <nirik> netcronin: https://pagure.io/fedora-infrastructure/issue/6342 might be an interesting one... 18:27:39 <cverna> congrats dustymabe :) 18:27:49 <maxamillion> dustymabe: congrats! 18:27:58 <lauratait> okay will look at that site as well 18:28:20 <netcronin> nirik: I'll look at it, tahnks! 18:28:25 <netcronin> congrats dustymabe! 18:28:31 <nirik> https://pagure.io/fedora-infrastructure/issue/6294 might also be not too bad... just some apache config change I think? 18:28:33 <dustymabe> thanks all :) 18:29:05 <ole88> congrats dustymabe! 18:29:37 <lauratait> congrats dustymabe 18:29:57 <profnixer> congrats dustymabe :) 18:31:39 <cverna> On a more generic info I have started to look after fedora-packages and I hope to cut a release once I have the PR that remove pkgdb merged 18:32:08 <cverna> Then we have to see if we manage to move this app to rhel7 18:32:30 <nirik> cverna: awesome. :) 18:32:44 <nirik> it really is a neat app... just needs love 18:32:57 <smooge> ok anything else 18:33:11 <smooge> #topic Open Floor 18:33:18 <smooge> dustymabe, you had something? 18:33:36 <cverna> ryanlerch gave some love to it --> https://cloud.githubusercontent.com/assets/592259/23255660/332c68ac-fa08-11e6-929a-937a07d50c45.png 18:34:28 <maxamillion> ryanlerch++ 18:34:28 <zodbot> maxamillion: Karma for ryanlerch changed to 8 (for the f26 release cycle): https://badges.fedoraproject.org/tags/cookie/any 18:34:54 <dustymabe> smooge: yes. do we have an SOP for signing ostree commits? I'd like to be able to ask more than just puiterwijk to be able to sign them when robosig doesn't do it for some reason 18:35:06 <nirik> that also would be great to get released. ;) 18:35:35 <dustymabe> also FYI i wrote a tool to detect when no signatures exist for a commit 18:35:41 <dustymabe> http://ostree-signed-commit-checker-fooplay.origin.dustymabe.com/ 18:35:52 <dustymabe> that is a temporary URL. I'll find a better place to put it 18:36:21 <nirik> I think puiterwijk had a doc somewhere, looking. 18:36:35 <nirik> but note the number of people who can sign things is (very deliberately) small. 18:36:55 <puiterwijk> I think the people able to do this is me, Kevin and possibly Dennis 18:37:12 <maxamillion> dustymabe: origin.dustymabe.com ... you running an openshift environment behind there? 18:37:16 <dustymabe> sure. I just need to be able to ask more than one person to be able to do it 18:37:17 <dustymabe> maxamillion: yes 18:37:20 <maxamillion> dustymabe: rocking :) 18:37:30 <dustymabe> openshift on atomic host on digital ocean 18:37:38 <dustymabe> fedora atomic host 18:37:58 <maxamillion> dustymabe: +1 18:38:19 <nirik> I cant seem to find the doc. ;) puiterwijk: can you mail me and dgilmore a doc on that so we can do them too if needed. 18:38:37 <puiterwijk> nirik: sure. Although robosignatory-signatomic --help should give you the instructions 18:38:58 <puiterwijk> or -signostree. Just do robosignatory-sign<tab><tab> and it'll be the obvious one 18:39:00 <nirik> yeah, and I think you told me how on irc or something, I just can't find a doc off hand. :) 18:39:09 <puiterwijk> ack 18:39:23 <cverna> dustymabe++ for your blog post on AH, very interesting 18:39:23 <zodbot> cverna: Karma for dustymabe changed to 9 (for the f26 release cycle): https://badges.fedoraproject.org/tags/cookie/any 18:39:30 <cverna> s/post/posts 18:39:36 <dustymabe> cverna: thanks 18:39:47 <smooge> Well it would also be good to have a followed process. Like dustymabe opens a ticket in <fill in ticket system> so people can see it when they aren't in meetings etc, people on the list who can sign things take the ticket, do the ticket, close the ticket 18:39:59 <nirik> yeah. 18:40:09 <nirik> probibly releng pagure would make most sense. 18:40:21 <dustymabe> smooge: yeah a ticket was opened against atomic-wg for this 18:40:35 <dustymabe> i should have opened it in another place after the fact 18:40:42 <smooge> well a ticket when you need things signed. 18:40:55 <dustymabe> now I know who can do the signing I'll open the ticket and follow up with those people 18:41:12 <dustymabe> nirik: or dgilmore - do you want to go through the excercise for this set of commits? 18:41:14 <smooge> ok cool 18:41:18 <dustymabe> exercise 18:41:24 * dustymabe can't spell 18:41:46 <nirik> sure, we can do so out of meeting...? 18:41:51 <dustymabe> nirik: yep 18:42:01 <dustymabe> FYI this is the ticket that was opened against atomic-wg: https://pagure.io/atomic-wg/issue/330 18:42:08 <dustymabe> i don't expect you guys to follow that tracker 18:42:12 <dustymabe> it's just an FUI 18:42:15 <dustymabe> FYI* 18:43:47 <ole88> has anyone had the opportunity to review the app lists I posted to issue #6140 last month? 18:44:54 <ole88> if these lists work, the next step is to decide what the service levels are and get the docs finished 18:45:16 <smooge> .ticket 6140 18:45:18 <zodbot> smooge: Issue #6140: Create a Service Level Expectations document - fedora-infrastructure - Pagure - https://pagure.io/fedora-infrastructure/issue/6140 18:45:39 * nirik can look... will update ticket when I do 18:45:50 <ole88> ok, thanks nirik 18:46:23 <smooge> is having a hard time looking at the html files in the ticket.. sorry 18:47:25 <smooge> and there.. 18:47:29 <smooge> oh nice list ole88 18:47:35 <smooge> thank you for your work on it 18:47:56 <ole88> you're welcome... I can send copies of the html files if needed 18:48:24 <smooge> no its more that I was clicking on it and my browser was wanting me to download 18:48:42 <ole88> I exported them from a sqlite database I created... 18:48:43 <puiterwijk> smooge: yep. Which is done on purpose 18:49:05 <smooge> puiterwijk, I had thought it was a misconfiguration on my part so have been fixing it 18:49:08 <smooge> anyway. 18:49:13 <puiterwijk> smooge: no 18:49:40 <smooge> ole88, the only thing to add would be the new modular related tools.. greenwave? bluescreen? 18:49:41 <puiterwijk> We explicitly make Pagure tell your browser to download it, since otherwise you could use it as a way to inject javascript 18:50:34 <smooge> other than that I think it looks good 18:50:36 <ole88> smooge: are those listed on the Fedora project site somewhere? I can fill them in pretty easily... 18:50:45 <smooge> I don't know.. 18:50:52 <nirik> side note: apps.fedoraproject.org needs some updating. 18:52:24 <ole88> nirik: is there a ticket for that? 18:52:44 <nirik> ole88: no, but I can make one... for example pkgdb is going away, etc. 18:53:10 <nirik> and yeah, it doesn't have any openshift apps in it yet... 18:53:31 * nirik makes a note to file a ticket 18:53:40 <cverna> faitout seems to be wrong in the table too 18:53:47 <smooge> #action nirik is going to take a note to file a ticket 18:54:01 <smooge> ok we are coming up to the end of the hour and other meetings are on deck 18:54:24 <smooge> I will say thank you all for coming and anything left should move over to #fedora-admin 18:54:28 <smooge> #endmeeting