18:00:26 <smooge> #startmeeting Infrastructure (2017-09-14)
18:00:26 <zodbot> Meeting started Thu Sep 14 18:00:26 2017 UTC.  The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:26 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:26 <zodbot> The meeting name has been set to 'infrastructure_(2017-09-14)'
18:00:26 <smooge> #meetingname infrastructure
18:00:26 <zodbot> The meeting name has been set to 'infrastructure'
18:00:26 <smooge> #topic aloha
18:00:26 <smooge> #chair smooge relrod nirik abadger1999 dgilmore threebean pingou puiterwijk pbrobinson
18:00:26 <zodbot> Current chairs: abadger1999 dgilmore nirik pbrobinson pingou puiterwijk relrod smooge threebean
18:00:39 <smooge> #topic New folks introductions
18:00:40 <smooge> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
18:00:52 <marc84> hi everyone
18:00:54 <nirik> morning
18:01:02 <profnixer> Bonsoir ;)
18:01:14 <puiterwijk> nirik: evening
18:01:18 <netcronin> Hi all.
18:01:31 * ole88 waves
18:01:51 <lauratait> hello all
18:02:32 * cverna waves
18:02:58 <smooge> #topic announcements and information
18:02:58 <smooge> #info PHX2 Colo Trip, Dec 4th - 9th
18:02:58 <smooge> #info Infrastructure will be down during that trip.
18:02:58 <smooge> #info beta freeze still going on.
18:02:59 <smooge> #info Moving .stg. ips ongoing
18:03:00 <smooge> #info high NFS4.1 traffic solved - kevin
18:03:03 <smooge> #info Moving i2 download server ongoing.
18:04:09 <maxamillion> .hello maxamillion
18:04:10 <zodbot> maxamillion: maxamillion 'Adam Miller' <maxamillion@gmail.com>
18:04:19 <dustymabe> .hello dustymabe
18:04:20 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dustymabe@redhat.com>
18:04:26 <smooge> The Beta Go/NoGo was a NoGo I believe
18:04:44 <nirik> yep. moving out another week of freeze
18:05:30 <smooge> nirik, do you want to outline the NFS 4.1 issue as you found it?
18:05:31 * relrod is semi-here
18:06:09 <nirik> sure
18:06:33 <nirik> basically we were seeing tons of i/o with nfs 4.1... but we had switched everything to 4.0... so it was a bit of a puzzle
18:06:49 <nirik> the netapp folks did a packet capture and saw errors coming from our rawhide-composer vm.
18:07:14 <nirik> I did see on it that it looked like it was doing lots of io and load was kinda high... but it was idle. No processes really doing anything.
18:07:29 <nirik> I tried umounting and mounting the nfs mounts again, but it didn't do anything.
18:07:32 <nirik> so I rebooted it.
18:07:43 <nirik> and that cleared up the phrantom nfs 4.1 traffic entirely.
18:09:05 <smooge> did the IO go down?
18:09:44 <nirik> yes way down
18:09:56 <nirik> from about 900MB/sec to about 300.
18:10:28 <nirik> and 30,000 IOPS to 20,000ish
18:10:56 <smooge> wow
18:11:28 <maxamillion> damn
18:11:31 <smooge> Are there any other announcements for the week?
18:12:08 <maxamillion> just be aware that OSBS in stage is busted and is going to be that way for a couple weeks while we wait on some things to land upstream
18:12:29 <maxamillion> the impact is basically zero, it's been busted for a few weeks and nobody has noticed to the best of my knowledge
18:12:36 <maxamillion> but wanted folks to be aware
18:13:04 <smooge> what does OSBS tie into in staging?
18:13:21 <maxamillion> koji
18:13:25 <maxamillion> koji -> OSBS
18:13:52 <maxamillion> so basically right now koji will send builds to OSBS for layered container builds but nothing will happen, because things get hung up indefinitely on the OSBS side
18:14:06 <smooge> ah thanks that was the next q from me
18:14:27 <smooge> thanks for the update
18:14:29 <maxamillion> if this becomes a problem, I can change a few playbooks, destroy it all and reprovision the old system in it's place
18:14:39 <maxamillion> sure thing
18:16:04 <smooge> ok I am not sure who is using it but if it is we will go through a ticket and FBR process for it
18:16:35 <smooge> OK next up.. things we should discuss. No one changed my topic so here it is
18:16:38 <smooge> #topic Sleep and why you should get lots
18:16:46 <threebean> sleep++
18:16:46 <zodbot> threebean: Karma for sleep changed to 1 (for the f26 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
18:16:54 <puiterwijk> jetlag--
18:17:12 <smooge> Sleep is an important part of every one's life. You should sleep when your body says it needs it even if it is in a meeting
18:17:27 <nirik> yeah, completely agreed.
18:17:27 <smooge> which from the nodding off I can tell is happening right now
18:17:31 * nirik goes to take a nap
18:17:46 <maxamillion> smooge: oh, only stage is affected ... prod is in good working order and I have the playbooks sectioned out so they won't overlap accidentally
18:17:49 <puiterwijk> Shall we call it a day and jsut spend the rest of the meeting on naps?
18:17:53 <maxamillion> smooge: or shouldn't ;)
18:18:07 <smooge> puiterwijk, that is possible
18:18:24 <smooge> maxamillion, cool. better than mine
18:18:31 <smooge> OH I had something
18:18:51 <smooge> #topic AWX and the bright bold Future
18:19:16 <smooge> So last week Ansible was able to make the big bold opensourcing of Tower as AWX.
18:19:33 <smooge> puiterwijk, stickster and maxamillion ? were all there for different things
18:20:29 <puiterwijk> I've been looking at how we can deploy AWX in Infra, and was planning to write it down and send an overview to the list later today
18:20:37 <puiterwijk> Especially since packaging of AWX is... slightly tricky
18:20:54 <dgilmore> puiterwijk: it looks like a huge challenge to package
18:21:10 <puiterwijk> dgilmore: oh, it's a hell.... It's not supposed to be deployed outside of containers
18:21:25 <smooge> well we can run containers on batcontainer
18:21:33 <dgilmore> puiterwijk: thats eems to be the new world
18:21:37 <puiterwijk> smooge: something like that was my idea, yes
18:21:47 <dgilmore> sems to be even
18:21:52 <dgilmore> gahh
18:21:53 <dgilmore> seems
18:21:56 <maxamillion> smooge: sorry
18:22:15 <maxamillion> puiterwijk: +1
18:22:30 <smooge> puiterwijk, cool.
18:22:51 <smooge> I look forward to having signed containers of it from staging real soon :P
18:23:06 <smooge> ok next up
18:23:06 * puiterwijk notes we *have* signed containers
18:23:26 * dgilmore looks forward to deploying on arm at home
18:23:50 <puiterwijk> Do note that if I don't misremember, there hasn't been an AWX release since opensourcing yet
18:24:03 <smooge> I expect it will be a rawhide
18:24:04 <puiterwijk> And there's at least one patch that I submitted after opensourcing that we'll need
18:24:07 <nirik> I wonder if they are making any moves to make things more portable... since I expect every distro will have issues with old versions of things.
18:24:07 <nirik> anyhow, look forward to puiterwijk's post.
18:24:07 <nirik> AWX will be very handy for a number of our uses.
18:24:30 <puiterwijk> nirik: no, last I heard they didn't plan on any other release mechanism
18:24:40 <nirik> thats sad.
18:24:41 <nirik> ok
18:24:51 <puiterwijk> The community might, but that's going to be hard.
18:25:03 <puiterwijk> The Ansible folks didn't plan on working on it, I think
18:25:52 <smooge> anything else on AWX etc ?
18:25:57 <dustymabe> nope
18:26:05 <smooge> or ansiblefest or ...
18:26:09 <smooge> ok then
18:26:14 <dustymabe> i do have one request/question for open floor
18:26:17 <smooge> #topic Apprentice Office Hours
18:26:50 <netcronin> Apprentice here, any admins need help on projects?
18:27:04 <netcronin> from a sys-admin perspective.
18:27:11 <dustymabe> FYI - i'm also now apprentice level
18:27:24 <smooge> Understood. At the moment I am just tyring to deal with budget requests and such
18:27:38 <nirik> netcronin: https://pagure.io/fedora-infrastructure/issue/6342 might be an interesting one...
18:27:39 <cverna> congrats dustymabe :)
18:27:49 <maxamillion> dustymabe: congrats!
18:27:58 <lauratait> okay will look at that site as well
18:28:20 <netcronin> nirik: I'll look at it, tahnks!
18:28:25 <netcronin> congrats dustymabe!
18:28:31 <nirik> https://pagure.io/fedora-infrastructure/issue/6294 might also be not too bad... just some apache config change I think?
18:28:33 <dustymabe> thanks all :)
18:29:05 <ole88> congrats dustymabe!
18:29:37 <lauratait> congrats dustymabe
18:29:57 <profnixer> congrats dustymabe :)
18:31:39 <cverna> On a more generic info I have started to look after fedora-packages and I hope to cut a release once I have the PR that remove pkgdb merged
18:32:08 <cverna> Then we have to see if we manage to move this app to rhel7
18:32:30 <nirik> cverna: awesome. :)
18:32:44 <nirik> it really is a neat app... just needs love
18:32:57 <smooge> ok anything else
18:33:11 <smooge> #topic Open Floor
18:33:18 <smooge> dustymabe, you had something?
18:33:36 <cverna> ryanlerch gave some love to it --> https://cloud.githubusercontent.com/assets/592259/23255660/332c68ac-fa08-11e6-929a-937a07d50c45.png
18:34:28 <maxamillion> ryanlerch++
18:34:28 <zodbot> maxamillion: Karma for ryanlerch changed to 8 (for the f26 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
18:34:54 <dustymabe> smooge: yes. do we have an SOP for signing ostree commits? I'd like to be able to ask more than just puiterwijk to be able to sign them when robosig doesn't do it for some reason
18:35:06 <nirik> that also would be great to get released. ;)
18:35:35 <dustymabe> also FYI i wrote a tool to detect when no signatures exist for a commit
18:35:41 <dustymabe> http://ostree-signed-commit-checker-fooplay.origin.dustymabe.com/
18:35:52 <dustymabe> that is a temporary URL. I'll find a better place to put it
18:36:21 <nirik> I think puiterwijk had a doc somewhere, looking.
18:36:35 <nirik> but note the number of people who can sign things is (very deliberately) small.
18:36:55 <puiterwijk> I think the people able to do this is me, Kevin and possibly Dennis
18:37:12 <maxamillion> dustymabe: origin.dustymabe.com ... you running an openshift environment behind there?
18:37:16 <dustymabe> sure. I just need to be able to ask more than one person to be able to do it
18:37:17 <dustymabe> maxamillion: yes
18:37:20 <maxamillion> dustymabe: rocking :)
18:37:30 <dustymabe> openshift on atomic host on digital ocean
18:37:38 <dustymabe> fedora atomic host
18:37:58 <maxamillion> dustymabe: +1
18:38:19 <nirik> I cant seem to find the doc. ;) puiterwijk: can you mail me and dgilmore a doc on that so we can do them too if needed.
18:38:37 <puiterwijk> nirik: sure. Although robosignatory-signatomic --help should give you the instructions
18:38:58 <puiterwijk> or -signostree. Just do robosignatory-sign<tab><tab> and it'll be the obvious one
18:39:00 <nirik> yeah, and I think you told me how on irc or something, I just can't find a doc off hand. :)
18:39:09 <puiterwijk> ack
18:39:23 <cverna> dustymabe++ for your blog post on AH, very interesting
18:39:23 <zodbot> cverna: Karma for dustymabe changed to 9 (for the f26 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
18:39:30 <cverna> s/post/posts
18:39:36 <dustymabe> cverna: thanks
18:39:47 <smooge> Well it would also be good to have a followed process. Like dustymabe opens a ticket in <fill in ticket system> so people can see it when they aren't in meetings etc, people on the list who can sign things take the ticket, do the ticket, close the ticket
18:39:59 <nirik> yeah.
18:40:09 <nirik> probibly releng pagure would make most sense.
18:40:21 <dustymabe> smooge: yeah a ticket was opened against atomic-wg for this
18:40:35 <dustymabe> i should have opened it in another place after the fact
18:40:42 <smooge> well a ticket when you need things signed.
18:40:55 <dustymabe> now I know who can do the signing I'll open the ticket and follow up with those people
18:41:12 <dustymabe> nirik: or dgilmore - do you want to go through the excercise for this set of commits?
18:41:14 <smooge> ok cool
18:41:18 <dustymabe> exercise
18:41:24 * dustymabe can't spell
18:41:46 <nirik> sure, we can do so out of meeting...?
18:41:51 <dustymabe> nirik: yep
18:42:01 <dustymabe> FYI this is the ticket that was opened against atomic-wg: https://pagure.io/atomic-wg/issue/330
18:42:08 <dustymabe> i don't expect you guys to follow that tracker
18:42:12 <dustymabe> it's just an FUI
18:42:15 <dustymabe> FYI*
18:43:47 <ole88> has anyone had the opportunity to review the app lists I posted to issue #6140 last month?
18:44:54 <ole88> if these lists work, the next step is to decide what the service levels are and get the docs finished
18:45:16 <smooge> .ticket 6140
18:45:18 <zodbot> smooge: Issue #6140: Create a Service Level Expectations document - fedora-infrastructure - Pagure - https://pagure.io/fedora-infrastructure/issue/6140
18:45:39 * nirik can look... will update ticket when I do
18:45:50 <ole88> ok, thanks nirik
18:46:23 <smooge> is having a hard time looking at the html files in the ticket.. sorry
18:47:25 <smooge> and there..
18:47:29 <smooge> oh nice list ole88
18:47:35 <smooge> thank you for your work on it
18:47:56 <ole88> you're welcome...  I can send copies of the html files if needed
18:48:24 <smooge> no its more that I was clicking on it and my browser was wanting me to download
18:48:42 <ole88> I exported them from a sqlite database I created...
18:48:43 <puiterwijk> smooge: yep. Which is done on purpose
18:49:05 <smooge> puiterwijk, I had thought it was a misconfiguration on my part so have been fixing it
18:49:08 <smooge> anyway.
18:49:13 <puiterwijk> smooge: no
18:49:40 <smooge> ole88, the only thing to add would be the new modular related tools.. greenwave? bluescreen?
18:49:41 <puiterwijk> We explicitly make Pagure tell your browser to download it, since otherwise you could use it as a way to inject javascript
18:50:34 <smooge> other than that I think it looks good
18:50:36 <ole88> smooge: are those listed on the Fedora project site somewhere?  I can fill them in pretty easily...
18:50:45 <smooge> I don't know..
18:50:52 <nirik> side note: apps.fedoraproject.org needs some updating.
18:52:24 <ole88> nirik: is there a ticket for that?
18:52:44 <nirik> ole88: no, but I can make one... for example pkgdb is going away, etc.
18:53:10 <nirik> and yeah, it doesn't have any openshift apps in it yet...
18:53:31 * nirik makes a note to file a ticket
18:53:40 <cverna> faitout seems to be wrong in the table too
18:53:47 <smooge> #action nirik is going to take a note to file a ticket
18:54:01 <smooge> ok we are coming up to the end of the hour and other meetings are on deck
18:54:24 <smooge> I will say thank you all for coming and anything left should move over to #fedora-admin
18:54:28 <smooge> #endmeeting