14:01:17 <smooge> #startmeeting Infrastructure (2018-06-07)
14:01:17 <zodbot> Meeting started Thu Jun  7 14:01:17 2018 UTC.
14:01:17 <zodbot> This meeting is logged and archived in a public location.
14:01:17 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:17 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:01:17 <zodbot> The meeting name has been set to 'infrastructure_(2018-06-07)'
14:01:17 <smooge> #meetingname infrastructure
14:01:17 <zodbot> The meeting name has been set to 'infrastructure'
14:01:17 <smooge> #topic aloha
14:01:17 <smooge> #chair smooge relrod nirik pingou puiterwijk tflink
14:01:17 <zodbot> Current chairs: nirik pingou puiterwijk relrod smooge tflink
14:01:23 <smooge> Good morning everyone
14:01:29 <nirik> good morning.
14:01:31 * relrod waves
14:01:33 <abompard> .hello2
14:01:34 <zodbot> abompard: abompard 'Aurelien Bompard' <aurelien@bompard.org>
14:01:35 <marc84> good morning everyone
14:01:37 <creaked> morning
14:01:41 <creaked> .hello2
14:01:42 <zodbot> creaked: creaked 'Will Chellman' <creaked@gmail.com>
14:01:44 <tflink> .hello2
14:01:45 <zodbot> tflink: tflink 'Tim Flink' <tflink@redhat.com>
14:01:46 <aikidouke> .hello2
14:01:48 <zodbot> aikidouke: aikidouke 'None' <zachvatwork@gmail.com>
14:01:50 <vgolwala> .hello2
14:01:53 <zodbot> vgolwala: vgolwala 'Vismay Golwala' <vgolwala@redhat.com>
14:02:58 <bowlofeggs> .hello2
14:02:59 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <rbarlow@redhat.com>
14:03:49 <threebean> .hello2
14:03:53 <zodbot> threebean: Sorry, but you don't exist
14:03:54 <contyk> o/
14:03:56 <threebean> awwww
14:03:58 <contyk> .hello psabata
14:03:59 <zodbot> contyk: psabata 'Petr Šabata' <psabata@redhat.com>
14:04:04 <janslow> .hello janslow
14:04:04 <aikidouke> oh noes
14:04:04 <zodbot> janslow: janslow 'James Anslow' <james@ddxor.com>
14:04:40 <aikidouke> alas poor threebean...
14:04:52 <smooge> #topic New folks introductions
14:04:52 <smooge> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
14:05:17 <smooge> Hello any new people wanting to introduce themselves
14:05:19 * cverna waves
14:05:59 <janslow> Hi, been around IRC and in the apprentice group in the past but dropped out. Back here now, did mail to the list a couple hours ago- so hi again :)
14:06:16 <janslow> Hoping to pick up some activities again, will be in IRC
14:06:16 <bowlofeggs> greeting janslow!
14:06:24 <nirik> welcome back.
14:06:37 <smooge> welcome back
14:06:41 <vgolwala> welcome back janslow
14:07:06 <janslow> Thanks all :)
14:08:03 * pingou here
14:08:07 <smooge> #topic announcements and information
14:08:08 <smooge> #info relrod PTO 9 Jun - 19 Jun
14:08:08 <smooge> #info smooge PTO 8 Jun, 15 Jun -> 18 Jun
14:08:08 <smooge> #info Need to work out oncall for 2018-06-15 -> 2018-06-21
14:08:08 <smooge> #info Office hour poll is available please add your preferences -> https://framadate.org/bhNm9poKGUV93hDG
14:08:09 <smooge> #info OSBS outage tomorrow 2018-06-08 09:00 UTC (switching registry authentication)
14:08:39 <nirik> there's also the qa db outage later today?
14:08:40 <smooge> #info QA outage tonight 2018-06-07 23:00 UTC
14:08:54 <smooge> yep
14:09:19 * nirik added those to the infrastructure-outage calendar. A great way to keep up on outages.
14:09:43 <threebean> o - is the QA outage about beefing up resultsdb?
14:09:53 <tflink> resultsdb stg, yes
14:10:11 <nirik> yeah.
14:10:20 <smooge> well resultsdb is not being touched. db-qa01 is getting moved to a more perfomant host
14:10:26 <threebean> cool, cool.
14:10:36 <nirik> it seems stg is getting swamped... we aren't sure why.
14:10:55 <nirik> it might be related to greenwave/waiverdb in stg...
14:10:56 <smooge> so this affects all qa will be affected because both stg and prod are on the same db
14:11:28 <tflink> not all qa
14:11:40 <tflink> but everything except prod resultsdb
14:12:00 <smooge> where does prod resultsdb go?
14:12:05 <tflink> we moved prod resultsdb to a dedicated db host a while back
14:12:09 <tflink> db-qa02.qa
14:12:32 <tflink> db vm, rather
14:12:33 <smooge> my grep missed that
14:12:40 <smooge> oops
14:12:50 <smooge> ok
14:12:57 <bowlofeggs> it seems like it would be good to figure out what changed
14:13:14 <bowlofeggs> because if we deploy what changed to prod without realizing it, that might not be great
14:13:19 <smooge> well the first thing is that db-qa01 is running on 2 CPUs and small ram
14:13:29 <tflink> AFAIK, the issue in stg coincided with an upgrade to greenwave and waiverdb in stg
14:13:30 <smooge> most of our dbs are running on a LOT more CPUs
14:13:54 <tflink> also AFAIK, prod resultsdb isn't seeing the same performance issues that stg is
14:13:57 <bowlofeggs> yeah but if the upgrade to greenwave/waiverdb is the cause, that would concern me
14:14:05 <tflink> why's that?
14:14:19 <smooge> bowlofeggs, I need to make the apples and oranges the same before I can say if there is a cause
14:14:21 <nirik> was kparal investigating? or should someone else?
14:14:22 <tflink> the db for stg is very underpowered and that's the bottleneck for resultsdb usually
14:14:30 <bowlofeggs> well, i wouldn't want that same upgrade to go to prod unless we know prod can handle it
14:14:36 <tflink> kparal and frantisekz were looking into it
14:14:43 <bowlofeggs> yeah it does sound reasonable to make the HW the same
14:14:44 <nirik> ok
14:15:00 <tflink> bowlofeggs: prod has a dedicated host for the db, everything else from dev to stg is sharing the same small db server
14:15:14 <bowlofeggs> yeah i agree that upgrading the db shoudl help
14:15:14 <tflink> prod resultsdb, rather
14:15:28 <smooge> so once they are made the same, and if the box then goes futznoid, then it is time to look at the deployment
14:15:29 <bowlofeggs> i'm advocating that we should *also* pause prod deployments of greenwave/waiverdb
14:15:36 <bowlofeggs> until we know that it was jsut HW
14:15:40 <tflink> openqa, taskotron's buildbots, dev/stg resultsdb
14:15:42 <threebean> bowlofeggs: they are paused.
14:15:53 <tflink> ok, I misunderstood
14:16:12 <threebean> indefinitely, until we sort out everything we want to in staging, including both bodhi integration (that UX milestone) and resultsdb performance.
14:16:49 <bowlofeggs> cool. my concerns are assuaged :)
14:16:51 <tflink> would it be better to just move stg resultsdb to a new vm and keep everything else as is for now?
14:17:06 <tflink> so that it matches prod resultsdb and doesn't need downtime for everything else
14:17:29 <nirik> well, I think db-qa01 should get more cpus/mem in any case...
14:17:39 * tflink feels like he's ratholing the meeting
14:17:57 <nirik> but we could split out stg resultsdb if you like, that seems reasonable.
14:18:06 <threebean> tflink++
14:18:06 <zodbot> threebean: Karma for tflink changed to 1 (for the f28 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
14:18:12 <smooge> ok any other announcements?
14:18:22 * tflink doesn't care much either way, we've just been upgrading/changing stuff as it's needed
14:20:02 <smooge> I think we need to do 1 thing at a time. a) move db-qa01 to more cpu/ram tonight. b) evaluate how it is working. c) look at moving other tools to another db if needed
14:20:26 <pingou> +1
14:20:35 <tflink> works for me, the bigger downtime is moving db-qa01.qa
14:20:56 <smooge> d) redesign it to have a db-qa01-stg.qa for staging items
14:21:04 <smooge> or some similar thing
14:21:11 <smooge> ok next up
14:21:14 <smooge> #topic Oncall
14:21:14 <smooge> #info Smooge is on call from 2018-05-29->2018-06-07
14:21:14 <smooge> #info Nirik is on call from 2018-06-07->2018-06-14
14:21:14 <smooge> #info (!Smooge|!Relrod) is on call from 2018-06-15->2018-06-18
14:21:15 <smooge> #info who fills the above
14:21:29 <nirik> I can just take it.
14:21:36 <nirik> I should be around...
14:22:08 <smooge> relrod, would you be available to take it after your PTO is over?
14:22:32 <relrod> smooge: sure, after the 19th
14:22:41 <smooge> ok
14:22:48 <smooge> #info Nirik is on call from 2018-06-07->2018-06-19
14:23:10 <smooge> #info Relrod is on call from 2018-06-20->2018-06-26
14:24:31 <smooge> #topic Flock - nirik
14:24:52 <nirik> hey, so I just wanted to mention flock... this year in dresden from aug 8-11th
14:25:01 * pingou won't attend this year :(
14:25:03 <nirik> https://flocktofedora.org/
14:25:15 <nirik> pingou: 😢
14:25:30 <abompard> pingou: oh noes
14:25:48 <nirik> also, I wanted to see if we could coordinate any talks... I was thinking of doing a infra workshop/hackfest like always...
14:26:01 <bowlofeggs> i was considering doing a talk "bring your app to openshift"
14:26:08 <nirik> and possibly something else, but I am not sure what. I have a lot of ideas, but not sure what would be of interest.
14:26:14 <bowlofeggs> about how infra does openshift with our ansible stuff
14:26:21 <puiterwijk> bowlofeggs: awesome.
14:26:25 <bowlofeggs> basically, hijack puiterwijk's work and pass it off as my own
14:26:26 * relrod is planning to do a talk about revamping our Apprentice program.
14:26:29 <puiterwijk> bowlofeggs: how about doing that together? :)
14:26:35 <bowlofeggs> puiterwijk: sure ;)
14:26:38 <puiterwijk> Or you can steal it, works for me
14:26:43 <nirik> bowlofeggs: cool. Although that title might make people bring their... less than steller thing and ask us to run it
14:27:02 <bowlofeggs> nirik: yeah i meant to convey that it was for infra apps
14:27:03 * puiterwijk notes nirik is trying to be politically correct :)
14:27:04 <pingou> we'll pretend the title doesn't imply our openshift :D
14:27:45 <bowlofeggs> nirik:  i tried to put a stellar app in openshift, and it melted the physical host
14:27:51 <bowlofeggs> turns out that stars are really hot
14:28:03 <bowlofeggs> it didn't just melt it, it turned it into a nuclear plasma
14:28:04 <pingou> "Bring your app to OpenShift \n\n\n <span style="tiny">(Not necessarily, Fedora Infra's openshift)</span>"
14:28:11 <bowlofeggs> hahaha
14:28:55 <nirik> I was considering a openshift for infra roadmap/state of type thing... or another rawhide talk, or a 'talk to your infrastructure' type thing where people help us prioritize or fix nagging issues, or...
14:29:54 <nirik> but it would be good if we at least let everyone know what we are planning so we don't overlap... or we do, but co-present things then. :)
14:31:27 <bowlofeggs> +1
14:32:05 <cverna> should we start put these ideas in the flock issue tracker
14:32:07 <nirik> anyhow, just wanted to bring it up and toss out some ideas... we can move on now unless anyone has more on it
14:32:36 <nirik> cverna: well, once we submit there we are committed to doing it... so sure, but only if we are sure what we want to do
14:33:07 <cverna> nirik: ok :)
14:33:18 <nirik> I think relrod already has his in...
14:33:28 <relrod> yep
14:33:50 <nirik> https://pagure.io/flock/issues is current talk submissions.
14:34:01 <nirik> (not accepted, just submitted)
14:35:32 <smooge> ok any other ideas for flock infra talks?
14:36:34 <smooge> #topic Tickets discussion
14:36:34 <smooge> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=2
14:36:37 <bowlofeggs> i wonder if a best practices talk could be good
14:36:39 <bowlofeggs> oh sorry
14:36:40 <bowlofeggs> haha
14:36:57 <nirik> bowlofeggs: it could yeah.
14:37:07 <smooge> https://pagure.io/fedora-infrastructure/issue/6966
14:37:07 <nirik> so I did mark a few tickets meeting...
14:37:14 <smooge> #info https://pagure.io/fedora-infrastructure/issue/6966
14:37:27 <smooge> nirik sorry this is your salso
14:37:28 <nirik> so this is to set a memory limit on koji requests.
14:37:46 <nirik> smooge: no no, thanks for showing them...
14:38:03 <nirik> I was hoping to get puiterwijk's take on this one...
14:38:15 * puiterwijk checks
14:38:19 <nirik> it does seem good to set a limit rather than hit OOM
14:38:36 <puiterwijk> Ah. Yes. But also we should get the API call removed...
14:39:04 <puiterwijk> i.e. listRPMs() with arg None should just not be a valid call, in my opinion
14:39:07 <puiterwijk> Anyway. Sure.
14:39:19 <nirik> ok, what should we set it to though?
14:39:24 <bowlofeggs> 96 GiB
14:39:47 <nirik> 640k? thats enough for anyone
14:39:53 <bowlofeggs> haha
14:40:00 <puiterwijk> Well, it shouldn't be trying to keep everything in memory.... So I'd say in the order of <1GB
14:40:18 <puiterwijk> (if it has >1GB of stuff in memory.... there's something weird)
14:40:31 <nirik> ok, we can try 1GB
14:40:37 <nirik> and see what happens
14:40:53 <nirik> smooge: next ticket please?
14:41:13 <smooge> trying
14:41:36 <smooge> #info https://pagure.io/fedora-infrastructure/issue/5120
14:41:48 <smooge> my firefox went on a walkabout
14:41:53 <puiterwijk> smooge: maybe use .ticket?
14:41:55 <puiterwijk> .ticket 5120
14:41:59 <zodbot> puiterwijk: Issue #5120: [RFR] zanata2fedmsg - fedora-infrastructure - Pagure - https://pagure.io/fedora-infrastructure/issue/5120
14:42:09 <puiterwijk> So, on this one, I actually did some stuff the other day.
14:42:15 <nirik> ah yeah, I see threebean replied here... he can't sponsor this one anymore...
14:42:16 <puiterwijk> I gavce a summary in their ticket
14:42:21 * threebean waves
14:42:26 <puiterwijk> nirik: I've sorta taken it over unofficially...
14:42:31 <threebean> yeah - the motivation for zanata2fedmsg was essentially to enable some new badges.
14:42:41 <puiterwijk> But someone needs to write more code
14:42:42 <threebean> I don't know if we had any other concrete needs for it.
14:42:48 <nirik> puiterwijk: ok, great.
14:43:00 <nirik> puiterwijk: I didn't see the summary in there? did it not post?
14:43:01 <puiterwijk> https://github.com/fedora-infra/zanata2fedmsg/projects/1  - that is the stuff I identified so far that's critical
14:43:14 <puiterwijk> nirik: https://github.com/fedora-infra/zanata2fedmsg/issues/1#issuecomment-392697199
14:43:24 <puiterwijk> I did not see the fedora-infra ticket.... This is where everyone pinged me on
14:43:35 <nirik> ah.. ok.
14:44:00 <nirik> ok then, can point to that and just wait more for implementation. Next!
14:44:10 <threebean> wait, wait.  who is asking for forward motion on that RFR these days?
14:44:31 <threebean> is there a need?  if not, we might be able to drop it, close the rfr, and close down the github project.  reduce clutter(!)
14:44:47 <puiterwijk> threebean: jflory7 and dchen
14:45:04 <threebean> cool, if they have a use, then that's that.  :)
14:45:13 <puiterwijk> I'm not opposed to doing that, definitely not. Just thought I'd point out where we were
14:45:51 <puiterwijk> And we might even be able to shove one of them onto the Zanata folks...
14:46:08 <nirik> is it worth asking for sure if they want it still? or it's clear that they do?
14:46:18 <smooge> ok so it sounds like we need input and if we don't get it in N days we clear it?
14:46:26 * threebean nods
14:46:28 <puiterwijk> nirik: I am reasonably sure Justin still wants it for badges...
14:46:39 <threebean> yeah, and that makes sense.
14:46:45 <nirik> ok, sure.
14:46:45 <puiterwijk> smooge: no, we need development work done. Either on zanata2fedmsg or zanata
14:46:54 <threebean> the translators deserve good representation in the badges app.. which they don't get today.
14:46:58 <puiterwijk> (well, and at least one word changfe on z2f)
14:46:59 <smooge> puiterwijk, well that is what I mean my input
14:47:10 <threebean> (packagers are and have been over-represented there.)
14:47:24 <puiterwijk> If we can shove the cron stuff over to the zanata side, it should be a single word change on our side
14:47:35 <puiterwijk> I'll bug Alex about it tonight
14:48:37 <nirik> puiterwijk++
14:49:29 <nirik> next ticket?
14:49:54 <smooge> that was it
14:49:58 <nirik> cool. :)
14:50:11 * nirik thinks it's good to go over these old tickets in meeting...
14:50:15 <smooge> nope not
14:50:19 <smooge> .ticket 4806
14:50:21 <zodbot> smooge: Issue #4806: It's to bite the bullet and change occurances of "CLA" to "FPCA". - fedora-infrastructure - Pagure - https://pagure.io/fedora-infrastructure/issue/4806
14:50:33 <cverna> yes it is nice to go through the tickets
14:51:00 <smooge> what needs to be done on this ticket?
14:51:13 <puiterwijk> smooge: wait for CAIAPI, as nirik commented
14:51:16 <nirik> oh yeah, so this is something we should consider in the new caapi world
14:51:20 <nirik> but...
14:51:20 <bowlofeggs> i don't want to bite any bullets…
14:51:39 * smooge is also getting hit by notifications and pages so has to remind people we are ending in 8 minutes
14:51:41 <nirik> should we really be tracking it in infra? or should we open a new issue on caapi somewhere? or in it's design docs?
14:52:17 <puiterwijk> nirik: For now, let's track it here. Once I get time to start on that, I'll bring it over
14:52:46 <nirik> ok, fair enough. It just seemed like it would be better next to any devel work.
14:52:52 <nirik> ok, move on
14:53:38 <smooge> ok
14:53:45 <aikidouke> was going to mention 5290, but it can wait. I left comment on issue
14:53:50 <aikidouke> .ticket 5290
14:53:54 <zodbot> aikidouke: Issue #5290: Generate infrastructure map - fedora-infrastructure - Pagure - https://pagure.io/fedora-infrastructure/issue/5290
14:54:40 <smooge> take the ticket if you want
14:54:59 <smooge> aikidouke, the comment looks good idea
14:55:10 <smooge> argh english not working
14:55:29 <aikidouke> read fine to me smooge :)
14:55:42 <smooge> relrod, would it be ok if I put your talk about for next-next meeting?
14:55:49 <relrod> smooge: yep
14:55:59 <aikidouke> I can do some more digging and put together something for next meeting-ish
14:56:01 <aikidouke> ty
14:56:07 <smooge> no problem and thank you
14:56:34 <smooge> I am going to ask that apprentices with questions ask on #fedora-admin today in lieu of office minutes
14:56:40 <smooge> and go to open floor
14:56:45 <smooge> #topic open floor
14:57:14 <threebean> a question!
14:57:18 <smooge> ok
14:57:26 <puiterwijk> threebean: no
14:57:32 <puiterwijk> Sorry, what was your question?
14:57:39 <threebean> do we know when folks will be getting together to start talking PDC?  last update on the list was that we'd do it sometime after GDPR fun settled down.
14:57:41 <threebean> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/thread/JHKHWYU5XK7H2P2QZZCCQR4ZRCTY3OSB/
14:58:12 <smooge> next week?
14:58:14 <pingou> threebean: I've been meaning to send back an email about this
14:58:23 <pingou> I was thinking something like Tuesday next week
14:59:00 <smooge> ok I am going to close out this meeting in 1 minutes
14:59:01 <threebean> sure :)  please see if you can invite cqi.  he's interested.
14:59:20 <threebean> real quick - second thing - we talked on the list about wanting some kind of factory2 interaction with this meeting
14:59:22 <threebean> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/thread/LPCRSNGLPC2M3ZX6DPXSADWDAGWUAQPD/
14:59:47 <threebean> I'll plan to be back next week, if that's helpful.
15:00:03 <smooge> ok how about putting it on the agenda for next weeks meeting
15:00:11 <threebean> sure.  is that still in gobby?
15:00:17 <nirik> that would be lovely. Nice to see your nick. ;)
15:01:07 <threebean> :)
15:01:14 <smooge> threebean, yep. ifininote.fedoraproject.org
15:01:34 <puiterwijk> infininote.fedoraproject.org
15:01:38 <puiterwijk> or, easier, gobby.fedoraproject.org :)
15:01:59 <threebean> cool.  thanks!
15:01:59 <smooge> ok time to shut this down so other meetings can start.
15:02:07 <smooge> #endmeeting