14:01:17 <smooge> #startmeeting Infrastructure (2018-06-07) 14:01:17 <zodbot> Meeting started Thu Jun 7 14:01:17 2018 UTC. 14:01:17 <zodbot> This meeting is logged and archived in a public location. 14:01:17 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:01:17 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 14:01:17 <zodbot> The meeting name has been set to 'infrastructure_(2018-06-07)' 14:01:17 <smooge> #meetingname infrastructure 14:01:17 <zodbot> The meeting name has been set to 'infrastructure' 14:01:17 <smooge> #topic aloha 14:01:17 <smooge> #chair smooge relrod nirik pingou puiterwijk tflink 14:01:17 <zodbot> Current chairs: nirik pingou puiterwijk relrod smooge tflink 14:01:23 <smooge> Good morning everyone 14:01:29 <nirik> good morning. 14:01:31 * relrod waves 14:01:33 <abompard> .hello2 14:01:34 <zodbot> abompard: abompard 'Aurelien Bompard' <aurelien@bompard.org> 14:01:35 <marc84> good morning everyone 14:01:37 <creaked> morning 14:01:41 <creaked> .hello2 14:01:42 <zodbot> creaked: creaked 'Will Chellman' <creaked@gmail.com> 14:01:44 <tflink> .hello2 14:01:45 <zodbot> tflink: tflink 'Tim Flink' <tflink@redhat.com> 14:01:46 <aikidouke> .hello2 14:01:48 <zodbot> aikidouke: aikidouke 'None' <zachvatwork@gmail.com> 14:01:50 <vgolwala> .hello2 14:01:53 <zodbot> vgolwala: vgolwala 'Vismay Golwala' <vgolwala@redhat.com> 14:02:58 <bowlofeggs> .hello2 14:02:59 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <rbarlow@redhat.com> 14:03:49 <threebean> .hello2 14:03:53 <zodbot> threebean: Sorry, but you don't exist 14:03:54 <contyk> o/ 14:03:56 <threebean> awwww 14:03:58 <contyk> .hello psabata 14:03:59 <zodbot> contyk: psabata 'Petr Å abata' <psabata@redhat.com> 14:04:04 <janslow> .hello janslow 14:04:04 <aikidouke> oh noes 14:04:04 <zodbot> janslow: janslow 'James Anslow' <james@ddxor.com> 14:04:40 <aikidouke> alas poor threebean... 14:04:52 <smooge> #topic New folks introductions 14:04:52 <smooge> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves 14:05:17 <smooge> Hello any new people wanting to introduce themselves 14:05:19 * cverna waves 14:05:59 <janslow> Hi, been around IRC and in the apprentice group in the past but dropped out. Back here now, did mail to the list a couple hours ago- so hi again :) 14:06:16 <janslow> Hoping to pick up some activities again, will be in IRC 14:06:16 <bowlofeggs> greeting janslow! 14:06:24 <nirik> welcome back. 14:06:37 <smooge> welcome back 14:06:41 <vgolwala> welcome back janslow 14:07:06 <janslow> Thanks all :) 14:08:03 * pingou here 14:08:07 <smooge> #topic announcements and information 14:08:08 <smooge> #info relrod PTO 9 Jun - 19 Jun 14:08:08 <smooge> #info smooge PTO 8 Jun, 15 Jun -> 18 Jun 14:08:08 <smooge> #info Need to work out oncall for 2018-06-15 -> 2018-06-21 14:08:08 <smooge> #info Office hour poll is available please add your preferences -> https://framadate.org/bhNm9poKGUV93hDG 14:08:09 <smooge> #info OSBS outage tomorrow 2018-06-08 09:00 UTC (switching registry authentication) 14:08:39 <nirik> there's also the qa db outage later today? 14:08:40 <smooge> #info QA outage tonight 2018-06-07 23:00 UTC 14:08:54 <smooge> yep 14:09:19 * nirik added those to the infrastructure-outage calendar. A great way to keep up on outages. 14:09:43 <threebean> o - is the QA outage about beefing up resultsdb? 14:09:53 <tflink> resultsdb stg, yes 14:10:11 <nirik> yeah. 14:10:20 <smooge> well resultsdb is not being touched. db-qa01 is getting moved to a more perfomant host 14:10:26 <threebean> cool, cool. 14:10:36 <nirik> it seems stg is getting swamped... we aren't sure why. 14:10:55 <nirik> it might be related to greenwave/waiverdb in stg... 14:10:56 <smooge> so this affects all qa will be affected because both stg and prod are on the same db 14:11:28 <tflink> not all qa 14:11:40 <tflink> but everything except prod resultsdb 14:12:00 <smooge> where does prod resultsdb go? 14:12:05 <tflink> we moved prod resultsdb to a dedicated db host a while back 14:12:09 <tflink> db-qa02.qa 14:12:32 <tflink> db vm, rather 14:12:33 <smooge> my grep missed that 14:12:40 <smooge> oops 14:12:50 <smooge> ok 14:12:57 <bowlofeggs> it seems like it would be good to figure out what changed 14:13:14 <bowlofeggs> because if we deploy what changed to prod without realizing it, that might not be great 14:13:19 <smooge> well the first thing is that db-qa01 is running on 2 CPUs and small ram 14:13:29 <tflink> AFAIK, the issue in stg coincided with an upgrade to greenwave and waiverdb in stg 14:13:30 <smooge> most of our dbs are running on a LOT more CPUs 14:13:54 <tflink> also AFAIK, prod resultsdb isn't seeing the same performance issues that stg is 14:13:57 <bowlofeggs> yeah but if the upgrade to greenwave/waiverdb is the cause, that would concern me 14:14:05 <tflink> why's that? 14:14:19 <smooge> bowlofeggs, I need to make the apples and oranges the same before I can say if there is a cause 14:14:21 <nirik> was kparal investigating? or should someone else? 14:14:22 <tflink> the db for stg is very underpowered and that's the bottleneck for resultsdb usually 14:14:30 <bowlofeggs> well, i wouldn't want that same upgrade to go to prod unless we know prod can handle it 14:14:36 <tflink> kparal and frantisekz were looking into it 14:14:43 <bowlofeggs> yeah it does sound reasonable to make the HW the same 14:14:44 <nirik> ok 14:15:00 <tflink> bowlofeggs: prod has a dedicated host for the db, everything else from dev to stg is sharing the same small db server 14:15:14 <bowlofeggs> yeah i agree that upgrading the db shoudl help 14:15:14 <tflink> prod resultsdb, rather 14:15:28 <smooge> so once they are made the same, and if the box then goes futznoid, then it is time to look at the deployment 14:15:29 <bowlofeggs> i'm advocating that we should *also* pause prod deployments of greenwave/waiverdb 14:15:36 <bowlofeggs> until we know that it was jsut HW 14:15:40 <tflink> openqa, taskotron's buildbots, dev/stg resultsdb 14:15:42 <threebean> bowlofeggs: they are paused. 14:15:53 <tflink> ok, I misunderstood 14:16:12 <threebean> indefinitely, until we sort out everything we want to in staging, including both bodhi integration (that UX milestone) and resultsdb performance. 14:16:49 <bowlofeggs> cool. my concerns are assuaged :) 14:16:51 <tflink> would it be better to just move stg resultsdb to a new vm and keep everything else as is for now? 14:17:06 <tflink> so that it matches prod resultsdb and doesn't need downtime for everything else 14:17:29 <nirik> well, I think db-qa01 should get more cpus/mem in any case... 14:17:39 * tflink feels like he's ratholing the meeting 14:17:57 <nirik> but we could split out stg resultsdb if you like, that seems reasonable. 14:18:06 <threebean> tflink++ 14:18:06 <zodbot> threebean: Karma for tflink changed to 1 (for the f28 release cycle): https://badges.fedoraproject.org/tags/cookie/any 14:18:12 <smooge> ok any other announcements? 14:18:22 * tflink doesn't care much either way, we've just been upgrading/changing stuff as it's needed 14:20:02 <smooge> I think we need to do 1 thing at a time. a) move db-qa01 to more cpu/ram tonight. b) evaluate how it is working. c) look at moving other tools to another db if needed 14:20:26 <pingou> +1 14:20:35 <tflink> works for me, the bigger downtime is moving db-qa01.qa 14:20:56 <smooge> d) redesign it to have a db-qa01-stg.qa for staging items 14:21:04 <smooge> or some similar thing 14:21:11 <smooge> ok next up 14:21:14 <smooge> #topic Oncall 14:21:14 <smooge> #info Smooge is on call from 2018-05-29->2018-06-07 14:21:14 <smooge> #info Nirik is on call from 2018-06-07->2018-06-14 14:21:14 <smooge> #info (!Smooge|!Relrod) is on call from 2018-06-15->2018-06-18 14:21:15 <smooge> #info who fills the above 14:21:29 <nirik> I can just take it. 14:21:36 <nirik> I should be around... 14:22:08 <smooge> relrod, would you be available to take it after your PTO is over? 14:22:32 <relrod> smooge: sure, after the 19th 14:22:41 <smooge> ok 14:22:48 <smooge> #info Nirik is on call from 2018-06-07->2018-06-19 14:23:10 <smooge> #info Relrod is on call from 2018-06-20->2018-06-26 14:24:31 <smooge> #topic Flock - nirik 14:24:52 <nirik> hey, so I just wanted to mention flock... this year in dresden from aug 8-11th 14:25:01 * pingou won't attend this year :( 14:25:03 <nirik> https://flocktofedora.org/ 14:25:15 <nirik> pingou: š¢ 14:25:30 <abompard> pingou: oh noes 14:25:48 <nirik> also, I wanted to see if we could coordinate any talks... I was thinking of doing a infra workshop/hackfest like always... 14:26:01 <bowlofeggs> i was considering doing a talk "bring your app to openshift" 14:26:08 <nirik> and possibly something else, but I am not sure what. I have a lot of ideas, but not sure what would be of interest. 14:26:14 <bowlofeggs> about how infra does openshift with our ansible stuff 14:26:21 <puiterwijk> bowlofeggs: awesome. 14:26:25 <bowlofeggs> basically, hijack puiterwijk's work and pass it off as my own 14:26:26 * relrod is planning to do a talk about revamping our Apprentice program. 14:26:29 <puiterwijk> bowlofeggs: how about doing that together? :) 14:26:35 <bowlofeggs> puiterwijk: sure ;) 14:26:38 <puiterwijk> Or you can steal it, works for me 14:26:43 <nirik> bowlofeggs: cool. Although that title might make people bring their... less than steller thing and ask us to run it 14:27:02 <bowlofeggs> nirik: yeah i meant to convey that it was for infra apps 14:27:03 * puiterwijk notes nirik is trying to be politically correct :) 14:27:04 <pingou> we'll pretend the title doesn't imply our openshift :D 14:27:45 <bowlofeggs> nirik: i tried to put a stellar app in openshift, and it melted the physical host 14:27:51 <bowlofeggs> turns out that stars are really hot 14:28:03 <bowlofeggs> it didn't just melt it, it turned it into a nuclear plasma 14:28:04 <pingou> "Bring your app to OpenShift \n\n\n <span style="tiny">(Not necessarily, Fedora Infra's openshift)</span>" 14:28:11 <bowlofeggs> hahaha 14:28:55 <nirik> I was considering a openshift for infra roadmap/state of type thing... or another rawhide talk, or a 'talk to your infrastructure' type thing where people help us prioritize or fix nagging issues, or... 14:29:54 <nirik> but it would be good if we at least let everyone know what we are planning so we don't overlap... or we do, but co-present things then. :) 14:31:27 <bowlofeggs> +1 14:32:05 <cverna> should we start put these ideas in the flock issue tracker 14:32:07 <nirik> anyhow, just wanted to bring it up and toss out some ideas... we can move on now unless anyone has more on it 14:32:36 <nirik> cverna: well, once we submit there we are committed to doing it... so sure, but only if we are sure what we want to do 14:33:07 <cverna> nirik: ok :) 14:33:18 <nirik> I think relrod already has his in... 14:33:28 <relrod> yep 14:33:50 <nirik> https://pagure.io/flock/issues is current talk submissions. 14:34:01 <nirik> (not accepted, just submitted) 14:35:32 <smooge> ok any other ideas for flock infra talks? 14:36:34 <smooge> #topic Tickets discussion 14:36:34 <smooge> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=2 14:36:37 <bowlofeggs> i wonder if a best practices talk could be good 14:36:39 <bowlofeggs> oh sorry 14:36:40 <bowlofeggs> haha 14:36:57 <nirik> bowlofeggs: it could yeah. 14:37:07 <smooge> https://pagure.io/fedora-infrastructure/issue/6966 14:37:07 <nirik> so I did mark a few tickets meeting... 14:37:14 <smooge> #info https://pagure.io/fedora-infrastructure/issue/6966 14:37:27 <smooge> nirik sorry this is your salso 14:37:28 <nirik> so this is to set a memory limit on koji requests. 14:37:46 <nirik> smooge: no no, thanks for showing them... 14:38:03 <nirik> I was hoping to get puiterwijk's take on this one... 14:38:15 * puiterwijk checks 14:38:19 <nirik> it does seem good to set a limit rather than hit OOM 14:38:36 <puiterwijk> Ah. Yes. But also we should get the API call removed... 14:39:04 <puiterwijk> i.e. listRPMs() with arg None should just not be a valid call, in my opinion 14:39:07 <puiterwijk> Anyway. Sure. 14:39:19 <nirik> ok, what should we set it to though? 14:39:24 <bowlofeggs> 96 GiB 14:39:47 <nirik> 640k? thats enough for anyone 14:39:53 <bowlofeggs> haha 14:40:00 <puiterwijk> Well, it shouldn't be trying to keep everything in memory.... So I'd say in the order of <1GB 14:40:18 <puiterwijk> (if it has >1GB of stuff in memory.... there's something weird) 14:40:31 <nirik> ok, we can try 1GB 14:40:37 <nirik> and see what happens 14:40:53 <nirik> smooge: next ticket please? 14:41:13 <smooge> trying 14:41:36 <smooge> #info https://pagure.io/fedora-infrastructure/issue/5120 14:41:48 <smooge> my firefox went on a walkabout 14:41:53 <puiterwijk> smooge: maybe use .ticket? 14:41:55 <puiterwijk> .ticket 5120 14:41:59 <zodbot> puiterwijk: Issue #5120: [RFR] zanata2fedmsg - fedora-infrastructure - Pagure - https://pagure.io/fedora-infrastructure/issue/5120 14:42:09 <puiterwijk> So, on this one, I actually did some stuff the other day. 14:42:15 <nirik> ah yeah, I see threebean replied here... he can't sponsor this one anymore... 14:42:16 <puiterwijk> I gavce a summary in their ticket 14:42:21 * threebean waves 14:42:26 <puiterwijk> nirik: I've sorta taken it over unofficially... 14:42:31 <threebean> yeah - the motivation for zanata2fedmsg was essentially to enable some new badges. 14:42:41 <puiterwijk> But someone needs to write more code 14:42:42 <threebean> I don't know if we had any other concrete needs for it. 14:42:48 <nirik> puiterwijk: ok, great. 14:43:00 <nirik> puiterwijk: I didn't see the summary in there? did it not post? 14:43:01 <puiterwijk> https://github.com/fedora-infra/zanata2fedmsg/projects/1 - that is the stuff I identified so far that's critical 14:43:14 <puiterwijk> nirik: https://github.com/fedora-infra/zanata2fedmsg/issues/1#issuecomment-392697199 14:43:24 <puiterwijk> I did not see the fedora-infra ticket.... This is where everyone pinged me on 14:43:35 <nirik> ah.. ok. 14:44:00 <nirik> ok then, can point to that and just wait more for implementation. Next! 14:44:10 <threebean> wait, wait. who is asking for forward motion on that RFR these days? 14:44:31 <threebean> is there a need? if not, we might be able to drop it, close the rfr, and close down the github project. reduce clutter(!) 14:44:47 <puiterwijk> threebean: jflory7 and dchen 14:45:04 <threebean> cool, if they have a use, then that's that. :) 14:45:13 <puiterwijk> I'm not opposed to doing that, definitely not. Just thought I'd point out where we were 14:45:51 <puiterwijk> And we might even be able to shove one of them onto the Zanata folks... 14:46:08 <nirik> is it worth asking for sure if they want it still? or it's clear that they do? 14:46:18 <smooge> ok so it sounds like we need input and if we don't get it in N days we clear it? 14:46:26 * threebean nods 14:46:28 <puiterwijk> nirik: I am reasonably sure Justin still wants it for badges... 14:46:39 <threebean> yeah, and that makes sense. 14:46:45 <nirik> ok, sure. 14:46:45 <puiterwijk> smooge: no, we need development work done. Either on zanata2fedmsg or zanata 14:46:54 <threebean> the translators deserve good representation in the badges app.. which they don't get today. 14:46:58 <puiterwijk> (well, and at least one word changfe on z2f) 14:46:59 <smooge> puiterwijk, well that is what I mean my input 14:47:10 <threebean> (packagers are and have been over-represented there.) 14:47:24 <puiterwijk> If we can shove the cron stuff over to the zanata side, it should be a single word change on our side 14:47:35 <puiterwijk> I'll bug Alex about it tonight 14:48:37 <nirik> puiterwijk++ 14:49:29 <nirik> next ticket? 14:49:54 <smooge> that was it 14:49:58 <nirik> cool. :) 14:50:11 * nirik thinks it's good to go over these old tickets in meeting... 14:50:15 <smooge> nope not 14:50:19 <smooge> .ticket 4806 14:50:21 <zodbot> smooge: Issue #4806: It's to bite the bullet and change occurances of "CLA" to "FPCA". - fedora-infrastructure - Pagure - https://pagure.io/fedora-infrastructure/issue/4806 14:50:33 <cverna> yes it is nice to go through the tickets 14:51:00 <smooge> what needs to be done on this ticket? 14:51:13 <puiterwijk> smooge: wait for CAIAPI, as nirik commented 14:51:16 <nirik> oh yeah, so this is something we should consider in the new caapi world 14:51:20 <nirik> but... 14:51:20 <bowlofeggs> i don't want to bite any bulletsā¦ 14:51:39 * smooge is also getting hit by notifications and pages so has to remind people we are ending in 8 minutes 14:51:41 <nirik> should we really be tracking it in infra? or should we open a new issue on caapi somewhere? or in it's design docs? 14:52:17 <puiterwijk> nirik: For now, let's track it here. Once I get time to start on that, I'll bring it over 14:52:46 <nirik> ok, fair enough. It just seemed like it would be better next to any devel work. 14:52:52 <nirik> ok, move on 14:53:38 <smooge> ok 14:53:45 <aikidouke> was going to mention 5290, but it can wait. I left comment on issue 14:53:50 <aikidouke> .ticket 5290 14:53:54 <zodbot> aikidouke: Issue #5290: Generate infrastructure map - fedora-infrastructure - Pagure - https://pagure.io/fedora-infrastructure/issue/5290 14:54:40 <smooge> take the ticket if you want 14:54:59 <smooge> aikidouke, the comment looks good idea 14:55:10 <smooge> argh english not working 14:55:29 <aikidouke> read fine to me smooge :) 14:55:42 <smooge> relrod, would it be ok if I put your talk about for next-next meeting? 14:55:49 <relrod> smooge: yep 14:55:59 <aikidouke> I can do some more digging and put together something for next meeting-ish 14:56:01 <aikidouke> ty 14:56:07 <smooge> no problem and thank you 14:56:34 <smooge> I am going to ask that apprentices with questions ask on #fedora-admin today in lieu of office minutes 14:56:40 <smooge> and go to open floor 14:56:45 <smooge> #topic open floor 14:57:14 <threebean> a question! 14:57:18 <smooge> ok 14:57:26 <puiterwijk> threebean: no 14:57:32 <puiterwijk> Sorry, what was your question? 14:57:39 <threebean> do we know when folks will be getting together to start talking PDC? last update on the list was that we'd do it sometime after GDPR fun settled down. 14:57:41 <threebean> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/thread/JHKHWYU5XK7H2P2QZZCCQR4ZRCTY3OSB/ 14:58:12 <smooge> next week? 14:58:14 <pingou> threebean: I've been meaning to send back an email about this 14:58:23 <pingou> I was thinking something like Tuesday next week 14:59:00 <smooge> ok I am going to close out this meeting in 1 minutes 14:59:01 <threebean> sure :) please see if you can invite cqi. he's interested. 14:59:20 <threebean> real quick - second thing - we talked on the list about wanting some kind of factory2 interaction with this meeting 14:59:22 <threebean> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/thread/LPCRSNGLPC2M3ZX6DPXSADWDAGWUAQPD/ 14:59:47 <threebean> I'll plan to be back next week, if that's helpful. 15:00:03 <smooge> ok how about putting it on the agenda for next weeks meeting 15:00:11 <threebean> sure. is that still in gobby? 15:00:17 <nirik> that would be lovely. Nice to see your nick. ;) 15:01:07 <threebean> :) 15:01:14 <smooge> threebean, yep. ifininote.fedoraproject.org 15:01:34 <puiterwijk> infininote.fedoraproject.org 15:01:38 <puiterwijk> or, easier, gobby.fedoraproject.org :) 15:01:59 <threebean> cool. thanks! 15:01:59 <smooge> ok time to shut this down so other meetings can start. 15:02:07 <smooge> #endmeeting