15:04:57 <smooge> #startmeeting Infrastructure (2019-09-26) 15:04:57 <zodbot> Meeting started Thu Sep 26 15:04:57 2019 UTC. 15:04:57 <zodbot> This meeting is logged and archived in a public location. 15:04:57 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:04:57 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:04:57 <zodbot> The meeting name has been set to 'infrastructure_(2019-09-26)' 15:04:57 <smooge> #meetingname infrastructure 15:04:57 <smooge> #topic aloha 15:04:57 <zodbot> The meeting name has been set to 'infrastructure' 15:04:57 <smooge> #chair nirik pingou relrod smooge tflink cverna mizdebsk mkonecny abompard bowlofeggs 15:04:57 <zodbot> Current chairs: abompard bowlofeggs cverna mizdebsk mkonecny nirik pingou relrod smooge tflink 15:05:10 <mizdebsk> hello 15:05:10 <relrod> morning 15:05:14 <smooge> My apologies for lateness. I got looking at disk space issues and missed the clock 15:05:18 <mkonecny> .hello zlopez 15:05:20 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@packetseekers.eu> 15:05:45 <mkonecny> smooge: Don't worry, this could happen to everybody 15:06:20 <smooge> nirik has power issues so may not be as available here as usual 15:06:23 * cverna waves 15:06:27 <smooge> #topic Next chair 15:06:27 <smooge> #info magic eight ball says: 15:06:49 <cverna> I ll take next week chair 15:07:05 <smooge> bowlofeggs did the meeting last week. I am doing it this week. Next week is cverna 15:07:14 <smooge> hey magic eight ball said it too 15:07:20 <cverna> +1 15:07:23 <cverna> :-) 15:07:30 <smooge> #info cverna will chair the next meeting 15:07:38 <smooge> #topic New folks introductions 15:07:38 <smooge> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves 15:07:38 <smooge> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted 15:07:51 <nirik> morning 15:08:17 <austinpowered> .hello 15:08:17 <zodbot> austinpowered: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1". 15:08:24 <nirik> yeah, I might drop off... but should be able to get back on hopefully. 15:10:18 <smooge> ok 15:10:20 <smooge> next up 15:10:42 <smooge> #topic announcements and information 15:10:42 <smooge> #info We are looking for people to maintain Fedocal and Nuancier - mkonecny 15:10:50 <mizdebsk> #info Koschei prod was moved to OpenSift - mizdebsk 15:10:52 <smooge> #info Final freeze starts soon 15:11:02 <smooge> congrats mizdebsk 15:11:43 <smooge> any other announcements this week? 15:11:45 <nirik> well, we have all next week... I don't know if we want to think about a update cycle, or not 15:11:58 <nirik> yeah, great work mizdebsk. Good to get more things in openshift. 15:13:06 <smooge> not sure.. that is 3 days concentrated on reboots 15:13:32 <smooge> is there anything else we need to land before the freeze? 15:13:52 <smooge> if not we could schedule it and do it for Tues->Thurs 15:14:16 <nirik> I'm not sure it's worth it... let me look at whats pending and we can discuss out of meeting? 15:14:22 <smooge> sure 15:14:26 <mkonecny> I will probably update Anitya and the-new-hotness in production 15:14:30 <mizdebsk> i am working on improving monitoring for koschei in openshift 15:14:39 <smooge> #topic Oncall 15:14:39 <smooge> #info https://fedoraproject.org/wiki/Infrastructure/Oncall 15:14:39 <smooge> #info Summary of last week: (from bowlofeggs ) 15:14:47 <nirik> I would like to give kojidb more memory... (which will need a really short outage). Will try and do it off hours sometime. 15:14:58 <mizdebsk> (tickets #7588 and #7115) 15:15:12 <mizdebsk> since we don't have staging nagios and nagios is frozen, this has to happen between freezes 15:15:38 * nirik nods 15:16:48 <smooge> i think bowlofeggs is unavailable so will go over what I saw. pretty quiet week 15:17:01 <smooge> who is going to cover starting today? 15:17:05 <nirik> yeah, not too crazy... 15:17:14 <nirik> our ticket queue has been growing again tho 15:17:28 <nirik> I could take it if no one else... 15:18:03 <smooge> I am unavailable this weekend but I can do next week 15:18:36 <smooge> #info nirik is on call 2019-09-26 -> 2019-10-03 15:18:36 <cverna> I can take it too 15:18:40 <pingou> I'm afk the coming 10 days, but I can take it from the 10th 15:18:43 <cverna> since I do the meeting 15:19:29 <smooge> cverna, can you do next week, I can take the next 15:19:48 <cverna> I do next week 15:19:50 <smooge> #info cverna is on call 2019-10-03 -> 2019-10-10 15:20:24 <smooge> #info pingou is on call 2019-10-10->2019-10-17 15:20:35 <smooge> #info smooge is on call 2019-10-17->2019-10-?? 15:20:52 <pingou> 24 15:21:10 <nirik> final freeze starts on the 8th, FYI 15:21:27 <nirik> 22nd or 29th for release. 15:21:46 <smooge> nirik has a pilgrimage sometime around then 15:21:54 <bowlofeggs> sup bros - i'm here now (sorry) 15:21:58 <nirik> yeah, I might be away. need to plan that 15:22:04 <smooge> #topic Monitoring discussion 15:22:04 <smooge> #info https://nagios.fedoraproject.org/nagios 15:22:04 <smooge> #info Go over existing out items and fix 15:22:27 <smooge> hi bowlofeggs 15:23:00 <nirik> osbs is waiting reinstall or whatever 15:23:10 <mizdebsk> os-node04/05 alerts are caused by new monitoring for koschei at openshift 15:23:11 <cverna> yes I did not have time to do it :( 15:23:24 <nirik> 2 drives out are on their way to the DC 15:23:30 <mizdebsk> they will be fixed by me - more details in ticket 7115 15:23:39 <nirik> fas fedmsgs still needs us to add that to the openshift setup 15:23:42 <smooge> they will be fixed and we can hopefully clear those next week 15:23:52 <nirik> and thats all I think. 15:24:37 <smooge> we have 9 more emags which have arrived in the colocation which will need installation and adding. That should help on some of the osbs items 15:24:50 <smooge> #topic Monitoring discussion 15:24:50 <smooge> #info https://nagios.fedoraproject.org/nagios 15:24:50 <smooge> #info Go over existing out items and fix 15:25:04 <smooge> and that is what I pasted already geez 15:25:21 <smooge> #topic backlog discussion 15:25:21 <smooge> #info go over our backlog and discuss and determine priority 15:25:32 <smooge> nirik, your item 15:25:44 <nirik> yeah... hopefully I stay online. ;) 15:25:59 <nirik> so, I have marked some infra and some releng tickets a while back with 'backlog' 15:26:10 <nirik> these are all things that have been sitting there for a while. 15:26:28 <nirik> They may even not be something we want to do, or might be something we just keep never getting time for. 15:26:59 <nirik> I'd like to propose we start with a small list of them and try and take a few a week... in small teams 15:27:27 <nirik> that does mean we take more people's time, but I think it could be very good to help train people and also to have better ideas how to do things, etc. 15:27:45 <cverna> +1 also the oncall should act has a shield to protect others 15:27:52 <nirik> some of them may also be things that need privs, but even so, others can suggest actions, talk to the one doing the work, etc 15:28:02 <cverna> And allow them to focus on their tickets 15:28:08 * nirik nods. 15:28:25 <nirik> so, with that in mind. I have: https://board.net/p/fedora-infra-backlog that has all the backlog items I marked... 15:28:38 <nirik> (note: there's tons more, just these are oldish ones I thought we wanted to do) 15:29:00 <nirik> I'd like everyone to think that they have 5 x's... 15:29:08 * dustymabe late 15:29:19 <nirik> and place your x after the items you think are high pri or we should do first or whatever. 15:29:32 <nirik> ie, you could do xxxxx on something you really want, or spread them around. 15:29:48 <nirik> I'm happy to try and explain what the tickets are (If I know) 15:30:47 <nirik> although some actually are in progress already (since I marked them a while ago) 15:31:12 <nirik> 8158 pingou fixed. ;) 15:31:28 <cverna> \o/ 15:31:54 <pingou> xx on that one :) 15:32:24 * nirik lost cout of his votes, but I think I am done. ;) 15:33:02 * cverna is done too 15:33:06 * pingou done 15:33:47 * relrod too 15:33:54 <nirik> The move builds thing is already in progress... but... It should go in ansible and I should show everyone how it works (documentation) and so forth, so I think there's still a good chunk there. 15:34:22 <nirik> ok, so, lets see. 15:34:33 <pingou> ansible-report 15:34:38 <pingou> registry.fp.o 15:34:40 <nirik> looks like a 3 way tie. ;) 15:34:53 <pingou> and new aarch64 builders 15:35:01 <pingou> the top 4 15:35:09 * mkonecny done with x's 15:35:11 <nirik> awesome. 15:35:59 <nirik> ok, so I will mark those and we can see who is interested in helping on each one (I can do that on the list out of meeting) 15:36:15 <mkonecny> We talked about 8167 today, when we helped with cverna moving the elections 15:36:34 <nirik> yeah, I am not sure how best to do that one... 15:37:02 <nirik> oh, might have an idea. 15:37:29 <nirik> smooge: can we sync rhos-15 ? (just came out). I bet it has the newer rabbitmq we need on el7 15:37:42 <smooge> i will see 15:37:53 <smooge> we need to clean up some of the old stuff we have 15:38:13 <pingou> how many do we want to pick and work on this week? 15:38:18 <pingou> 1 or 2, more? 15:38:29 <nirik> I think we should start small and work up. 15:38:40 <mkonecny> You mean for tomorrow? 15:39:10 <pingou> so the top 3 (w/ equal number of votes) are: aarch64, ansible-report and koji archiving 15:39:14 <nirik> also I think one thing we can do from the ops side is try and schedule a time to actually work on a specific one... 15:39:25 <nirik> so interested people can join on irc 15:39:39 * mkonecny likes the idea 15:40:18 * nirik already did try and install one aarch64 box, but it didn't work and I am not yet sure why. 15:40:37 <nirik> so, lets try and focus on those 3 this coming week (until next meeting)? 15:40:43 <pingou> +1 15:40:44 <smooge> that sounds good 15:40:53 <cverna> +1 15:41:00 <nirik> I am not sure the hook will be that hard... I haven;'t looked 15:41:26 <smooge> nirik, for the emag boxes we should talk with arrfab as they had to get them up and running for C8 15:41:54 <nirik> smooge: ok, good idea. They all have centos7alt on them, but I had no way to login. ;) 15:42:11 <nirik> I did get it to pxe boot... but then it went into la la land. 15:43:03 <nirik> so I have a meeting tomorrow from 16-16:30utc. 15:43:07 <cverna> Oh we should create a ticket for fasclient and python3 too so that it is visible 15:43:21 <nirik> but I could try and be available before or after that to work on any of these. 15:43:27 <nirik> cverna: oh! yes, great idea. 15:43:34 <nirik> I can file it? 15:43:37 <cverna> And then we can focus on that next week maybe 15:43:42 <nirik> oh wait, we do have one 15:43:57 <nirik> .ticket 8210 15:43:58 <zodbot> nirik: Issue #8210: FAS clients not installable on Fedora 31+ - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8210 15:44:03 <cverna> So let's tag it with backlog 15:44:09 <nirik> ok. 15:44:57 <nirik> for everyone following along: There's also a internal jira we are playing with that has these backlog items... but it's all manual syncing between them, and we are not putting anything other than a pointer to pagure in the jira tickets 15:46:09 <cverna> Yep pagure tickets are the source of truth 15:46:35 <nirik> so, perhaps tomorrow morning I will try and work on aarch64 installs... and will try and get input from anyone who wants to know the setup or help me out. 15:46:48 <smooge> ok anything else on this. there is a hard stop in this channel at 11:55 for other meetings 15:46:52 <nirik> I'd like to wait on the koji archiving until mboddu is around... 15:47:06 <nirik> I will also mail the list about this all... thats all I had 15:47:15 * nirik is amazed, the power is still up. 15:47:43 * mboddu reading back 15:48:44 * pingou would be happy to help w/ fasclient to py3 15:50:13 <mboddu> nirik: Re: koji archiving, what do you want from me and when do you want to work on it? 15:50:21 <nirik> just updated the ticket. I have been testing on rawhide-test box... so anyone with access there could try it. 15:50:53 <nirik> mboddu: just wanted you to be around when I explain how it works and get your help to figure out how to put it in ansible, etc. :) 15:51:17 <nirik> so, perhaps we find a time next week? 15:51:26 <mboddu> nirik: Ah okay, I would love to help and sure, that works 15:51:31 <smooge> #topic Fedora CoreOS related tickets 15:51:31 <smooge> #info for 2019-09-26 meeting 15:51:31 <smooge> #info https://github.com/coreos/fedora-coreos-tracker/blob/master/Fedora-Requests.md#existing-requests-for-fedora-infra 15:51:31 <smooge> \ 15:52:00 <nirik> and there goes my power. ;) 15:52:05 <pingou> ó/ 15:52:14 <smooge> I think this is other backlog we need to look at after we go to later 15:52:36 <pingou> smooge: that links gives me a 404 15:52:41 <pingou> .headers https://github.com/coreos/fedora-coreos-tracker/blob/master/Fedora-Requests.md#existing-requests-for-fedora-infra 15:52:42 <zodbot> pingou: HTTP Error 404: Not Found 15:52:50 <smooge> ok well that makes it easy to deal with tehn 15:53:01 <pingou> \ó/ 15:53:06 <smooge> #topic Open Floor 15:53:29 <smooge> thank you all for coming. remember next week we will be working on the 3 high backlog items 15:53:29 <nirik> It was moved to hackmd 15:53:30 <pingou> it's moved to hackmd but the commit doesn't say where 15:53:45 <nirik> yeah. dunno. 15:53:58 <smooge> we have 2 minutes left 15:54:05 <smooge> anything else for this meeting? 15:54:25 * nirik has nothing off hand 15:54:39 <dustymabe> I do 15:54:45 <dustymabe> sorry 15:54:48 <dustymabe> the new link is 15:54:53 <dustymabe> https://hackmd.io/5uB7hOJKSjGUt65iLgPnbA 15:54:59 <nirik> thanks dustymabe 15:55:04 <dustymabe> sorry about that 15:55:22 <dustymabe> so #8218 15:55:33 <dustymabe> I think bowlofeggs said last time he was working on docs on how to add people to communishift 15:55:49 <nirik> yeah 15:55:51 <dustymabe> if I could get a peak at those docs maybe I could take a stab at https://pagure.io/fedora-infrastructure/issue/8218#comment-598925 15:56:02 <dustymabe> I think that basic thing should be easy to do ^^ 15:56:16 * pingou has to step out 15:56:19 <pingou> cheers everyone! 15:56:34 <nirik> dustymabe: sure, can you update the ticket with that request? 15:56:43 <dustymabe> the request for the docs? 15:56:54 <nirik> yeah. I don't think they exist yet, he is collecting info for them 15:57:18 <dustymabe> done 15:57:21 * nirik listens to the chirp of the UPS. How long will it last... 15:57:29 <dustymabe> next item is #8208 15:58:02 <dustymabe> it seems that both tflink and the couros teams want this 15:58:04 <nirik> yeah, I have had no cycles to look at that. 15:58:13 <nirik> but I think it would be good to try and get working. 15:58:30 <dustymabe> yeah I need that working so I can run this one workload in communishift 15:58:41 <cverna> Can we tag this with the backlog tag 15:58:43 <nirik> yeah... ditto for qa 15:58:47 <nirik> cverna: sure! 15:58:54 <cverna> Then it will go in our queue 15:58:56 <nirik> cverna: can you sync the pagure backlog to the jira one? 15:59:01 <cverna> Yep 15:59:09 * nirik hears fast beeps... not long now. 15:59:42 <smooge> ok time to call this meeting 15:59:45 <smooge> #endmeeting