16:02:47 <mobrien> #startmeeting Infrastructure (2020-01-14) 16:02:47 <zodbot> Meeting started Thu Jan 14 16:02:47 2021 UTC. 16:02:47 <zodbot> This meeting is logged and archived in a public location. 16:02:47 <zodbot> The chair is mobrien. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:02:47 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:02:47 <zodbot> The meeting name has been set to 'infrastructure_(2020-01-14)' 16:02:47 <mobrien> #meetingname infrastructure 16:02:47 <zodbot> The meeting name has been set to 'infrastructure' 16:02:47 <mobrien> #chair nirik pingou smooge cverna mizdebsk mkonecny abompard siddharthvipul mobrien 16:02:47 <mobrien> #info Agenda is at: https://board.net/p/fedora-infra 16:02:47 <mobrien> #info About our team: https://docs.fedoraproject.org/en-US/cpe/ 16:02:47 <zodbot> Current chairs: abompard cverna mizdebsk mkonecny mobrien nirik pingou siddharthvipul smooge 16:02:48 <mobrien> #topic aloha 16:02:54 <nirik> morning 16:03:00 <mobrien> apologies for the lateness 16:03:03 <mobrien> .hi 16:03:04 <zodbot> mobrien: mobrien 'Mark O'Brien' <markobri@redhat.com> 16:03:09 <mobrien> Hello everyone and welcome 16:03:18 <darknao> .hi 16:03:19 <mkonecny> .hi zlopez 16:03:19 <zodbot> darknao: darknao 'Francois Andrieu' <naolwen@gmail.com> 16:03:21 <zodbot> mkonecny: Sorry, but you don't exist 16:03:26 <mkonecny> .hello zlopez 16:03:28 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@psmail.xyz> 16:03:28 <pingou> ó/ 16:03:34 <dtometzki> hi 16:03:37 <darknao> \o 16:03:39 <mkonecny> I was worried for a moment :-D 16:04:05 <nirik> ha 16:04:11 <mobrien> I was too mkonecny, I was sure you existed. I've seen you before :) 16:04:32 <mobrien> #topic New folks introductions 16:04:33 <mobrien> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves 16:04:33 <mobrien> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted 16:04:45 <mobrien> Do we have any new people here today? 16:04:56 <dtometzki> yes 16:05:11 <mobrien> Feel free to introduce yourself with as much/little info as you like 16:05:27 <mobrien> experience or areas of interest etc 16:05:37 <dtometzki> I send already an E-Mail with my self introduction 16:06:16 <mobrien> Ah yes I believe I read that. Welcome dtometzki. I hope you enjoy it here 16:06:45 <mobrien> if you have anyquestions for us, don't hesitate to ask 16:06:46 <dtometzki> My real name is Damian iam 46 years old and iam a Consultant for an SAP Hoster 16:07:36 <darknao> welcome dtometzki ! 16:07:36 <dtometzki> Iam responsible for the linux infrastructure and for the cloud infrastructure on Aws and Azure 16:08:15 <nirik> welcome! 16:08:41 <dtometzki> And I would like help in the Infrastructure or linux kernel package 16:09:24 <dtometzki> I see there are a lot of groups. 16:10:09 <nirik> Yeah, it can be a bit daunting... 16:10:14 <dtometzki> My question is how can i start to help you ? 16:10:40 <nirik> but do hang around, ask questions and when you see something interesting to you, chime in and offer to help out... 16:10:54 <dtometzki> I dont know the groups in detal but sysadmin or sysadmin-kernel was intresting 16:11:01 <dtometzki> detail 16:11:11 <mobrien> There is a tracker here of issues in the fedora infrastructure https://pagure.io/fedora-infrastructure/issues if there is anything there that you think you could help with or offer advice on feel free to comment 16:11:32 <dtometzki> ok 16:12:44 <mobrien> if you see anything people are working on that seems of interest feel free to ask to join in or help. We are a welcoming group :) 16:12:51 <nirik> we don't usually do too much directly with the kernel... for that #fedora-kernel would be the place to chime in. 16:13:52 <dtometzki> No Ithink it is a good starting point 16:14:05 <dtometzki> :-) 16:14:06 <mkonecny> We usually hang it #fedora-apps, #fedora-admin or #fedora-noc 16:14:17 <mkonecny> s/it/in 16:15:34 <mkonecny> So feel free to ping us if you see that we are working on anything interesting 16:15:34 <dtometzki> I think most of the peoples are redhat guys ? 16:16:42 <nirik> it's a mix. ;) some Red Hat employees, some community members, everyone welcome 16:17:43 <mobrien> Ok I think we can move to the next topic 16:18:10 <mobrien> ### Determine who the next chair is 16:18:10 <mobrien> #info magic eight ball says: 16:18:16 <mobrien> #info 2021-01-14 - mobrien 16:18:17 <mobrien> #info 2021-01-21 - mkonecny 16:18:44 <mkonecny> I have the next one :-) 16:19:02 <mobrien> Do we have a volunteer for the week after? 16:19:14 <mobrien> 2021-01-28? 16:20:46 <dtometzki> What is the task of an volunteer ? 16:21:40 <mobrien> Essentially you do what I am doing now 16:21:40 <mkonecny> You will do what mobrien does, everything is written in agenda https://board.net/p/fedora-infra 16:22:38 <mobrien> If you would like to volunteer I could shadow you but no pressure if not 16:23:13 <nirik> yeah, I could also take it if preferred. 16:23:43 <smooge> is here now 16:24:00 <dtometzki> yes perhaps i will read it and next time i can do it. If that is ok ? 16:25:08 <mobrien> no problem dtometzki so will I put in nirik for the 28th and you can decide next week if you'd like to do the one after? 16:25:32 <dtometzki> yes 16:25:41 <mobrien> +1 16:25:56 <mobrien> #info chair 2021-01-28 - nirik 16:26:13 <mobrien> Ok lets move to the announcements 16:26:23 <mobrien> #topic announcements and information 16:26:23 <mobrien> #info CPE Infra&Releng EU-hours team has a Monday through Friday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting 16:26:23 <mobrien> #info CPE Infra&Releng NA-hours team has a Monday through Friday 30 minute meeting going through tickets at 1800 UTC in #fedora-admin 16:26:23 <mobrien> #info Datacenter move is over, but some items still need to be done: see https://fedoraproject.org/wiki/Infrastructure/2020-post-datacenter-move-known-issues 16:26:50 <mobrien> Does anyone have anything else to announce? 16:26:50 <nirik> #info f34 mass rebuild will start next week (2021-01-20) 16:27:54 <mobrien> should we expect and effect to services with that? 16:28:01 <mobrien> s/and/any 16:28:19 <nirik> nope, just busyness/high loads. 16:28:39 <nirik> and we want to make sure not to make any changes that mess it up... 16:28:45 <mobrien> I know I've been here for a few of those but I have a memory of a sieve 16:29:31 <mobrien> ok so if no more announcements we'll move on 16:29:54 <mobrien> #topic Oncall 16:29:54 <mobrien> #info https://fedoraproject.org/wiki/Infrastructure/Oncall 16:29:54 <mobrien> #info mobrien is oncall for 2021-01-07 to 2021-01-14 16:29:54 <mobrien> #info siddharthvipul1 is oncall for 2021-01-14 to 2021-01-21 16:29:54 <mobrien> #info ??? is oncall for 2021-01-21 to 2021-01-28 16:30:18 <mobrien> Anyone like to take on call for 2021-01-21 to 2021-01-28? 16:30:39 <pingou> I can 16:30:46 <pingou> it's been a while since I was :) 16:30:55 <mobrien> Thanks pingou 16:30:59 <nirik> thanks pingou 16:31:30 <mobrien> #info pingou is oncall for 2021-01-21 to 2021-01-28 16:32:04 <mobrien> I don't think siddharthvipul is here to take oncall so I'll hold onto it and he can take it tomorrow 16:32:12 <pingou> I'll take the meeting chair on the 28th as well 16:32:35 <mobrien> that ok with you nirik? 16:32:48 <pingou> oh sorry, I didn't realize it was already assigned 16:32:49 <nirik> sure, wfm 16:33:03 <mobrien> well you have it now pingou :) 16:33:12 <pingou> sorry nirik less work for you :( 16:33:30 <mobrien> #info Summary of last week: (from current oncall ) 16:33:53 <nirik> sure, thats fine with me too. 16:34:10 <mobrien> There was one ping about email not reaching someone but smooge looked into it and it appears to be working correctly 16:34:21 <mobrien> and that was it 16:35:30 <mobrien> movin along ... 16:35:38 <mobrien> #topic Monitoring discussion [nirik] 16:35:38 <mobrien> #info https://nagios.fedoraproject.org/nagios 16:35:38 <mobrien> #info Go over existing out items and fix 16:35:59 <nirik> so, I think we are pretty unchanged since last week. 16:36:08 <nirik> an outage caused by a down aarch64 host. 16:36:22 <nirik> and some small issues with stg or other services 16:36:38 <nirik> badges seems unhappy. 16:36:46 <nirik> meant to ask misc to look into that 16:36:52 <pingou> we had a small pagure.io outage on Friday which surprised me 16:37:06 <pingou> as the change that trigger the outage was from Jan 4th 16:37:15 <pingou> nirik: was there a master playbook run at the end of last week? 16:37:59 <nirik> last week, I don't think so... I can check tho. 16:38:13 <pingou> otherwise, I really don't know how we triggered it at ~9am UTC last Friday 16:38:54 <mkonecny> I have some issue with staging right now. I deployed a new version of release-monitoring and the staging is still serving the previous one, even if the stg.os.fp.o says that the new one is running 16:41:06 <nirik> mkonecny: no idea. caching? 16:41:53 <mkonecny> I tried incognito mode in browser and even a different browser, so I don't think this is something on my side 16:42:32 <nirik> pingou: I only see one run with changes on the 8th... and that was you running it. ;) So, likely fixing it? 16:42:35 <mkonecny> It looks like it's cached somewhere else, because the requests are being served according to logs in openshift 16:43:02 <pingou> nirik: yup (and this morning I fixed stg for the same issue) 16:43:42 <pingou> mkonecny: check the build logs 16:43:50 <pingou> they should give the commit built 16:43:57 <nirik> but... back to nagios. ;) Nothing new we can move on 16:44:32 <mkonecny> pingou: The commit is correct, this is why I'm confused 16:44:49 <pingou> weird 16:45:00 <mkonecny> I will try a rebuilt 16:45:35 <mobrien> one more thing on the nagios discussion. We have had alerts a few times this week for disk space in koji. It is being caused by the access log growing to over 10G in under a day, something to note if it pops up again 16:46:32 <mobrien> We have no learning topic set for this week so I will skip it unless someone wants to volunteer one 16:47:11 <nirik> mobrien: oh yeah, I think we should just add ~100GB disk to those vm's. 16:48:31 <mobrien> sounds like a good solution nirik 16:48:52 * nirik can do that and share the log of it. we have a SOP tho, so it's pretty easy. 16:49:26 <mobrien> I can do it tomorrow morning if you want? 16:50:21 <nirik> sure, if you would like. It shouldn't cause an outage as there is a pair of hub nodes. 16:51:02 <nirik> https://docs.pagure.org/infra-docs/sysadmin-guide/sops/guestdisk.html 16:51:05 <mobrien> Ya. It makes sense for me to do it early eu time anyway, there will be less people online 16:51:40 <nirik> cool. I'd say make em 100GB... don't forget to up that in ansible also in case we ever redeploy them 16:51:48 <mobrien> ok so time for the final topic 16:51:59 <mobrien> #topic Open Floor 16:52:23 <mobrien> anyone have anything they would like to share/discuss/sing 16:53:06 <nirik> not off hand here... 16:53:29 <smooge> I am working on getting hardware fixed 16:53:34 <smooge> there are multiple problems 16:53:50 <smooge> I am also dealing with a house plumbing issue so I am a bit distracted 16:53:52 <nirik> as always. ;( 16:54:05 <smooge> it gets better and better problems 16:54:14 <nirik> and our koji netapp volume is being bloated again... :( but we will track it down 16:54:40 <darknao> speaking of netapp, i do have a question about our openshift 16:54:47 <nirik> there's more snapshot space taken up on it than content, which always seemed wacky to me. 16:55:42 <darknao> let's say I require a pv there for some app, how does it work ? do I need to fill a rfr or is there any requirements ? 16:55:50 <nirik> oh, and we still need to get out staging setup working with ipa/ssh/sudo... nills sent a PR and it gets part way, but doesn't work. ;( 16:56:13 <nirik> darknao: yes, it has to be manually created on the netapp and then defined in openshift. 16:56:16 <zodbot> geppetto: Error: Can't start another meeting, one is in progress. 16:56:17 <geppetto> #meetingname fpc 16:56:17 <geppetto> #topic Roll Call 16:56:17 <zodbot> The meeting name has been set to 'fpc' 16:56:30 <geppetto> Damn it … sorry 16:56:38 <geppetto> Wrong paste 16:56:44 <geppetto> #undo 16:56:52 <nirik> geppetto: no worries, we are almost done here. 16:56:57 <mobrien> no worries geppetto we'll be finished in a min 16:57:14 * limburgher here 16:57:16 * geppetto nods … no rush 16:57:27 <geppetto> Meant to post the 5 min. warning 16:57:40 <mobrien> I guess we are finished here if nobody has anything else? 16:58:15 <mobrien> #endmeeting