16:02:47 <mobrien> #startmeeting Infrastructure (2020-01-14)
16:02:47 <zodbot> Meeting started Thu Jan 14 16:02:47 2021 UTC.
16:02:47 <zodbot> This meeting is logged and archived in a public location.
16:02:47 <zodbot> The chair is mobrien. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:02:47 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:02:47 <zodbot> The meeting name has been set to 'infrastructure_(2020-01-14)'
16:02:47 <mobrien> #meetingname infrastructure
16:02:47 <zodbot> The meeting name has been set to 'infrastructure'
16:02:47 <mobrien> #chair nirik pingou smooge cverna mizdebsk mkonecny abompard siddharthvipul mobrien
16:02:47 <mobrien> #info Agenda is at: https://board.net/p/fedora-infra
16:02:47 <mobrien> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:02:47 <zodbot> Current chairs: abompard cverna mizdebsk mkonecny mobrien nirik pingou siddharthvipul smooge
16:02:48 <mobrien> #topic aloha
16:02:54 <nirik> morning
16:03:00 <mobrien> apologies for the lateness
16:03:03 <mobrien> .hi
16:03:04 <zodbot> mobrien: mobrien 'Mark O'Brien' <markobri@redhat.com>
16:03:09 <mobrien> Hello everyone and welcome
16:03:18 <darknao> .hi
16:03:19 <mkonecny> .hi zlopez
16:03:19 <zodbot> darknao: darknao 'Francois Andrieu' <naolwen@gmail.com>
16:03:21 <zodbot> mkonecny: Sorry, but you don't exist
16:03:26 <mkonecny> .hello zlopez
16:03:28 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@psmail.xyz>
16:03:28 <pingou> ó/
16:03:34 <dtometzki> hi
16:03:37 <darknao> \o
16:03:39 <mkonecny> I was worried for a moment :-D
16:04:05 <nirik> ha
16:04:11 <mobrien> I was too mkonecny, I was sure you existed. I've seen you before :)
16:04:32 <mobrien> #topic New folks introductions
16:04:33 <mobrien> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:04:33 <mobrien> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:04:45 <mobrien> Do we have any new people here today?
16:04:56 <dtometzki> yes
16:05:11 <mobrien> Feel free to introduce yourself with as much/little info as you like
16:05:27 <mobrien> experience or areas of interest etc
16:05:37 <dtometzki> I send already an E-Mail with my self introduction
16:06:16 <mobrien> Ah yes I believe I read that. Welcome dtometzki. I hope you enjoy it here
16:06:45 <mobrien> if you have anyquestions for us, don't hesitate to ask
16:06:46 <dtometzki> My real name is Damian iam 46 years old and iam a Consultant for an SAP Hoster
16:07:36 <darknao> welcome dtometzki !
16:07:36 <dtometzki> Iam responsible for the linux infrastructure and for the cloud infrastructure on Aws and Azure
16:08:15 <nirik> welcome!
16:08:41 <dtometzki> And I would like help in the Infrastructure or linux kernel package
16:09:24 <dtometzki> I see there are a lot of groups.
16:10:09 <nirik> Yeah, it can be a bit daunting...
16:10:14 <dtometzki> My question is how can i start to help you ?
16:10:40 <nirik> but do hang around, ask questions and when you see something interesting to you, chime in and offer to help out...
16:10:54 <dtometzki> I dont know the groups in detal but sysadmin or sysadmin-kernel was intresting
16:11:01 <dtometzki> detail
16:11:11 <mobrien> There is a tracker here of issues in the fedora infrastructure https://pagure.io/fedora-infrastructure/issues if there is anything there that you think you could help with or offer advice on feel free to comment
16:11:32 <dtometzki> ok
16:12:44 <mobrien> if you see anything people are working on that seems of interest feel free to ask to join in or help. We are a welcoming group :)
16:12:51 <nirik> we don't usually do too much directly with the kernel... for that #fedora-kernel would be the place to chime in.
16:13:52 <dtometzki> No Ithink it is a good starting point
16:14:05 <dtometzki> :-)
16:14:06 <mkonecny> We usually hang it #fedora-apps, #fedora-admin or #fedora-noc
16:14:17 <mkonecny> s/it/in
16:15:34 <mkonecny> So feel free to ping us if you see that we are working on anything interesting
16:15:34 <dtometzki> I think most of the peoples are redhat guys ?
16:16:42 <nirik> it's a mix. ;) some Red Hat employees, some community members, everyone welcome
16:17:43 <mobrien> Ok I think we can move to the next topic
16:18:10 <mobrien> ### Determine who the next chair is
16:18:10 <mobrien> #info magic eight ball says:
16:18:16 <mobrien> #info 2021-01-14 - mobrien
16:18:17 <mobrien> #info 2021-01-21 - mkonecny
16:18:44 <mkonecny> I have the next one :-)
16:19:02 <mobrien> Do we have a volunteer for the week after?
16:19:14 <mobrien> 2021-01-28?
16:20:46 <dtometzki> What is the task of an volunteer ?
16:21:40 <mobrien> Essentially you do what I am doing now
16:21:40 <mkonecny> You will do what mobrien does, everything is written in agenda https://board.net/p/fedora-infra
16:22:38 <mobrien> If you would like to volunteer I could shadow you but no pressure if not
16:23:13 <nirik> yeah, I could also take it if preferred.
16:23:43 <smooge> is here now
16:24:00 <dtometzki> yes perhaps i will read it and next time i can do it. If that is ok ?
16:25:08 <mobrien> no problem dtometzki so will I put in nirik for the 28th and you can decide next week if you'd like to do the one after?
16:25:32 <dtometzki> yes
16:25:41 <mobrien> +1
16:25:56 <mobrien> #info chair 2021-01-28 - nirik
16:26:13 <mobrien> Ok lets move to the announcements
16:26:23 <mobrien> #topic announcements and information
16:26:23 <mobrien> #info CPE Infra&Releng EU-hours team has a Monday through Friday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:26:23 <mobrien> #info CPE Infra&Releng NA-hours team has a Monday through Friday 30 minute meeting going through tickets at 1800 UTC in #fedora-admin
16:26:23 <mobrien> #info Datacenter move is over, but some items still need to be done: see https://fedoraproject.org/wiki/Infrastructure/2020-post-datacenter-move-known-issues
16:26:50 <mobrien> Does anyone have anything else to announce?
16:26:50 <nirik> #info f34 mass rebuild will start next week (2021-01-20)
16:27:54 <mobrien> should we expect and effect to services with that?
16:28:01 <mobrien> s/and/any
16:28:19 <nirik> nope, just busyness/high loads.
16:28:39 <nirik> and we want to make sure not to make any changes that mess it up...
16:28:45 <mobrien> I know I've been here for a few of those but I have a memory of a sieve
16:29:31 <mobrien> ok so if no more announcements we'll move on
16:29:54 <mobrien> #topic Oncall
16:29:54 <mobrien> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:29:54 <mobrien> #info mobrien is oncall for 2021-01-07 to 2021-01-14
16:29:54 <mobrien> #info siddharthvipul1 is oncall for 2021-01-14 to 2021-01-21
16:29:54 <mobrien> #info ??? is oncall for 2021-01-21 to 2021-01-28
16:30:18 <mobrien> Anyone like to take on call for  2021-01-21 to 2021-01-28?
16:30:39 <pingou> I can
16:30:46 <pingou> it's been a while since I was :)
16:30:55 <mobrien> Thanks pingou
16:30:59 <nirik> thanks pingou
16:31:30 <mobrien> #info pingou is oncall for 2021-01-21 to 2021-01-28
16:32:04 <mobrien> I don't think siddharthvipul is here to take oncall so I'll hold onto it and he can take it tomorrow
16:32:12 <pingou> I'll take the meeting chair on the 28th as well
16:32:35 <mobrien> that ok with you nirik?
16:32:48 <pingou> oh sorry, I didn't realize it was already assigned
16:32:49 <nirik> sure, wfm
16:33:03 <mobrien> well you have it now pingou :)
16:33:12 <pingou> sorry nirik less work for you :(
16:33:30 <mobrien> #info Summary of last week: (from current oncall )
16:33:53 <nirik> sure, thats fine with me too.
16:34:10 <mobrien> There was one ping about email not reaching someone but smooge looked into it and it appears to be working correctly
16:34:21 <mobrien> and that was it
16:35:30 <mobrien> movin along ...
16:35:38 <mobrien> #topic Monitoring discussion [nirik]
16:35:38 <mobrien> #info https://nagios.fedoraproject.org/nagios
16:35:38 <mobrien> #info Go over existing out items and fix
16:35:59 <nirik> so, I think we are pretty unchanged since last week.
16:36:08 <nirik> an outage caused by a down aarch64 host.
16:36:22 <nirik> and some small issues with stg or other services
16:36:38 <nirik> badges seems unhappy.
16:36:46 <nirik> meant to ask misc to look into that
16:36:52 <pingou> we had a small pagure.io outage on Friday which surprised me
16:37:06 <pingou> as the change that trigger the outage was from Jan 4th
16:37:15 <pingou> nirik: was there a master playbook run at the end of last week?
16:37:59 <nirik> last week, I don't think so... I can check tho.
16:38:13 <pingou> otherwise, I really don't know how we triggered it at ~9am UTC last Friday
16:38:54 <mkonecny> I have some issue with staging right now. I deployed a new version of release-monitoring and the staging is still serving the previous one, even if the stg.os.fp.o says that the new one is running
16:41:06 <nirik> mkonecny: no idea. caching?
16:41:53 <mkonecny> I tried incognito mode in browser and even a different browser, so I don't think this is something on my side
16:42:32 <nirik> pingou: I only see one run with changes on the 8th... and that was you running it. ;) So, likely fixing it?
16:42:35 <mkonecny> It looks like it's cached somewhere else, because the requests are being served according to logs in openshift
16:43:02 <pingou> nirik: yup (and this morning I fixed stg for the same issue)
16:43:42 <pingou> mkonecny: check the build logs
16:43:50 <pingou> they should give the commit built
16:43:57 <nirik> but... back to nagios. ;) Nothing new we can move on
16:44:32 <mkonecny> pingou: The commit is correct, this is why I'm confused
16:44:49 <pingou> weird
16:45:00 <mkonecny> I will try a rebuilt
16:45:35 <mobrien> one more thing on the nagios discussion. We have had alerts a few times this week for disk space in koji. It is being caused by the access log growing to over 10G in under a day, something to note if it pops up again
16:46:32 <mobrien> We have no learning topic set for this week so I will skip it unless someone wants to volunteer one
16:47:11 <nirik> mobrien: oh yeah, I think we should just add ~100GB disk to those vm's.
16:48:31 <mobrien> sounds like a good solution nirik
16:48:52 * nirik can do that and share the log of it. we have a SOP tho, so it's pretty easy.
16:49:26 <mobrien> I can do it tomorrow morning if you want?
16:50:21 <nirik> sure, if you would like. It shouldn't cause an outage as there is a pair of hub nodes.
16:51:02 <nirik> https://docs.pagure.org/infra-docs/sysadmin-guide/sops/guestdisk.html
16:51:05 <mobrien> Ya. It makes sense for me to do it early eu time anyway, there will be less people online
16:51:40 <nirik> cool. I'd say make em 100GB... don't forget to up that in ansible also in case we ever redeploy them
16:51:48 <mobrien> ok so time for the final topic
16:51:59 <mobrien> #topic Open Floor
16:52:23 <mobrien> anyone have anything they would like to share/discuss/sing
16:53:06 <nirik> not off hand here...
16:53:29 <smooge> I am working on getting hardware fixed
16:53:34 <smooge> there are multiple problems
16:53:50 <smooge> I am also dealing with a house plumbing issue so I am a bit distracted
16:53:52 <nirik> as always. ;(
16:54:05 <smooge> it gets better and better problems
16:54:14 <nirik> and our koji netapp volume is being bloated again... :( but we will track it down
16:54:40 <darknao> speaking of netapp, i do have a question about our openshift
16:54:47 <nirik> there's more snapshot space taken up on it than content, which always seemed wacky to me.
16:55:42 <darknao> let's say I require a pv there for some app, how does it work ? do I need to fill a rfr or is there any requirements ?
16:55:50 <nirik> oh, and we still need to get out staging setup working with ipa/ssh/sudo... nills sent a PR and it gets part way, but doesn't work. ;(
16:56:13 <nirik> darknao: yes, it has to be manually created on the netapp and then defined in openshift.
16:56:16 <zodbot> geppetto: Error: Can't start another meeting, one is in progress.
16:56:17 <geppetto> #meetingname fpc
16:56:17 <geppetto> #topic Roll Call
16:56:17 <zodbot> The meeting name has been set to 'fpc'
16:56:30 <geppetto> Damn it … sorry
16:56:38 <geppetto> Wrong paste
16:56:44 <geppetto> #undo
16:56:52 <nirik> geppetto: no worries, we are almost done here.
16:56:57 <mobrien> no worries geppetto we'll be finished in a min
16:57:14 * limburgher here
16:57:16 * geppetto nods … no rush
16:57:27 <geppetto> Meant to post the 5 min. warning
16:57:40 <mobrien> I guess we are finished here if nobody has anything else?
16:58:15 <mobrien> #endmeeting