16:00:12 <nirik> #startmeeting Infrastructure (2022-12-01)
16:00:12 <zodbot> Meeting started Thu Dec  1 16:00:12 2022 UTC.
16:00:12 <zodbot> This meeting is logged and archived in a public location.
16:00:12 <zodbot> The chair is nirik. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:00:12 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:12 <zodbot> The meeting name has been set to 'infrastructure_(2022-12-01)'
16:00:12 <nirik> #meetingname infrastructure
16:00:12 <zodbot> The meeting name has been set to 'infrastructure'
16:00:12 <nirik> #chair nirik zlopez nb bodanel dtometzki jnsamyak
16:00:12 <nirik> #info Agenda is at: https://board.net/p/fedora-infra
16:00:12 <zodbot> Current chairs: bodanel dtometzki jnsamyak nb nirik zlopez
16:00:12 <nirik> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:00:14 <nirik> #info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
16:00:16 <nirik> #topic greetings!
16:00:31 <smooge> morning
16:00:47 <nirik> morning
16:00:59 <copperi[m]> hi
16:01:49 <saibug[m]> morning 👋
16:02:14 <phsmoura> .hi
16:02:15 <zodbot> phsmoura: phsmoura 'Pedro Moura' <pmoura@redhat.com>
16:03:27 <darknao> .hi
16:03:28 <zodbot> darknao: darknao 'Francois Andrieu' <darknao@drkn.ninja>
16:03:36 * nirik watches the sun shine down on... snow falling. Weird weather. ;)
16:05:03 <dtometzki> hi
16:05:07 <dtometzki> .hi
16:05:08 <zodbot> dtometzki: dtometzki 'Damian Tometzki' <linux@tometzki.de>
16:05:22 <nirik> ok, lets go and move on...
16:05:34 <smooge> sounds good
16:05:36 <nirik> #topic New folks introductions
16:05:36 <nirik> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:05:36 <nirik> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:05:40 <Penguinpee> .hello gui1ty
16:05:41 <nirik> any new folks today?
16:05:41 <zodbot> Penguinpee: gui1ty 'Sandro .' <gui1ty@penguinpee.nl>
16:06:52 <nirik> seems not. ;)
16:06:55 <nirik> #info chair 2022-12-01 - nirik
16:06:55 <nirik> #info chair 2022-12-08 - ??
16:06:55 <nirik> #info chair 2022-12-15 - ??
16:06:55 <nirik> #info chair 2022-12-22 - no meeting / holidays
16:06:55 <nirik> #info chair 2022-12-29 - no meeting / holidays
16:07:08 <nirik> anyone like to take the chair for the 8th or 15th?
16:07:22 <dtometzki> the 8th i can take
16:07:34 <Penguinpee> I can take the 15th.
16:08:15 <nirik> awesome
16:08:30 <nirik> thanks!
16:08:32 <nirik> #topic announcements and information
16:08:32 <nirik> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:08:32 <nirik> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:08:32 <nirik> #info If your team wants support from the Fedora Program Management Team, file an isssue: https://pagure.io/fedora-pgm/pgm_team/issues?template=support_request
16:08:34 <nirik> #info oncall should also handling #fedora-releng pings if possible
16:08:39 <nirik> any other announcements from anyone?
16:09:14 <nirik> #info nirik out for holidays from 2022-12-12 (but I will still be lurking for any emergencies)
16:09:41 <Penguinpee> Sorry, just saw I have another meeting on the 15th add exactly 16:00 UTC. So, I cannot take the 15th.
16:09:53 <nirik> ok, no biggie.
16:11:28 <zlopez> .hello zlopez
16:11:29 <zodbot> zlopez: zlopez 'Michal Konecny' <michal.konecny@pacse.eu>
16:11:38 <nirik> #topic Oncall
16:11:39 <nirik> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:11:39 <nirik> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
16:11:39 <nirik> ## .oncalltakeeu .oncalltakeus
16:11:39 <nirik> #info smooge is on call from 2022-11-25 to 2022-12-01
16:11:39 <nirik> #info ? is on call from 2022-12-02 to 2022-12-08
16:11:41 <nirik> #info ? is on call from 2022-12-08 to 2022-12-15
16:11:52 <nirik> anyone want to take oncall for the next week or the week after?
16:12:00 <phsmoura> I can take from 2022-12-02 to 2022-12-08
16:12:15 <nirik> thanks phsmoura
16:12:46 <smooge> is it here where I say what I saw oncall?
16:12:55 <nirik> #info Summary of last week: (from current oncall )
16:12:58 <nirik> yeah. :)
16:13:15 <smooge> so basically no one is using .oncall but using direct pings to nirik
16:13:41 <smooge> I took as many of those as I could on the off hours to tell them either to open tickets or see if I could fix the problem
16:14:06 <dtometzki> i think the problem is most of the users where do oncall cant really help
16:14:28 <nirik> well, it helps by taking the inturrupt...
16:14:31 <smooge> there were 6 issues ranging from koji problems, pkgs01 git/ssh problems, and a couple of spam like isues
16:14:31 <dtometzki> and nirik can really help
16:15:31 <phsmoura> its good to use oncall so new contributors that volunteer can also help nirik with the demand
16:15:44 <nirik> I can tell people to file tickets or oncall... but in some cases thats people getting back to me when I asked them something... but yeah.
16:16:09 <smooge> anyway that was it
16:16:24 <phsmoura> ## .oncalltakeeu .oncalltakeus
16:16:59 <dtometzki> phsmoura, without ##
16:17:08 <smooge> #info people directly pinging nirik versus using oncall and issues were koji or git related
16:17:08 <phsmoura> .oncalltakeeu .oncalltakeus
16:17:08 <zodbot> phsmoura: Error: You don't have the alias.add capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified.
16:17:19 <smooge> just one of them
16:17:41 <smooge> oncalltakeus needs to be different name since you aren't in the us
16:17:52 <nirik> NA vs EU?
16:18:29 <smooge> yeah
16:18:35 <phsmoura> sorry, what should be the comando to use?
16:18:46 <smooge> you shoudl try .oncalltakeus
16:18:56 <smooge> until it gets named differently :)
16:18:57 <phsmoura> .oncalltakeus
16:18:57 <zodbot> phsmoura: Error: You don't have the alias.add capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified.
16:19:17 <phsmoura> ok, need to work around this error
16:19:33 <smooge> oh that means zodbot is having issues with you. zod needs a timeout
16:19:43 <nirik> you need perms.
16:19:52 <nirik> let me see if I can recall how to do that. :)
16:21:36 <nirik> hum, we can do it outside of meeting.
16:21:57 <nirik> I think you need to make an account... /msg zodbot register
16:22:05 <nirik> and then I can add privs
16:22:17 <nirik> shall we move on for now?
16:22:46 <phsmoura> +1
16:22:57 <nirik> #topic Monitoring discussion [nirik]
16:22:57 <nirik> #info https://nagios.fedoraproject.org/nagios
16:22:57 <nirik> #info Go over existing out items and fix
16:23:11 <nirik> so, we did a mass update/reboot yesterday... and I upgraded a bunch of things.
16:23:35 <nirik> one thing that broke shows up on nagios...
16:23:49 <nirik> all the proxies show:
16:23:51 <nirik> proxy01.iad2.fedoraproject.org
16:23:51 <nirik> 
16:23:51 <nirik> 
16:23:51 <nirik> Check fedmsg-gateway consumers backlog
16:23:51 <nirik> 
16:23:52 <nirik> This service has 1 comment associated with it	This service problem has been acknowledged
16:23:54 <nirik> UNKNOWN 	12-01-2022 16:19:22 	0d 15h 43m 59s 	3/3 	UNKNOWN: fedmsg consumer GatewayConsumer not found
16:23:57 <nirik> 
16:23:59 <nirik> Check fedmsg-gateway consumers exceptions
16:24:01 <nirik> 
16:24:03 <nirik> This service has 1 comment associated with it	This service problem has been acknowledged
16:24:05 <nirik> UNKNOWN 	12-01-2022 16:19:22 	0d 15h 43m 59s 	3/3 	UNKNOWN: fedmsg consumers GatewayConsumer not found
16:24:08 <nirik> oops. that was not what I meant to paste. ;(
16:24:14 <nirik> the problem seems to be fedmsg-gateway-3 is broken. ;(
16:24:31 <nirik> I can file a ticket on it, but I have no idea how to fix it. Help very welcome.
16:25:55 <nirik> so, that problem amounts to about 75 alerts (since it's about 3 per proxy)
16:26:26 <nirik> otherwise everything is pretty back to normal.
16:26:32 <Penguinpee> I saw that error on badges-backend01 as well. But it got resolved after the updates/reboots.
16:27:00 <nirik> Penguinpee: oh yeah? huh...
16:27:31 <Penguinpee> Not sure I've got a clue, but I could at least take a look.
16:28:08 <nirik> https://paste.centos.org/view/ae313ba2
16:28:23 <Penguinpee> https://nagios.fedoraproject.org/nagios/cgi-bin//history.cgi?host=badges-backend01.iad2.fedoraproject.org&type=0&statetype=0&service=Check+fedmsg-hub+consumers+backlog&archive=1
16:28:32 <Penguinpee> That's what I saw yesterday
16:28:36 <nirik> the problem seems to be some kind of moksha / zmq problem
16:29:16 <Penguinpee> May be a service needs a kick. I'll take a look. Could you make a ticket and assign it to me?
16:29:52 <nirik> restarting it causes the same traceback. ;) It needs more than that.
16:29:58 <nirik> yeah, I plan to file a ticket...
16:30:32 <nirik> so, we don't have a learning topic today, so I guess backlog refinement? or shall we brainstorm learning topics or just close early?
16:30:35 <Penguinpee> A dependent service not being started maybe? It looks similar to what I observed on badges-backend01 last night.
16:30:58 <nirik> badges-backend01 is rhel7 I think tho, where proxies are fedora 37
16:31:48 <Penguinpee> and moksha is Python 2, right?
16:32:19 <nirik> I think it's python3 in this case...
16:33:41 <nirik> I think this breakage is also causing no external fedmsgs to come in...
16:34:15 <Penguinpee> "Cannot find qpid python module."
16:34:18 <nirik> anyhow, what would folks like to do; a) end early and get 30min back, b) brainstorm on learning sessions c) backlog refinement d) your choice here.
16:35:26 <Penguinpee> I'm for b). Two more meetings this year, should fit at least one more learning topic.
16:35:46 <phsmoura> while we dont have more answers let me check if it works now
16:35:49 <phsmoura> .oncalltakeus
16:35:49 <zodbot> phsmoura: Error: You don't have the alias.add capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified.
16:35:55 <phsmoura> :/
16:36:09 <phsmoura> a) end early and get 30min back
16:36:27 <nirik> phsmoura: try again now?
16:36:40 <phsmoura> .oncalltakeus
16:36:40 <zodbot> phsmoura: Kneel before zod!
16:36:49 <phsmoura> thanks nirik
16:37:07 <smooge> i am sadly for a due to needing to go to a live meeting
16:37:18 <nirik> ok, lets open floor it and close.
16:37:22 <nirik> #topic Open Floor
16:37:33 <nirik> any questions, comments, fav cookie recipies?
16:38:39 <Penguinpee> There will be a badges meeting on the 15th. If anyone is interested I can post the link.
16:38:55 <nirik> sure. Post away
16:39:29 <Penguinpee> https://discussion.fedoraproject.org/t/getting-fedora-badges-back-in-shape/42942/29
16:39:54 <dtometzki> thanks Penguinpee
16:40:19 <Penguinpee> Anyone interested can drop jflory7 a line and will get added to the invitation.
16:41:09 <nirik> thanks
16:41:15 <nirik> ok, thanks for coming everyone
16:41:21 <nirik> #endmeeting