16:00:13 <phsmoura> #startmeeting Infrastructure (2023-01-26)
16:00:13 <zodbot> Meeting started Thu Jan 26 16:00:13 2023 UTC.
16:00:13 <zodbot> This meeting is logged and archived in a public location.
16:00:13 <zodbot> The chair is phsmoura. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:00:13 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:13 <zodbot> The meeting name has been set to 'infrastructure_(2023-01-26)'
16:00:13 <phsmoura> #meetingname infrastructure
16:00:13 <zodbot> The meeting name has been set to 'infrastructure'
16:00:13 <phsmoura> #chair nirik zlopez nb bodanel dtometzki jnsamyak
16:00:13 <zodbot> Current chairs: bodanel dtometzki jnsamyak nb nirik phsmoura zlopez
16:00:13 <phsmoura> #info Agenda is at: https://board.net/p/fedora-infra
16:00:13 <phsmoura> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:00:14 <phsmoura> #info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
16:00:16 <phsmoura> #topic greetings!
16:00:21 <phsmoura> .hi
16:00:22 <zodbot> phsmoura: phsmoura 'Pedro Moura' <pmoura@redhat.com>
16:00:40 <dtometzki> .hi
16:00:41 <zodbot> dtometzki: dtometzki 'Damian Tometzki' <damian@riscv.tometzki.de>
16:00:51 <nirik> morning all.
16:01:16 <mkonecny> .hello zlopez
16:01:17 <zodbot> mkonecny: zlopez 'Michal Konecny' <michal.konecny@pacse.eu>
16:01:30 <petebuffon> .hello petebuffon
16:01:31 <zodbot> petebuffon: petebuffon 'Peter Buffon' <pabuffon@gmail.com>
16:01:35 <jednorozec> .hello humaton
16:01:36 <zodbot> jednorozec: humaton 'Tomáš Hrčka' <thrcka@redhat.com>
16:02:34 <phsmoura> #topic New folks introductions
16:02:35 <phsmoura> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:02:35 <phsmoura> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:03:03 <phsmoura> are there new people here? feel free to introduce yourself :)
16:04:50 <phsmoura> #topic Next chair
16:04:51 <phsmoura> #info magic eight ball says:
16:04:51 <phsmoura> #info chair 2023-02-02 - mkonecny
16:04:51 <phsmoura> #info chair 2023-02-09 - ?
16:04:51 <phsmoura> #info chair 2023-02-16- ?
16:05:28 <phsmoura> volunteers to be the host on feb 9th and 16th?
16:05:57 <darknao> .hi
16:05:58 <zodbot> darknao: darknao 'Francois Andrieu' <darknao@drkn.ninja>
16:06:01 <dherrera> .hi
16:06:02 <zodbot> dherrera: dherrera 'Diego Herrera' <dherrera@redhat.com>
16:08:00 <phsmoura> hi darknao and dherrera. just saying hi or you have interest to be the host for our next weekly?
16:08:19 <darknao> just saying hi :)
16:09:47 <phsmoura> ok. lets move to next topic we we are ok for next weekly cause we have mkonecny
16:09:49 <nirik> I guess I can do next week if no one else.
16:10:04 <nirik> oh yeah, misread
16:10:53 <zlopez> It seems the matrix bridge was lost or it's few minutes behind
16:11:08 <phsmoura> nirik: next week we have mkonecny, are you interest to put your name for 02-09?
16:11:28 <phsmoura> or wait til next meeting?
16:11:54 <dtometzki> i can take next week ?
16:11:59 <nirik> I can do 02-09
16:12:10 <dtometzki> or 02-09 ?
16:13:04 <dtometzki> Nirik you have a lot to do
16:13:15 <nirik> happy to let dtometzki take it too.
16:13:28 <phsmoura> dtometzki: ok so nirik on 02-09, maybe I can put your name on 02-16?
16:13:50 <phsmoura> ok
16:14:05 <phsmoura> #topic Next chair
16:14:05 <phsmoura> #info magic eight ball says:
16:14:05 <phsmoura> #info chair 2023-02-02 - mkonecny
16:14:05 <phsmoura> #info chair 2023-02-09 - dtometzki
16:14:05 <phsmoura> #info chair 2023-02-16- ?
16:14:21 <phsmoura> this is how it is now, we are fine for next couple weeks
16:14:43 <nirik> yeah, lets revisit next time...
16:14:47 <phsmoura> #topic announcements and information
16:14:48 <phsmoura> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:14:48 <phsmoura> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:14:48 <phsmoura> #info If your team wants support from the Fedora Program Management Team, file an isssue: https://pagure.io/fedora-pgm/pgm_team/issues?template=support_request
16:14:49 <phsmoura> #info oncall should also handling #fedora-releng pings if possible
16:14:51 <phsmoura> #info F38 branching from rawhide on  2023-02-07
16:15:19 <phsmoura> any other announcement to do?
16:16:44 <mkonecny> #info Anitya 1.7.0 released today (https://release-monitoring.org)
16:17:12 <mkonecny> #info F38 mass rebuild finished
16:18:20 <jednorozec> modules are still rebuilding
16:19:55 <phsmoura> ok, thanks mkonecny and jednorozec
16:21:02 <phsmoura> Is there anything we should discuss as a topic before oncall?
16:21:29 * nirik doesn't have anything
16:22:47 <phsmoura> lets move to oncall then
16:23:08 <phsmoura> #topic Oncall
16:23:09 <phsmoura> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:23:09 <phsmoura> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
16:23:29 <phsmoura> #info dtometzki is on call from 2023-01-19 to 2023-01-26
16:23:29 <phsmoura> #info  darknao is on call from 2023-01-26 to 2023-02-02
16:23:29 <phsmoura> #info ? is on call from 2023-02-02 to 2023-02-09
16:23:52 <jednorozec> .oncalltakeeu
16:23:53 <zodbot> jednorozec: Error: You don't have the alias.add capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified.
16:24:13 <phsmoura> volunteers from 2023-02-02 to 2023-02-09 ?
16:24:25 <darknao> jednorozec: i think that's my turn
16:24:25 <mkonecny> I can take it
16:25:02 <phsmoura> before the summary of last week
16:25:02 <jednorozec> ok
16:25:10 <darknao> .oncalltakeeu
16:25:10 <zodbot> darknao: Kneel before zod!
16:25:47 <phsmoura> sorry, jednorozec you were interested to be oncall from 2023-02-02 to 2023-02-09?
16:27:27 <jednorozec> darknao, took it so I can take the next one
16:28:50 <phsmoura> jednorozec: darknao took from 2023-01-26 to 2023-02-02
16:29:04 <phsmoura> 2023-02-02 to 2023-02-09 is the week after that
16:29:04 <jednorozec> oh
16:29:18 <jednorozec> sign me up for it
16:30:19 <phsmoura> ok, mkonecny you have shown interest as well can I put your name oncall from 2023-02-09 to 2023-02-16?
16:30:41 <mkonecny> phsmoura: I'm OK with that
16:31:10 <phsmoura> so this is how its for next weeks
16:31:12 <phsmoura> #info dtometzki is on call from 2023-01-19 to 2023-01-26
16:31:13 <phsmoura> #info  darknao is on call from 2023-01-26 to 2023-02-02
16:31:13 <phsmoura> #info jednorozec is on call from 2023-02-02 to 2023-02-09
16:31:13 <phsmoura> #info mkonecny is on call from 2023-02-09 to 2023-02-16
16:31:35 <phsmoura> #info Summary of last week: dtometzki
16:31:46 <phsmoura> now, summary of last week
16:32:01 <phsmoura> floor is yours dtometzki
16:32:10 <dtometzki> Summary: Ther was only one ping at weekend because of proxies
16:32:27 <dtometzki> s/Ther/There/
16:34:19 <nirik> hopefully thats all fixed now... althought there was one issue with a proxy yesterday, but seemed different.
16:34:58 <phsmoura> ok, thanks for being oncall last week :)
16:35:08 <phsmoura> lets move to monitoring
16:35:11 <phsmoura> #topic Monitoring discussion [nirik]
16:35:11 <phsmoura> #info https://nagios.fedoraproject.org/nagios
16:35:11 <phsmoura> #info Go over existing out items and fix
16:35:43 <phsmoura> do you have anything nirik?
16:35:51 <nirik> nothing too different this week...
16:36:00 <nirik> I did file a ticket on some alerts we should clean up.
16:36:08 <nirik> if anyone(s) want to work on it.
16:36:35 <nirik> otherwise just status quo... we can move on
16:37:50 <phsmoura> ok, we dont have learning topic today but next week we do
16:37:56 <phsmoura> #info 2023-02-02 Using Btrbk for BTRFS snapshot backups [eddiejennings]
16:38:17 <phsmoura> so lets move to backlog refinement and see blocked issues
16:38:31 <phsmoura> #topic Revisit blocked tickets
16:38:39 <phsmoura> #info Check if any blocked ticket is unblocked
16:38:39 <phsmoura> #link https://pagure.io/fedora-infrastructure/issues?status=Open&tags=blocked&priority=0&close_status=
16:39:00 <eddiejenningsjr> I’m at a conference today, but will be ready to go for Feb 2nd :)
16:39:18 <phsmoura> any updates on any of those?
16:39:20 <petebuffon> Excited to hear more about BTRFS!
16:39:25 <nirik> nope. all blocked still as far as I know
16:40:13 <phsmoura> ok
16:40:17 <phsmoura> #topic Fedora Infra backlog refinement
16:40:17 <phsmoura> #info Refine oldest tickets on Fedora Infra tracker
16:40:17 <phsmoura> #link https://pagure.io/fedora-infrastructure/issues?status=Open&order_key=last_updated&order=asc
16:40:34 <phsmoura> in the backlog anything we should look or close?
16:41:23 <phsmoura> .ticket 10383
16:41:24 <zodbot> phsmoura: Issue #10383: Upgrade Venus to Pluto - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10383
16:41:46 <phsmoura> we done this already now the task is to move fedora planet to ocp
16:42:04 <nirik> well, no, it's not really done until the new one is deployed. :)
16:42:05 <phsmoura> should we close this one?
16:42:20 <nirik> I'd keep that ticket open to track deployment...
16:42:27 <phsmoura> ah ok
16:42:30 <nirik> but might update it with the current status... ?
16:44:10 <nirik> I can pick random ones to talk about, but I think we have categorized all of them
16:44:42 <phsmoura> So I was doing ocp training this week to have more understanding about the tool, nirik gave access to communishift so I can play around and see how to deploy it. Im one chapter to finish the ocp training then Ill be trying to deploy it
16:45:06 <nirik> excellent.
16:46:28 <phsmoura> so is there any other issue we should discuss?
16:46:43 * nirik looks over the list
16:47:03 <nirik> .ticket 10564
16:47:05 <zodbot> nirik: Issue #10564: update firmware/passwords on the rest of our hardware - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10564
16:47:34 <nirik> perhaps we should close this one? I think phsmoura confirmed all the passwords were updated.
16:47:48 <nirik> firmware is always a question, but it changes from time to time.
16:48:09 <nirik> so we just need to check before mass update/reboot cycles and see if there's some update.
16:48:39 <nirik> I guess aheath1992 was going to try and come up with a checker, but... that seems a big project.
16:49:24 <nirik> thoughts?
16:49:25 <phsmoura> yep, and I need to see how to update firmware now right nirik? Intend to do it after fedora planet task
16:50:36 <nirik> well, it's somewhat a manual process currently... I am not sure we can script it fully.
16:50:40 <mkonecny> nirik: Checker for what?
16:51:00 <nirik> mkonecny: firmware updates
16:51:31 <nirik> so: look up current firmware versions, check upstream for that exact model and see if there's any newer, download, upload to mgmt interface, reboot...
16:51:33 <mkonecny> What is used for firmware updates now? fwupd?
16:52:07 <mkonecny> Oh, so it's really a manual process :-/
16:53:08 <phsmoura> at the time I was having a look at imptool and lib python for it.. dont know if that could help
16:53:09 <nirik> I'd love fwupd to work.
16:53:24 <nirik> I am not sure you can do it via ipmitool
16:53:47 <nirik> oh and side note: this last week I learned theres a 'freeipmi' set of tools. :)
16:55:23 <nirik> anyhow. I'm in favor of just closing this... and if it becomes possible to do programatically we can revisit?
16:55:35 <phsmoura> +1
16:56:34 <phsmoura> want to close it nirik?
16:56:39 <nirik> sure
16:56:55 <nirik> I guess we are at time almost. ;)
16:57:42 <phsmoura> yep, lets move to open floor in case anyone has something to share very quickly
16:57:48 <phsmoura> #topic Open Floor
16:59:33 <darknao> I can make a real quick update about openshift monitoring
16:59:39 <darknao> .ticket 10671
16:59:40 <zodbot> darknao: Issue #10671: Alert admins of the openshift apps when the pod crashes - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10671
17:00:13 <darknao> we have deployed an initial Alertmanager config with few rules on staging
17:00:44 <darknao> it's working quite well, but I've a few rules to tweak to make it less "spammy"
17:01:15 <darknao> I'll push an update on that later this week, then maybe onboard one or two more project to see how it scale
17:01:28 <nirik> good stuff!
17:01:30 <nirik> darknao++
17:01:40 <phsmoura> thanks darknao. thats it for today then
17:01:50 <darknao> then create an ansible role so appowner can just opt-in without rewriting standard rules
17:01:50 <phsmoura> thank you all for attending and have a great weekend :)
17:02:40 <mkonecny> Thanks phsmoura
17:03:11 <phsmoura> #endmeeting