16:00:28 <mkonecny> #startmeeting Infrastructure (2023-06-08)
16:00:28 <zodbot> Meeting started Thu Jun  8 16:00:28 2023 UTC.
16:00:28 <zodbot> This meeting is logged and archived in a public location.
16:00:28 <zodbot> The chair is mkonecny. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:00:28 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:28 <zodbot> The meeting name has been set to 'infrastructure_(2023-06-08)'
16:00:28 <mkonecny> #meetingname infrastructure
16:00:28 <zodbot> The meeting name has been set to 'infrastructure'
16:00:28 <mkonecny> #chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg
16:00:28 <zodbot> Current chairs: bodanel dtometzki jnsamyak lenkaseg mkonecny nb nirik zlopez
16:00:33 <mkonecny> #info Agenda is at: https://board.net/p/fedora-infra
16:00:33 <mkonecny> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:00:33 <mkonecny> #info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
16:00:34 <mkonecny> #topic greetings!
16:00:46 <mkonecny> Hi everybody
16:00:51 <lenkaseg> .hi
16:00:52 <zodbot> lenkaseg: lenkaseg 'Lenka Segura' <lenka@sepu.cz>
16:00:52 <nirik> morning
16:00:57 <mkonecny> I'm our host for today
16:01:15 <smooge> mello
16:01:56 <mkonecny> So how is everybody?
16:02:15 <patrikp[m]> Good evening.
16:02:32 <phsmoura> .hi
16:02:35 <zodbot> phsmoura: phsmoura 'Pedro Moura' <pmoura@redhat.com>
16:02:55 <jnsamyak> .hi
16:02:56 <zodbot> jnsamyak: jnsamyak 'Samyak Jain' <samyak.jn11@gmail.com>
16:03:33 * nirik is enjoying a ☕
16:03:52 <mkonecny> I have a water at my side
16:04:08 <eddiejenningsjr> .hello eddiejennings
16:04:09 <zodbot> eddiejenningsjr: eddiejennings 'Eddie Jennings' <eddie@eddiejennings.net>
16:05:13 <mkonecny> So first let's look if somebody new is on the meeting
16:05:20 <mkonecny> #topic New folks introductions
16:05:20 <mkonecny> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:05:20 <mkonecny> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:05:47 <mkonecny> If you are new here, don't be shy and share a little info about yourself :-)
16:05:57 <darknao> .hi
16:05:58 <zodbot> darknao: darknao 'Francois Andrieu' <darknao@drkn.ninja>
16:06:42 <saibug[m]> Hi
16:07:48 <mkonecny> It seems that there aren't any newcomers
16:08:04 <leonkhan> hello everyone !
16:08:05 <mkonecny> Let's look who will run this show next week
16:08:10 <leonkhan> good afternoon :)
16:08:11 <mkonecny> #topic Next chair
16:08:11 <mkonecny> #info magic eight ball says:
16:08:19 <mkonecny> #info chair 2023-06-08 - mkonecny
16:08:19 <mkonecny> #info chair 2023-06-15 - eddiejennings
16:08:19 <mkonecny> #info chair 2023-06-22 - lenkaseg
16:08:32 <eddiejenningsjr> I'm ready to go for the 15th
16:08:55 <mkonecny> #info chair 2023-06-29 - ???
16:09:08 <mkonecny> We are well prepared for the next few weeks
16:09:33 <mkonecny> But do we have a volunteer for 2023-06-29?
16:10:48 <phsmoura> me
16:10:56 <mkonecny> Sold!
16:11:12 <mkonecny> #info chair 2023-06-29 - phsmoura
16:12:00 <mkonecny> Let's see what news are waiting for us
16:12:07 <mkonecny> #topic announcements and information
16:12:07 <mkonecny> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0730 UTC in #centos-meeting
16:12:07 <mkonecny> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:12:58 <mkonecny> Any other announcement?
16:14:03 <dtometzki> .hi
16:14:04 <zodbot> dtometzki: dtometzki 'Damian Tometzki' <damian@riscv.tometzki.de>
16:14:53 <nirik> #info network upgrades next tuesday
16:14:53 <mkonecny> dtometzki: I like the domain riscv.tometzki.de
16:15:25 <dtometzki> :-)
16:16:24 <mkonecny> It doesn't seem there are any announcements so let's move on
16:16:35 <mkonecny> #topic Oncall
16:16:35 <mkonecny> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:16:35 <mkonecny> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
16:16:41 <mkonecny> #info eddiejennings is on call from 2023-06-02 to 2023-06-08
16:16:41 <mkonecny> #info dtometzki is on call from 2023-06-09 to 2023-06-15
16:16:41 <mkonecny> #info ??? is on call from 2023-06-16 to 2023-06-22
16:16:58 <mkonecny> Any volunteer for 2023-06-16 to 2023-06-22
16:17:00 <mkonecny> ?
16:17:51 <eddiejenningsjr> For anyone new, it's a relatively easy way to be involved. :)
16:17:54 * nirik can if no one else... or we can just wait
16:18:45 <mkonecny> We still have next week filled, so let's just wait
16:19:07 <mkonecny> #info Summary of last week: (from current oncall )
16:19:22 <mkonecny> Eddie Jennings, Jr.: Your time to shine!
16:20:15 <eddiejenningsjr> It was a quiet week.  I was summoned once for a storage issue with Koji (if I recall right).  Alas, I couldn't ssh into said server, so I suggested a ticket be submitted.  I assume it was resolved.
16:20:35 <dtometzki> .oncalltakeeu
16:20:35 <zodbot> dtometzki: Error: You don't have the alias.add capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified.
16:21:28 <eddiejenningsjr> Other than that, no other summonings.
16:21:37 <nirik> It may have been a bad builder that was out of space causing issues. I can't recall.
16:23:06 <mkonecny> So let's continue
16:23:11 <mkonecny> #topic Monitoring discussion [nirik]
16:23:12 <mkonecny> #info https://nagios.fedoraproject.org/nagios
16:23:12 <mkonecny> #info Go over existing out items and fix
16:24:00 <nirik> lets see
16:24:16 <nirik> oh yeah, hit a anoying one last night when I ran the noc playbook.
16:24:30 <nirik> Some of our changes to fix notifs monitoring seem to have broken some checks on proxies.
16:24:43 <nirik> we need to fix that. ;( I hacked it in with a link
16:25:30 <nirik> otherwise we are looking good.
16:25:39 <nirik> there have been some new fmn alerts still.
16:25:55 <mkonecny> I noticed them, but didn't realized there are new
16:26:34 <nirik> no, I meant the new fmn application, not the old one (notifs)
16:26:48 <nirik> the alerts are the same... high queue on rabbitmq
16:27:26 <mkonecny> I know, the new one had the issue when they were rebuilding cache
16:29:35 <nirik> yeah, that might be it. Not sure.
16:29:56 <mkonecny> It just shows up and then resolves itself
16:30:32 <mkonecny> Let's go to backlog refinement, as we had the security roundtable last week
16:30:37 <mkonecny> #topic Revisit blocked tickets
16:30:37 <mkonecny> #info Check if any blocked ticket is unblocked
16:30:37 <mkonecny> #link https://pagure.io/fedora-infrastructure/issues?status=Open&tags=blocked&priority=0&close_status=
16:31:36 <mkonecny> I'm currently looking into 10372
16:31:44 <mkonecny> .ticket 10372
16:31:45 <zodbot> mkonecny: Issue #10372: [proposal] Move Pagure (pagure.io and src.fp.o) to use OIDC for web login - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10372
16:32:06 <mkonecny> But I'm still getting familiar with Ipsilon
16:32:19 <nirik> ok. cool. ;) Would be great to move that forward.
16:32:52 <mkonecny> At least it's now being tested against CentOS 9 Stream and Fedora 37
16:33:04 <mkonecny> Fedora 38 fails because of the lasso library, which just fails without any error
16:33:43 <nirik> :(
16:34:08 <nirik> FWIW, we use fedora 38 for our instances...
16:34:22 <mkonecny> And it's working?
16:34:37 <mkonecny> The error was really random
16:34:55 <nirik> seems to be. I am not sure what that error would cause?
16:35:11 <mkonecny> The ipsilon just sometime responded with 500 and no error anywhere
16:35:35 <mkonecny> It happened usually during redirect to SSO
16:36:48 <mkonecny> But other than that I don't have any update about the blocked tickets
16:36:51 <nirik> huh, perhaps thats been happening? it's hard to say if it's random
16:37:33 * nirik has no other blocked tickets updates, they are all blocked as far as I can see.
16:38:23 <mkonecny> It didn't happened on CentOS 9 Stream or Fedora 37, hopefully it will be solved with next lasso release
16:38:52 <nirik> I don't even know what lasso is. ;)
16:39:07 <mkonecny> https://lasso.entrouvert.org/
16:39:23 <mkonecny> It's libre SSO library
16:40:25 <mkonecny> Let's go to next topic
16:40:32 <mkonecny> #topic Fedora Infra backlog refinement
16:40:32 <mkonecny> #info Refine oldest tickets on Fedora Infra tracker
16:40:32 <mkonecny> #link https://pagure.io/fedora-infrastructure/issues?status=Open&order_key=last_updated&order=asc
16:40:49 <mkonecny> .ticket 9728
16:40:51 <zodbot> mkonecny: Issue #9728: [rfe] src.fedoraproject.org:/forks/... is not in the list of allowed SCMs - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9728
16:41:07 <nirik> so... lets see
16:41:15 <mkonecny> The PR in ansible is still open
16:41:50 <nirik> I'm hoping if I can carve out enough time today to test in staging, we can merge that pr.
16:41:51 <nirik> (along with the dnf5 changes)
16:41:59 <mkonecny> #link https://pagure.io/fedora-infra/ansible/pull-request/1365
16:42:17 <mkonecny> I should add the PR to the issue as well
16:42:39 <nirik> sure. Hopefully I'll push it out later today, unless there's some issue with it.
16:43:04 <nirik> and the dnf5 mock changes
16:43:30 <mkonecny> .ticket 11090
16:43:31 <zodbot> mkonecny: Issue #11090: nagios: busgateway01 and notifs nagios alerts - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11090
16:45:02 <nirik> so most of these were cleaned up, but 2 are left... and this is what broke those proxy checks.
16:45:21 <nirik> so, hopefully we can fix it all up.
16:45:54 <mkonecny> Let me update the ticket
16:46:51 <nirik> ok
16:47:05 <mkonecny> .ticket 8167
16:47:06 <zodbot> mkonecny: Issue #8167: Adding topic authorization to our RabbitMQ instances - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8167
16:48:09 <nirik> I'm not sure whats left on this one. We could ask Aurélien B
16:48:11 <mkonecny> It seems that Aurélien B had a plan, but didn't started to work on it yet
16:48:37 <abompard> So I did start to work on it
16:48:57 <abompard> but I'm at a point where I need to commit it to rabbitmq and things may break
16:49:07 <abompard> so I'm trying to avoid breaking too many things
16:49:40 <mkonecny> Could we try it in staging first?
16:49:55 <abompard> Yes, but some apps aren't on staging so it's not 100% sure
16:50:09 <abompard> but yeah, definitely staging first
16:50:35 <mkonecny> I will update the ticket
16:51:13 <abompard> Thanks. I'm trying to find ways to avoid breaking prod when I activate it there
16:51:32 <nirik> +1 thanks for that
16:51:44 <nirik> we could schedule an outage for it sometime...
16:51:52 <abompard> outage: everything
16:52:03 <abompard> well, not exactly
16:52:31 <abompard> I've sent emails to app maintainers for info and still waiting for answers
16:52:35 <abompard> will resend
16:52:37 <nirik> someday we need to move our rabbitmq cluster over to rhel9...
16:53:22 <abompard> I'm sure the rabbitmq folks have docs to upgrade clusters
16:53:57 <nirik> yeah, that doesn't need to be soon, just pondering if we need outage for a while that would be a good time to do that too...
16:54:04 <nirik> but I can look into it.
16:54:13 <nirik> still need to move ipa and a bunch of other things too.
16:54:43 <mkonecny> One more ticket
16:54:44 <mkonecny> .ticket 10241
16:54:45 <zodbot> mkonecny: Issue #10241: Port apps to OIDC - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10241
16:55:14 <mkonecny> I started looking into the Pagure
16:55:42 <nirik> I would really love to make a push for nuking pdc...
16:55:54 <nirik> and probibly we should drop nuancier too.
16:56:15 <mkonecny> Is the nuancier still used by community?
16:56:41 <mkonecny> We already have PDC initiative in our backlog, we should probably push it more
16:57:20 <nirik> we have it running, and I suppose people look at/download wallpapers from it... but it's not been used for voting for a while.
16:57:36 <nirik> we had some community member interested in taking it over, but we never really handled it to them.
16:58:01 <nirik> https://pagure.io/fedora-infrastructure/issue/10739
16:58:32 <nirik> on mailman3, I think newer versions use django-all-auth, which does support oidc.
16:58:59 <mkonecny> #link https://pagure.io/cpe/initiatives-proposal/issue/5
16:59:05 <mkonecny> Here is the PDC proposal
16:59:53 <mkonecny> Are the community folks still interested in taking over nuancier
17:00:07 <mkonecny> If it's not used for voting?
17:00:38 <nirik> I don't know. I don't think we heard back from that one person... we could ask design team about it.
17:00:43 <mkonecny> Is the mailman3 already packaged for Fedora/EPEL?
17:00:58 <mkonecny> I will add this to my TODO list
17:02:01 <mkonecny> The nuancier is on my TODO list
17:02:03 <nirik> It is for fedora... they are working on epel9.
17:02:16 <nirik> https://fedora-arc.readthedocs.io/en/latest/pdc/index.html is the pdc arc investigation
17:03:04 <nirik> which was a while ago...
17:03:31 <mkonecny> It should still reflect current situation
17:03:40 <nirik> probibly.
17:04:10 <mkonecny> But we can revisit it when the initiative will be planned
17:04:38 <mkonecny> Ok, we are now over our time
17:04:43 <mkonecny> Let's end it here
17:04:53 <mkonecny> #endmeeting