2024-07-11 16:02:09 <@pcreech:matrix.org> !startmeeting
2024-07-11 16:02:11 <@meetbot:fedora.im> Meeting started at 2024-07-11 16:02:09 UTC
2024-07-11 16:02:11 <@meetbot:fedora.im> The Meeting name is 'Fedora Meeting 3'
2024-07-11 16:02:27 <@nphilipp:fedora.im> Does it need to have the full title?
2024-07-11 16:02:30 <@pcreech:matrix.org> !meetingname Infrastructure
2024-07-11 16:02:31 <@meetbot:fedora.im> The Meeting Name is now Infrastructure
2024-07-11 16:02:37 <@nphilipp:fedora.im> 👍
2024-07-11 16:02:44 <@pcreech:matrix.org> !info Agenda is at: https://board.net/p/fedora-infra
2024-07-11 16:02:44 <@pcreech:matrix.org> !info About our team: https://docs.fedoraproject.org/en-US/cpe/
2024-07-11 16:02:44 <@pcreech:matrix.org> !info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
2024-07-11 16:02:44 <@pcreech:matrix.org> !topic namaste
2024-07-11 16:02:44 <@pcreech:matrix.org> !chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
2024-07-11 16:03:22 <@pcreech:matrix.org> .hello pcreech17
2024-07-11 16:03:54 <@pcreech:matrix.org> just dusting off some cobwebs in my brain, welcome everyone!
2024-07-11 16:04:03 <@nphilipp:fedora.im> !hello nphilipp
2024-07-11 16:04:03 <@zodbot:fedora.im> Nils Philippsen (nphilipp) - he / him / his
2024-07-11 16:04:10 <@nirik:matrix.scrye.com> morning
2024-07-11 16:04:16 <@humaton:fedora.im> !hi
2024-07-11 16:04:17 <@zodbot:fedora.im> Tomáš Hrčka (humaton) - he / him / his
2024-07-11 16:04:27 <@nphilipp:fedora.im> (Attention letter is the exclamation mark with the Matrix bot)
2024-07-11 16:05:08 <@pcreech:matrix.org> yeah, i keep mixing that up, fingers are still used to the ol'e '.hello'
2024-07-11 16:05:16 <@pcreech:matrix.org> !hello pcreech17
2024-07-11 16:05:17 <@zodbot:fedora.im> Patrick Creech (pcreech17)
2024-07-11 16:05:31 <@jnsamyak:matrix.org> !hi
2024-07-11 16:05:32 <@pcreech:matrix.org> !info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
2024-07-11 16:05:32 <@pcreech:matrix.org> !topic New folks introductions
2024-07-11 16:05:32 <@pcreech:matrix.org> !info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
2024-07-11 16:05:32 <@zodbot:fedora.im> Samyak Jain (jnsamyak) - he / him / his
2024-07-11 16:05:34 <@james:fedora.im> !hi
2024-07-11 16:05:35 <@zodbot:fedora.im> James Antill (james)
2024-07-11 16:05:48 <@pcreech:matrix.org> We have any new folks today?
2024-07-11 16:06:14 <@maztaim:fedora.im> Hello, Tim Bosse here.   Just hanging out to see what's what.
2024-07-11 16:06:35 <@pcreech:matrix.org> Welcome in!
2024-07-11 16:06:57 <@nirik:matrix.scrye.com> welcome!
2024-07-11 16:07:05 <@Zlopez:matrix.org> !hi
2024-07-11 16:07:06 <@zodbot:fedora.im> Michal Konecny (zlopez)
2024-07-11 16:07:15 <@Zlopez:matrix.org> Sorry for being a little late
2024-07-11 16:08:19 <@pcreech:matrix.org> !info chair 2024-08-01 ???
2024-07-11 16:08:19 <@pcreech:matrix.org> !topic Next chair
2024-07-11 16:08:19 <@pcreech:matrix.org> !info magic eight ball says:
2024-07-11 16:08:19 <@pcreech:matrix.org> !info chair 2024-07-11 pcreech
2024-07-11 16:08:19 <@pcreech:matrix.org> !info chair 2024-07-18 zlopez
2024-07-11 16:08:19 <@pcreech:matrix.org> !info chair 2024-07-25 ???
2024-07-11 16:09:00 <@pcreech:matrix.org> Looks like we have next week handled by the one and only zlopez, anyone interested in taking any after?
2024-07-11 16:11:00 <@pcreech:matrix.org> I can probably do the first if no other volunteers
2024-07-11 16:13:05 <@nirik:matrix.scrye.com> I suppose I could do the 25th probibly
2024-07-11 16:13:45 <@pcreech:matrix.org> going once, twice,
2024-07-11 16:14:13 <@pcreech:matrix.org> sold!
2024-07-11 16:14:25 <@pcreech:matrix.org> !info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
2024-07-11 16:14:25 <@pcreech:matrix.org> !info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2024-07-11 16:14:25 <@pcreech:matrix.org> !topic announcements and information
2024-07-11 16:14:57 <@pcreech:matrix.org> !topic Oncall
2024-07-11 16:14:57 <@pcreech:matrix.org> !info https://fedoraproject.org/wiki/Infrastructure/Oncall
2024-07-11 16:14:57 <@pcreech:matrix.org> !info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
2024-07-11 16:15:09 <@pcreech:matrix.org> !info nirik is on call from 2024-06-28 to 2024-07-04
2024-07-11 16:15:09 <@pcreech:matrix.org> !info ? is on call from 2024-07-05 to 2024-07-11
2024-07-11 16:15:45 <@pcreech:matrix.org> looks like oncall info is kinda sparse as well....  last entry was ??? for up to today
2024-07-11 16:15:47 <@nirik:matrix.scrye.com> I guess I took it an extra week. :)
2024-07-11 16:16:14 <@nirik:matrix.scrye.com> The only oncall I can remember was this morning a bit ago... some folks wanting help with an openshift app.
2024-07-11 16:17:46 <@pcreech:matrix.org> !info ??? is on call from 2024-07-19 to 2024-07-25
2024-07-11 16:17:46 <@pcreech:matrix.org> !info ??? is on call from 2024-07-12 to 2024-07-18
2024-07-11 16:17:46 <@pcreech:matrix.org> 
2024-07-11 16:18:03 <@pcreech:matrix.org> I _think_ i got fthat formatted right.  any takers for either of those slots for on call?
2024-07-11 16:18:27 <@Zlopez:matrix.org> I can probably take the next week
2024-07-11 16:19:11 <@pcreech:matrix.org> starting tomorrow or starting next friday? 
2024-07-11 16:19:17 <@Zlopez:matrix.org> Tomorrow
2024-07-11 16:19:32 <@pcreech:matrix.org> enthusiastically sold!
2024-07-11 16:19:40 <@Zlopez:matrix.org> It will end together with me chairing the event :-)
2024-07-11 16:19:54 <@aggraxis:fedora.im> the oncallocalypse!
2024-07-11 16:20:20 <@Zlopez:matrix.org> !oncall
2024-07-11 16:20:21 <@zodbot:fedora.im> ● @Zlopez:matrix.org (zlopez) Current Time for them: 18:20 (Europe/Prague)
2024-07-11 16:20:21 <@zodbot:fedora.im> If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)
2024-07-11 16:20:21 <@zodbot:fedora.im> 
2024-07-11 16:20:21 <@zodbot:fedora.im> The following people are oncall:
2024-07-11 16:20:29 <@Zlopez:matrix.org> Everything set :-)
2024-07-11 16:20:32 <@pcreech:matrix.org> any takers for july 19 through 25th?
2024-07-11 16:21:58 <@pcreech:matrix.org> we'll leave it open then until next meeting.
2024-07-11 16:22:07 <@pcreech:matrix.org> !info Summary of last week: (from current oncall)
2024-07-11 16:22:20 <@nirik:matrix.scrye.com> The only oncall I can remember was this morning a bit ago... some folks wanting help with an openshift app.
2024-07-11 16:22:21 <@nirik:matrix.scrye.com> :)
2024-07-11 16:23:06 <@pcreech:matrix.org> I believe nirik said something before about being pinged this am about some help with an openshift app. 
2024-07-11 16:23:10 <@pcreech:matrix.org> yeah, that :D
2024-07-11 16:23:33 <@pcreech:matrix.org> !info Go over existing out items and fix
2024-07-11 16:23:33 <@pcreech:matrix.org> !info https://nagios.fedoraproject.org/nagios 
2024-07-11 16:23:33 <@pcreech:matrix.org> !topic Monitoring discussion [nirik]
2024-07-11 16:23:33 <@pcreech:matrix.org> 
2024-07-11 16:24:20 <@nirik:matrix.scrye.com> well, lets see.
2024-07-11 16:24:42 <@nirik:matrix.scrye.com> Things are pretty nasty looking right now, because I did a mass update/reboot yesterday and... a bunch of our aarch64 machines wouldn't come back
2024-07-11 16:24:53 <@pcreech:matrix.org> nirik any monitoring stuff this week?
2024-07-11 16:25:13 <@nirik:matrix.scrye.com> Otherwise, things are looking great. ;)
2024-07-11 16:25:35 <@nirik:matrix.scrye.com> I'll be poking at the aarch64 boxes today after meetings.
2024-07-11 16:26:24 <@pcreech:matrix.org> wow my client de-synced there for a bit, apologies. :D cool!
2024-07-11 16:26:49 <@nirik:matrix.scrye.com> seems to happen. pesky federation
2024-07-11 16:26:59 <@pcreech:matrix.org> yup
2024-07-11 16:27:05 <@pcreech:matrix.org> !ticket https://pagure.io/fedora-infrastructure/issue/11815
2024-07-11 16:27:05 <@pcreech:matrix.org> !info Check the progress on tickets related to RHEL7 EOL (end of June)
2024-07-11 16:27:05 <@pcreech:matrix.org> !topic RHEL7 EOL
2024-07-11 16:27:30 <@pcreech:matrix.org> Any RHEL7 EOL updates?
2024-07-11 16:28:10 <@nirik:matrix.scrye.com> so, just a few things left...
2024-07-11 16:28:13 <@nirik:matrix.scrye.com> all being worked on
2024-07-11 16:28:26 <@nirik:matrix.scrye.com> I hope we can close this out in the next few weeks.
2024-07-11 16:29:18 <@Zlopez:matrix.org> What remains is PDC, fedimg, github2fedmsg and fedmsg itself
2024-07-11 16:29:37 <@nirik:matrix.scrye.com> (and 2 virthosts... making some progress with them)
2024-07-11 16:30:25 <@pcreech:matrix.org> verynice, looks like things are progressing well
2024-07-11 16:30:40 <@pcreech:matrix.org> !info Check if any blocked ticket is unblocked
2024-07-11 16:30:40 <@pcreech:matrix.org> !topic Revisit blocked tickets
2024-07-11 16:30:40 <@pcreech:matrix.org> !link https://pagure.io/fedora-infrastructure/issues?status=Open&tags=blocked&priority=0&close_status=
2024-07-11 16:31:10 <@pcreech:matrix.org> any updates on any of the blocked issues?
2024-07-11 16:31:23 <@nirik:matrix.scrye.com> we might be able to do something about 11427 now...
2024-07-11 16:31:44 <@pcreech:matrix.org> !ticket 11427
2024-07-11 16:31:46 <@zodbot:fedora.im> ● **Assignee:** Not Assigned
2024-07-11 16:31:46 <@zodbot:fedora.im> 
2024-07-11 16:31:46 <@zodbot:fedora.im> ● **Opened:** 12 months ago by darknao
2024-07-11 16:31:46 <@zodbot:fedora.im> ● **Last Updated:** 5 months ago
2024-07-11 16:31:46 <@zodbot:fedora.im> **fedora-infrastructure #11427** (https://pagure.io/fedora-infrastructure/issue/11427):**DMARC validation failed for mail from lists.fp-o**
2024-07-11 16:31:55 <@Zlopez:matrix.org> With the new mailman deployment we should be able to do something about that
2024-07-11 16:32:11 <@Zlopez:matrix.org> But first we should solve the memory spikes
2024-07-11 16:32:17 <@nirik:matrix.scrye.com> needs investigation to see if we can set it globally.
2024-07-11 16:32:25 <@nirik:matrix.scrye.com> we had it per list before.
2024-07-11 16:33:18 <@nirik:matrix.scrye.com> yeah, so I guess we could leave this blocked for now until we figure out the memory problems
2024-07-11 16:34:02 <@Zlopez:matrix.org> I think I have a solution for that, but it's not great
2024-07-11 16:35:05 <@Zlopez:matrix.org> I was able to find that the memory spikes are caused when `/archives/search` endpoint is requeted
2024-07-11 16:35:38 <@Zlopez:matrix.org> So I have this PR https://pagure.io/fedora-infra/ansible/pull-request/2147 that will disable the search for now till the search index fully regenerate
2024-07-11 16:36:01 <@Zlopez:matrix.org> You can see how it works on https://lists.stg.fedoraproject.org/, where I tested it
2024-07-11 16:36:11 <@nirik:matrix.scrye.com> worth trying for sure. +1
2024-07-11 16:36:48 <@Zlopez:matrix.org> We should probably announce that, so people are not confused that the search field is missing
2024-07-11 16:37:22 <@Zlopez:matrix.org> I'm not sure how long the regeneration of search index will take
2024-07-11 16:37:33 <@Zlopez:matrix.org> Right now it has around 30 GB
2024-07-11 16:37:35 <@nirik:matrix.scrye.com> I suppose. I am not sure how many peopel use it even...
2024-07-11 16:37:44 <@nirik:matrix.scrye.com> most folks probibly use a search engine
2024-07-11 16:37:54 <@nirik:matrix.scrye.com> but I'm sure it's useful
2024-07-11 16:38:33 <@Zlopez:matrix.org> In that case we can just disable it and enable it after some time, hopefully that will prevent another memory spike
2024-07-11 16:38:48 <@nirik:matrix.scrye.com> yep. I think it's worth trying.
2024-07-11 16:39:05 <@nirik:matrix.scrye.com> note that we could give it more memory/cpus, but might need to move it to another virthost
2024-07-11 16:39:09 <@Zlopez:matrix.org> I will do it tomorrow, so I can deal with any issue if something happens
2024-07-11 16:39:32 <@nirik:matrix.scrye.com> it looks like the new one is using a lot less space than the old one (but I think the old one had a copy of the archives in files from mailman2)
2024-07-11 16:39:41 <@nirik:matrix.scrye.com> disk space that is.
2024-07-11 16:40:16 <@Zlopez:matrix.org> I didn't moved any archives when migrating, do you think we should move them?
2024-07-11 16:40:36 <@Zlopez:matrix.org> The LV should be still around on the vmhost
2024-07-11 16:40:43 <@nphilipp:fedora.im> Couldn’t there be a notice "search is missing because …" instead of the search bar
2024-07-11 16:41:02 <@nphilipp:fedora.im> Then we don’t need to trust people to read announcements 😉
2024-07-11 16:41:07 <@Zlopez:matrix.org> nils: I can definitely do that
2024-07-11 16:41:21 <@Zlopez:matrix.org> It's just HTML
2024-07-11 16:41:35 <@nirik:matrix.scrye.com> well, what good are they to us now?
2024-07-11 16:41:41 <@nirik:matrix.scrye.com> it's all in the db right?
2024-07-11 16:41:47 <@nirik:matrix.scrye.com> why store it 3 places?
2024-07-11 16:42:05 <@Zlopez:matrix.org> I'm not sure, somebody complained that some old mails are missing
2024-07-11 16:42:34 <@Zlopez:matrix.org> !ticket 12048
2024-07-11 16:42:34 <@zodbot:fedora.im> **fedora-infrastructure #12048** (https://pagure.io/fedora-infrastructure/issue/12048):**anaconda-devel list archive prior to 2021 missing**
2024-07-11 16:42:34 <@zodbot:fedora.im> ● **Assignee:** Not Assigned
2024-07-11 16:42:34 <@zodbot:fedora.im> ● **Last Updated:** 8 hours ago
2024-07-11 16:42:34 <@zodbot:fedora.im> ● **Opened:** 15 hours ago by ngompa
2024-07-11 16:42:34 <@zodbot:fedora.im> 
2024-07-11 16:42:49 <@nirik:matrix.scrye.com> yeah, I saw, but that needs investigating. I am not sure where the old mails were missing from... was it from the old redhat.com list? or when we made the initial devel list? or when we moved from mailman2 to mailman3
2024-07-11 16:43:02 <@nirik:matrix.scrye.com> there's multiple places things were moved/imported.
2024-07-11 16:43:40 <@nirik:matrix.scrye.com> I guess: 1) was it in old redhat.com one imported into mailman2, 2) was it in mailman2 imported to mailman3 3) was it in old mailman3 before we upgraded
2024-07-11 16:43:41 <@Zlopez:matrix.org> My initial assumptions were that everything is in DB
2024-07-11 16:44:04 <@nirik:matrix.scrye.com> right, I am saying it may never have been in our db if it wasn't imported from a previous step
2024-07-11 16:44:16 <@Zlopez:matrix.org> I was looking at the archive and the oldest mail is testing the new deployment
2024-07-11 16:44:18 <@pcreech:matrix.org> Seeing as how this is still a vibrant conversation, and my hand cramp is not excited about going through backlog refinement tickets.... switching to open floor to keep this going :D 
2024-07-11 16:44:20 <@pcreech:matrix.org> !topic Open Floor
2024-07-11 16:45:10 <@nirik:matrix.scrye.com> anyhow, we should look and try and figure out where it went
2024-07-11 16:45:37 <@Zlopez:matrix.org> That sounds like a job for detective :-)
2024-07-11 16:46:00 <@Zlopez:matrix.org> Oldest mail in the list https://lists.fedoraproject.org/archives/list/anaconda-devel@lists.fedoraproject.org/thread/ENOLPEVESN7BYWQKVEKYW3ROYN2WHDB5/
2024-07-11 16:46:10 <@nirik:matrix.scrye.com> A private eye in a fedora?
2024-07-11 16:46:32 <@pcreech:matrix.org> A publicly available private eye even....
2024-07-11 16:46:40 <@Zlopez:matrix.org> Some shadowman?
2024-07-11 16:47:47 <@nirik:matrix.scrye.com> perhaps so.
2024-07-11 16:48:15 <@nirik:matrix.scrye.com> one more small one for open floor... there's a sync of the developers.fedoraproject.org site thats throwing errors if anyone wants to fix it.
2024-07-11 16:48:24 <@nirik:matrix.scrye.com> I can file a ticket on it.
2024-07-11 16:48:55 <@nirik:matrix.scrye.com> AttributeError: module 'feedparser' has no attribute '_HTMLSanitizer'
2024-07-11 16:48:55 <@nirik:matrix.scrye.com>   File "/usr/local/bin/rss.py", line 12, in <module>
2024-07-11 16:48:55 <@nirik:matrix.scrye.com> ~~~ Traceback (most recent call last):
2024-07-11 16:48:55 <@nirik:matrix.scrye.com>     feedparser._HTMLSanitizer.unacceptable_elements_with_end_tag.add('<div>')
2024-07-11 16:49:13 <@pcreech:matrix.org> nirik: sounds like something I might be able to play around with in my spare time.
2024-07-11 16:49:20 <@Zlopez:matrix.org> File a ticket, that always helps
2024-07-11 16:49:47 <@nirik:matrix.scrye.com> cool. yeah, it's likely something that changed with feedparser in rhel9
2024-07-11 16:50:01 <@pcreech:matrix.org> file ticket, ping me with it and i'll at least start messing around
2024-07-11 16:51:42 <@nirik:matrix.scrye.com> https://pagure.io/fedora-infrastructure/issue/12050
2024-07-11 16:52:15 <@pcreech:matrix.org> awesome
2024-07-11 16:52:36 <@Zlopez:matrix.org> Few documentation PRs I wanted to mention here:
2024-07-11 16:52:36 <@Zlopez:matrix.org> * https://pagure.io/infra-docs-fpo/pull-request/305
2024-07-11 16:52:36 <@Zlopez:matrix.org> * https://pagure.io/infra-docs-fpo/pull-request/303
2024-07-11 16:52:45 <@Zlopez:matrix.org> Just improving the docs
2024-07-11 16:53:35 <@nirik:matrix.scrye.com> cool. will review.
2024-07-11 16:53:51 <@nirik:matrix.scrye.com> I hope to land more before flock... if I can get time with less fires.
2024-07-11 16:54:20 <@Zlopez:matrix.org> The situation slowly improves :-)
2024-07-11 16:54:31 <@nirik:matrix.scrye.com> Oh, and... speaking of flock....
2024-07-11 16:55:02 <@nirik:matrix.scrye.com> I plan to collect all the ideas from folks about the workshop and add some more and post about it. Perhaps we can get an idea in advance what we want to cover.
2024-07-11 16:55:45 <@Zlopez:matrix.org> I have the Fedora infra talk as well
2024-07-11 16:56:13 <@nirik:matrix.scrye.com> let me know if you would like help with that.
2024-07-11 16:56:35 <@Zlopez:matrix.org> I need to check what I exactly wrote in proposal
2024-07-11 16:57:27 <@Zlopez:matrix.org> But let you know if I would need any help
2024-07-11 16:57:44 <@nirik:matrix.scrye.com> The schedule is out now and as usual, there's too many good things to go to all at the same time. ;)
2024-07-11 16:57:51 <@nirik:matrix.scrye.com> anyhow, we are almost out of time...
2024-07-11 16:58:03 <@pcreech:matrix.org> That we are! Last call for open floor items!
2024-07-11 16:58:25 <@Zlopez:matrix.org> nirik: And there is always a great hallway track :-D
2024-07-11 16:58:51 <@pcreech:matrix.org> Thanks everyone for joining my super scuffed, cobwebbed run of the weekly fedora infra meeting!  You don't have to go home, and you can stay here, but the meeting will no longer be logged!  Bye Bye!
2024-07-11 16:58:52 <@nirik:matrix.scrye.com> always
2024-07-11 16:59:00 <@aggraxis:fedora.im> the schedule is out?
2024-07-11 16:59:05 <@aggraxis:fedora.im> I haven't seen it yet
2024-07-11 16:59:09 <@Zlopez:matrix.org> pcreech: Thanks for running it
2024-07-11 16:59:20 <@aggraxis:fedora.im> yes thanks pcreech !!!
2024-07-11 16:59:25 <@pcreech:matrix.org> !endmeeting