#meeting-3:fedoraproject.org log

<@pcreech:matrix.org>

16:02:09

!startmeeting

<@meetbot:fedora.im>

16:02:11

Meeting started at 2024-07-11 16:02:09 UTC

<@meetbot:fedora.im>

16:02:11

The Meeting name is 'Fedora Meeting 3'

<@nphilipp:fedora.im>

16:02:27

Does it need to have the full title?

<@pcreech:matrix.org>

16:02:30

!meetingname Infrastructure

<@meetbot:fedora.im>

16:02:31

The Meeting Name is now Infrastructure

<@nphilipp:fedora.im>

16:02:37

👍

<@pcreech:matrix.org>

16:02:44

!info Agenda is at: https://board.net/p/fedora-infra

<@pcreech:matrix.org>

16:02:44

!info About our team: https://docs.fedoraproject.org/en-US/cpe/

<@pcreech:matrix.org>

16:02:44

!info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra

<@pcreech:matrix.org>

16:02:44

!topic namaste

<@pcreech:matrix.org>

16:02:44

!chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp

<@pcreech:matrix.org>

16:03:22

.hello pcreech17

<@pcreech:matrix.org>

16:03:54

just dusting off some cobwebs in my brain, welcome everyone!

<@nphilipp:fedora.im>

16:04:03

!hello nphilipp

<@zodbot:fedora.im>

16:04:03

Nils Philippsen (nphilipp) - he / him / his

<@nirik:matrix.scrye.com>

16:04:10

morning

<@humaton:fedora.im>

16:04:16

!hi

<@zodbot:fedora.im>

16:04:17

Tomáš Hrčka (humaton) - he / him / his

<@nphilipp:fedora.im>

16:04:27

(Attention letter is the exclamation mark with the Matrix bot)

<@pcreech:matrix.org>

16:05:08

yeah, i keep mixing that up, fingers are still used to the ol'e '.hello'

<@pcreech:matrix.org>

16:05:16

!hello pcreech17

<@zodbot:fedora.im>

16:05:17

Patrick Creech (pcreech17)

<@jnsamyak:matrix.org>

16:05:31

!hi

<@pcreech:matrix.org>

16:05:32

!info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/

<@pcreech:matrix.org>

16:05:32

!topic New folks introductions

<@pcreech:matrix.org>

16:05:32

!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves

<@zodbot:fedora.im>

16:05:32

Samyak Jain (jnsamyak) - he / him / his

<@james:fedora.im>

16:05:34

!hi

<@zodbot:fedora.im>

16:05:35

James Antill (james)

<@pcreech:matrix.org>

16:05:48

We have any new folks today?

<@maztaim:fedora.im>

16:06:14

Hello, Tim Bosse here. Just hanging out to see what's what.

<@pcreech:matrix.org>

16:06:35

Welcome in!

<@nirik:matrix.scrye.com>

16:06:57

welcome!

<@Zlopez:matrix.org>

16:07:05

!hi

<@zodbot:fedora.im>

16:07:06

Michal Konecny (zlopez)

<@Zlopez:matrix.org>

16:07:15

Sorry for being a little late

<@pcreech:matrix.org>

16:08:19

!info chair 2024-08-01 ???

<@pcreech:matrix.org>

16:08:19

!topic Next chair

<@pcreech:matrix.org>

16:08:19

!info magic eight ball says:

<@pcreech:matrix.org>

16:08:19

!info chair 2024-07-11 pcreech

<@pcreech:matrix.org>

16:08:19

!info chair 2024-07-18 zlopez

<@pcreech:matrix.org>

16:08:19

!info chair 2024-07-25 ???

<@pcreech:matrix.org>

16:09:00

Looks like we have next week handled by the one and only zlopez, anyone interested in taking any after?

<@pcreech:matrix.org>

16:11:00

I can probably do the first if no other volunteers

<@nirik:matrix.scrye.com>

16:13:05

I suppose I could do the 25th probibly

<@pcreech:matrix.org>

16:13:45

going once, twice,

<@pcreech:matrix.org>

16:14:13

sold!

<@pcreech:matrix.org>

16:14:25

!info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3

<@pcreech:matrix.org>

16:14:25

!info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org

<@pcreech:matrix.org>

16:14:25

!topic announcements and information

<@pcreech:matrix.org>

16:14:57

!topic Oncall

<@pcreech:matrix.org>

16:14:57

!info https://fedoraproject.org/wiki/Infrastructure/Oncall

<@pcreech:matrix.org>

16:14:57

!info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/

<@pcreech:matrix.org>

16:15:09

!info nirik is on call from 2024-06-28 to 2024-07-04

<@pcreech:matrix.org>

16:15:09

!info ? is on call from 2024-07-05 to 2024-07-11

<@pcreech:matrix.org>

16:15:45

looks like oncall info is kinda sparse as well.... last entry was ??? for up to today

<@nirik:matrix.scrye.com>

16:15:47

I guess I took it an extra week. :)

<@nirik:matrix.scrye.com>

16:16:14

The only oncall I can remember was this morning a bit ago... some folks wanting help with an openshift app.

<@pcreech:matrix.org>

16:17:46

!info ??? is on call from 2024-07-19 to 2024-07-25

<@pcreech:matrix.org>

16:17:46

!info ??? is on call from 2024-07-12 to 2024-07-18

<@pcreech:matrix.org>

16:17:46

<@pcreech:matrix.org>

16:18:03

I _think_ i got fthat formatted right. any takers for either of those slots for on call?

<@Zlopez:matrix.org>

16:18:27

I can probably take the next week

<@pcreech:matrix.org>

16:19:11

starting tomorrow or starting next friday?

<@Zlopez:matrix.org>

16:19:17

Tomorrow

<@pcreech:matrix.org>

16:19:32

enthusiastically sold!

<@Zlopez:matrix.org>

16:19:40

It will end together with me chairing the event :-)

<@aggraxis:fedora.im>

16:19:54

the oncallocalypse!

<@Zlopez:matrix.org>

16:20:20

!oncall

<@zodbot:fedora.im>

16:20:21

● @Zlopez:matrix.org (zlopez) Current Time for them: 18:20 (Europe/Prague)

<@zodbot:fedora.im>

16:20:21

If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)

<@zodbot:fedora.im>

16:20:21

<@zodbot:fedora.im>

16:20:21

The following people are oncall:

<@Zlopez:matrix.org>

16:20:29

Everything set :-)

<@pcreech:matrix.org>

16:20:32

any takers for july 19 through 25th?

<@pcreech:matrix.org>

16:21:58

we'll leave it open then until next meeting.

<@pcreech:matrix.org>

16:22:07

!info Summary of last week: (from current oncall)

<@nirik:matrix.scrye.com>

16:22:20

The only oncall I can remember was this morning a bit ago... some folks wanting help with an openshift app.

<@nirik:matrix.scrye.com>

16:22:21

<@pcreech:matrix.org>

16:23:06

I believe nirik said something before about being pinged this am about some help with an openshift app.

<@pcreech:matrix.org>

16:23:10

yeah, that :D

<@pcreech:matrix.org>

16:23:33

!info Go over existing out items and fix

<@pcreech:matrix.org>

16:23:33

!info https://nagios.fedoraproject.org/nagios

<@pcreech:matrix.org>

16:23:33

!topic Monitoring discussion [nirik]

<@pcreech:matrix.org>

16:23:33

<@nirik:matrix.scrye.com>

16:24:20

well, lets see.

<@nirik:matrix.scrye.com>

16:24:42

Things are pretty nasty looking right now, because I did a mass update/reboot yesterday and... a bunch of our aarch64 machines wouldn't come back

<@pcreech:matrix.org>

16:24:53

nirik any monitoring stuff this week?

<@nirik:matrix.scrye.com>

16:25:13

Otherwise, things are looking great. ;)

<@nirik:matrix.scrye.com>

16:25:35

I'll be poking at the aarch64 boxes today after meetings.

<@pcreech:matrix.org>

16:26:24

wow my client de-synced there for a bit, apologies. :D cool!

<@nirik:matrix.scrye.com>

16:26:49

seems to happen. pesky federation

<@pcreech:matrix.org>

16:26:59

yup

<@pcreech:matrix.org>

16:27:05

!ticket https://pagure.io/fedora-infrastructure/issue/11815

<@pcreech:matrix.org>

16:27:05

!info Check the progress on tickets related to RHEL7 EOL (end of June)

<@pcreech:matrix.org>

16:27:05

!topic RHEL7 EOL

<@pcreech:matrix.org>

16:27:30

Any RHEL7 EOL updates?

<@nirik:matrix.scrye.com>

16:28:10

so, just a few things left...

<@nirik:matrix.scrye.com>

16:28:13

all being worked on

<@nirik:matrix.scrye.com>

16:28:26

I hope we can close this out in the next few weeks.

<@Zlopez:matrix.org>

16:29:18

What remains is PDC, fedimg, github2fedmsg and fedmsg itself

<@nirik:matrix.scrye.com>

16:29:37

(and 2 virthosts... making some progress with them)

<@pcreech:matrix.org>

16:30:25

verynice, looks like things are progressing well

<@pcreech:matrix.org>

16:30:40

!info Check if any blocked ticket is unblocked

<@pcreech:matrix.org>

16:30:40

!topic Revisit blocked tickets

<@pcreech:matrix.org>

16:30:40

!link https://pagure.io/fedora-infrastructure/issues?status=Open&tags=blocked&priority=0&close_status=

<@pcreech:matrix.org>

16:31:10

any updates on any of the blocked issues?

<@nirik:matrix.scrye.com>

16:31:23

we might be able to do something about 11427 now...

<@pcreech:matrix.org>

16:31:44

!ticket 11427

<@zodbot:fedora.im>

16:31:46

● **Assignee:** Not Assigned

<@zodbot:fedora.im>

16:31:46

<@zodbot:fedora.im>

16:31:46

● **Opened:** 12 months ago by darknao

<@zodbot:fedora.im>

16:31:46

● **Last Updated:** 5 months ago

<@zodbot:fedora.im>

16:31:46

**fedora-infrastructure #11427** (https://pagure.io/fedora-infrastructure/issue/11427):**DMARC validation failed for mail from lists.fp-o**

<@Zlopez:matrix.org>

16:31:55

With the new mailman deployment we should be able to do something about that

<@Zlopez:matrix.org>

16:32:11

But first we should solve the memory spikes

<@nirik:matrix.scrye.com>

16:32:17

needs investigation to see if we can set it globally.

<@nirik:matrix.scrye.com>

16:32:25

we had it per list before.

<@nirik:matrix.scrye.com>

16:33:18

yeah, so I guess we could leave this blocked for now until we figure out the memory problems

<@Zlopez:matrix.org>

16:34:02

I think I have a solution for that, but it's not great

<@Zlopez:matrix.org>

16:35:05

I was able to find that the memory spikes are caused when `/archives/search` endpoint is requeted

<@Zlopez:matrix.org>

16:35:38

So I have this PR https://pagure.io/fedora-infra/ansible/pull-request/2147 that will disable the search for now till the search index fully regenerate

<@Zlopez:matrix.org>

16:36:01

You can see how it works on https://lists.stg.fedoraproject.org/, where I tested it

<@nirik:matrix.scrye.com>

16:36:11

worth trying for sure. +1

<@Zlopez:matrix.org>

16:36:48

We should probably announce that, so people are not confused that the search field is missing

<@Zlopez:matrix.org>

16:37:22

I'm not sure how long the regeneration of search index will take

<@Zlopez:matrix.org>

16:37:33

Right now it has around 30 GB

<@nirik:matrix.scrye.com>

16:37:35

I suppose. I am not sure how many peopel use it even...

<@nirik:matrix.scrye.com>

16:37:44

most folks probibly use a search engine

<@nirik:matrix.scrye.com>

16:37:54

but I'm sure it's useful

<@Zlopez:matrix.org>

16:38:33

In that case we can just disable it and enable it after some time, hopefully that will prevent another memory spike

<@nirik:matrix.scrye.com>

16:38:48

yep. I think it's worth trying.

<@nirik:matrix.scrye.com>

16:39:05

note that we could give it more memory/cpus, but might need to move it to another virthost

<@Zlopez:matrix.org>

16:39:09

I will do it tomorrow, so I can deal with any issue if something happens

<@nirik:matrix.scrye.com>

16:39:32

it looks like the new one is using a lot less space than the old one (but I think the old one had a copy of the archives in files from mailman2)

<@nirik:matrix.scrye.com>

16:39:41

disk space that is.

<@Zlopez:matrix.org>

16:40:16

I didn't moved any archives when migrating, do you think we should move them?

<@Zlopez:matrix.org>

16:40:36

The LV should be still around on the vmhost

<@nphilipp:fedora.im>

16:40:43

Couldn’t there be a notice "search is missing because …" instead of the search bar

<@nphilipp:fedora.im>

16:41:02

Then we don’t need to trust people to read announcements 😉

<@Zlopez:matrix.org>

16:41:07

nils: I can definitely do that

<@Zlopez:matrix.org>

16:41:21

It's just HTML

<@nirik:matrix.scrye.com>

16:41:35

well, what good are they to us now?

<@nirik:matrix.scrye.com>

16:41:41

it's all in the db right?

<@nirik:matrix.scrye.com>

16:41:47

why store it 3 places?

<@Zlopez:matrix.org>

16:42:05

I'm not sure, somebody complained that some old mails are missing

<@Zlopez:matrix.org>

16:42:34

!ticket 12048

<@zodbot:fedora.im>

16:42:34

**fedora-infrastructure #12048** (https://pagure.io/fedora-infrastructure/issue/12048):**anaconda-devel list archive prior to 2021 missing**

<@zodbot:fedora.im>

16:42:34

● **Assignee:** Not Assigned

<@zodbot:fedora.im>

16:42:34

● **Last Updated:** 8 hours ago

<@zodbot:fedora.im>

16:42:34

● **Opened:** 15 hours ago by ngompa

<@zodbot:fedora.im>

16:42:34

<@nirik:matrix.scrye.com>

16:42:49

yeah, I saw, but that needs investigating. I am not sure where the old mails were missing from... was it from the old redhat.com list? or when we made the initial devel list? or when we moved from mailman2 to mailman3

<@nirik:matrix.scrye.com>

16:43:02

there's multiple places things were moved/imported.

<@nirik:matrix.scrye.com>

16:43:40

I guess: 1) was it in old redhat.com one imported into mailman2, 2) was it in mailman2 imported to mailman3 3) was it in old mailman3 before we upgraded

<@Zlopez:matrix.org>

16:43:41

My initial assumptions were that everything is in DB

<@nirik:matrix.scrye.com>

16:44:04

right, I am saying it may never have been in our db if it wasn't imported from a previous step

<@Zlopez:matrix.org>

16:44:16

I was looking at the archive and the oldest mail is testing the new deployment

<@pcreech:matrix.org>

16:44:18

Seeing as how this is still a vibrant conversation, and my hand cramp is not excited about going through backlog refinement tickets.... switching to open floor to keep this going :D

<@pcreech:matrix.org>

16:44:20

!topic Open Floor

<@nirik:matrix.scrye.com>

16:45:10

anyhow, we should look and try and figure out where it went

<@Zlopez:matrix.org>

16:45:37

That sounds like a job for detective :-)

<@Zlopez:matrix.org>

16:46:00

Oldest mail in the list https://lists.fedoraproject.org/archives/list/anaconda-devel@lists.fedoraproject.org/thread/ENOLPEVESN7BYWQKVEKYW3ROYN2WHDB5/

<@nirik:matrix.scrye.com>

16:46:10

A private eye in a fedora?

<@pcreech:matrix.org>

16:46:32

A publicly available private eye even....

<@Zlopez:matrix.org>

16:46:40

Some shadowman?

<@nirik:matrix.scrye.com>

16:47:47

perhaps so.

<@nirik:matrix.scrye.com>

16:48:15

one more small one for open floor... there's a sync of the developers.fedoraproject.org site thats throwing errors if anyone wants to fix it.

<@nirik:matrix.scrye.com>

16:48:24

I can file a ticket on it.

<@nirik:matrix.scrye.com>

16:48:55

AttributeError: module 'feedparser' has no attribute '_HTMLSanitizer'

<@nirik:matrix.scrye.com>

16:48:55

File "/usr/local/bin/rss.py", line 12, in <module>

<@nirik:matrix.scrye.com>

16:48:55

~~~ Traceback (most recent call last):

<@nirik:matrix.scrye.com>

16:48:55

feedparser._HTMLSanitizer.unacceptable_elements_with_end_tag.add('<div>')

<@pcreech:matrix.org>

16:49:13

nirik: sounds like something I might be able to play around with in my spare time.

<@Zlopez:matrix.org>

16:49:20

File a ticket, that always helps

<@nirik:matrix.scrye.com>

16:49:47

cool. yeah, it's likely something that changed with feedparser in rhel9

<@pcreech:matrix.org>

16:50:01

file ticket, ping me with it and i'll at least start messing around

<@nirik:matrix.scrye.com>

16:51:42

https://pagure.io/fedora-infrastructure/issue/12050

<@pcreech:matrix.org>

16:52:15

awesome

<@Zlopez:matrix.org>

16:52:36

Few documentation PRs I wanted to mention here:

<@Zlopez:matrix.org>

16:52:36

* https://pagure.io/infra-docs-fpo/pull-request/305

<@Zlopez:matrix.org>

16:52:36

* https://pagure.io/infra-docs-fpo/pull-request/303

<@Zlopez:matrix.org>

16:52:45

Just improving the docs

<@nirik:matrix.scrye.com>

16:53:35

cool. will review.

<@nirik:matrix.scrye.com>

16:53:51

I hope to land more before flock... if I can get time with less fires.

<@Zlopez:matrix.org>

16:54:20

The situation slowly improves :-)

<@nirik:matrix.scrye.com>

16:54:31

Oh, and... speaking of flock....

<@nirik:matrix.scrye.com>

16:55:02

I plan to collect all the ideas from folks about the workshop and add some more and post about it. Perhaps we can get an idea in advance what we want to cover.

<@Zlopez:matrix.org>

16:55:45

I have the Fedora infra talk as well

<@nirik:matrix.scrye.com>

16:56:13

let me know if you would like help with that.

<@Zlopez:matrix.org>

16:56:35

I need to check what I exactly wrote in proposal

<@Zlopez:matrix.org>

16:57:27

But let you know if I would need any help

<@nirik:matrix.scrye.com>

16:57:44

The schedule is out now and as usual, there's too many good things to go to all at the same time. ;)

<@nirik:matrix.scrye.com>

16:57:51

anyhow, we are almost out of time...

<@pcreech:matrix.org>

16:58:03

That we are! Last call for open floor items!

<@Zlopez:matrix.org>

16:58:25

nirik: And there is always a great hallway track :-D

<@pcreech:matrix.org>

16:58:51

Thanks everyone for joining my super scuffed, cobwebbed run of the weekly fedora infra meeting! You don't have to go home, and you can stay here, but the meeting will no longer be logged! Bye Bye!

<@nirik:matrix.scrye.com>

16:58:52

always

<@aggraxis:fedora.im>

16:59:00

the schedule is out?

<@aggraxis:fedora.im>

16:59:05

I haven't seen it yet

<@Zlopez:matrix.org>

16:59:09

pcreech: Thanks for running it

<@aggraxis:fedora.im>

16:59:20

yes thanks pcreech !!!

<@pcreech:matrix.org>

16:59:25

!endmeeting