16:00:54 <dtometzki> #startmeeting Infrastructure (2022-02-10)
16:00:54 <zodbot> Meeting started Thu Feb 10 16:00:54 2022 UTC.
16:00:54 <zodbot> This meeting is logged and archived in a public location.
16:00:54 <zodbot> The chair is dtometzki. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:00:54 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:54 <zodbot> The meeting name has been set to 'infrastructure_(2022-02-10)'
16:00:54 <dtometzki> #meetingname infrastructure
16:00:54 <zodbot> The meeting name has been set to 'infrastructure'
16:00:54 <dtometzki> #chair nirik siddharthvipul mobrien zlopez pingou bodanel dtometzki jnsamyak computerkid
16:00:54 <dtometzki> #info Agenda is at: https://board.net/p/fedora-infra
16:00:54 <dtometzki> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:00:54 <zodbot> Current chairs: bodanel computerkid dtometzki jnsamyak mobrien nirik pingou siddharthvipul zlopez
16:00:55 <dtometzki> #topic greetings!
16:01:09 <dtometzki> .hi
16:01:10 <zodbot> dtometzki: dtometzki 'Damian Tometzki' <linux@tometzki.de>
16:01:11 <mobrien> .hi
16:01:13 <zodbot> mobrien: mobrien 'Mark O'Brien' <markobri@redhat.com>
16:01:18 <mkonecny> .hello zlopez
16:01:19 <zodbot> mkonecny: zlopez 'Michal Konecny' <michal.konecny@psmail.xyz>
16:01:43 <phsmoura> .hi
16:01:44 <zodbot> phsmoura: phsmoura 'Pedro Moura' <moura.pedro123@gmail.com>
16:01:48 <petebuffon> .hello petebuffon
16:01:49 <zodbot> petebuffon: petebuffon 'Peter Buffon' <pabuffon@gmail.com>
16:03:02 <dtometzki> hello together :-)
16:03:06 <nirik> morning
16:03:15 <dtometzki> morning nirik
16:03:50 * nirik 🌊s
16:04:00 <dtometzki> #topic New folks introductions
16:04:00 <dtometzki> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:04:00 <dtometzki> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:04:30 <dtometzki> Any newcomers here ?
16:05:59 <dtometzki> please say hello and give us a short intro
16:07:21 <dtometzki> ok no one , let´s go to the next topic
16:07:32 <dtometzki> #topic Next chair
16:07:32 <dtometzki> #info magic eight ball says:
16:07:32 <dtometzki> #info chair 2022-02-10 - dtometzki
16:07:32 <dtometzki> #info chair 2022-02-17 - mkonecny
16:07:32 <dtometzki> #info chair 2022-02-24 -petebuffon
16:08:08 <dtometzki> Anyone for 2022-03-03 ?
16:08:48 <dtometzki> come on :-)
16:09:01 <mobrien> If we don't have a volunteer I think 2 weeks ahead is good
16:09:17 <dtometzki> ok then let us go forward
16:09:26 <dtometzki> #topic announcements and information
16:09:26 <dtometzki> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:09:26 <dtometzki> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:09:26 <dtometzki> #info If your team wants support from the Fedora Program Management Team, file an isssue: https://pagure.io/fedora-pgm/pgm_team/issues?template=support_request
16:09:28 <dtometzki> #info matrix sig is forming, see https://discussion.fedoraproject.org/t/bi-weekly-meeting-for-matrix-sig-or-matrix-team/35947/3 if interested
16:09:31 <dtometzki> #info Mass rebuild for F36 finished
16:09:41 <dtometzki> any new news to add ?
16:10:08 <dtometzki> i think the mass rebuild we can remove or ?
16:10:16 <nirik> #info branched (f36) composes running fine
16:10:18 <nirik> dtometzki: yep.
16:10:31 <nirik> #info rawhide composes (f37) waiting on signing to finish to resume.
16:11:23 <dtometzki> thanks nirik
16:11:53 <mkonecny> #info API breaking change in bugzilla will happen 28th February
16:12:11 <nirik> yeah, we really need to get going on that, not much time. ;(
16:12:18 <nirik> #info mass update/reboot next week
16:12:22 <dtometzki> yeah
16:12:31 <nirik> #info Red Hat recharge day next friday
16:13:02 <dtometzki> thanks
16:13:18 <dtometzki> #topic Oncall
16:13:18 <dtometzki> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:13:18 <dtometzki> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
16:13:18 <dtometzki> ## .oncalltakeeu .oncalltakeus
16:13:18 <dtometzki> #info nirik on call from 2022-01-27 to 2022-02-03
16:13:19 <dtometzki> #info mobrien on call from 2022-02-04 to 2022-02-10
16:13:21 <dtometzki> #info dtometzki on call from 2022-02-11 to 2022-02-17
16:13:33 <mkonecny> @nirik I already prepared the-new-hotness for it, but need new API key
16:13:36 <mobrien> oh I need to send the mail on those reboots..
16:14:16 <dtometzki> Anyone for the week from 2022-02-18 ?
16:15:20 <mkonecny> I can take it
16:16:09 <dtometzki> #info mkonecny on call from 2022-02-18 to 2022-02-24
16:16:23 <dtometzki> great mkonecny thanks
16:17:18 <dtometzki> #info Summary of last week: (from current oncall )
16:17:27 <mobrien> I had a couple of pings
16:17:50 <dtometzki> a short summary =
16:17:52 <dtometzki> ?
16:17:55 <mobrien> Friday night there was an issue with mirrorlist which nirik fixed
16:18:11 <mobrien> Tuesday morning pagure was down, a httpd reload fixed it
16:18:38 <mobrien> There was also one earlier today about dns but was answered before I could respond
16:18:44 <mobrien> self solved
16:18:51 <dtometzki> yes
16:19:09 <dtometzki> ok
16:19:20 <mkonecny> I noticed that the pagure prod instance is not in nagios
16:20:04 <nirik> it actually is... it's just called pagure02.fedoraproject.org
16:20:24 <nirik> but we aren't monitoring any specific pages or anything, just the normal disk space/etc...
16:21:06 <mkonecny> I tried to filter services by name in nagios and it only found stg instance
16:21:53 <dtometzki> its a good point to to the next topic
16:21:54 <nirik> https://nagios.fedoraproject.org/nagios/cgi-bin//status.cgi?navbarsearch=1&host=pagure02
16:22:05 <dtometzki> #topic Monitoring discussion [nirik]
16:22:05 <dtometzki> #info https://nagios.fedoraproject.org/nagios
16:22:05 <dtometzki> #info Go over existing out items and fix
16:22:32 <mkonecny> nirik:  Strange, not sure why I didn't found it :-/
16:23:01 <nirik> so, it's pretty much standard... the usual few arm boxes down, and misc other things... nothing too out of the ordinary.
16:23:13 <mkonecny> So if you are searching for pagure, it doesn't find pagure02 :-D
16:23:22 <nirik> I did make some changes to try and reduce alerts recently, might keep doing that... the alerts about low swap are kind of useless.
16:23:33 <nirik> mkonecny: nice. ;(
16:24:02 <mkonecny> For next time I need to search for pagure* :-)
16:24:04 <nirik> we have some certs that are expiring soon... but I had to switch back to an old cert for a rawhide compose bug. ;(
16:24:36 <nirik> mobrien: SSL WARNING - Certificate 'apps.ocp.fedoraproject.org' expires in 20 day(s) (2022-03-02 23:26 +0000/GMT).  needs renewing
16:25:08 <nirik> othewise pretty much as usual.... we cna move on unless folks have any questions.
16:25:26 <dtometzki> any questions ?
16:25:27 <mobrien> I will update that cert
16:26:03 <dtometzki> For today we have no learning topic
16:26:28 <dtometzki> next week docs pipline
16:26:38 <nirik> cool.
16:27:27 <mkonecny> We can do backlog refinement then
16:27:32 <dtometzki> yes
16:27:35 * nirik nods
16:27:35 <dtometzki> #topic Fedora Infra backlog refinement
16:27:35 <dtometzki> #info Refine oldest tickets on https://pagure.io/fedora-infrastructure/issues
16:28:04 <mkonecny> .ticket 9691
16:28:05 <zodbot> mkonecny: Issue #9691: STG: Move vmguests from vmhost-x86-01/02/03 to vmhost-x86-11/12 - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9691
16:28:31 <nirik> still needs doing.... just haven't had a chance due to so many other things going on.
16:28:47 * nirik can update the ticket
16:29:00 <nirik> perhaps we could do this next week when we do stg updates/reboots.
16:29:58 <mobrien> Oh yes, I went to do that before christmas but ran into firewall issues
16:30:01 <mkonecny> Could phsmoura do this?
16:30:20 <mobrien> They have since been fixed but I haven't had time to get back to it
16:30:40 <dtometzki> .tcket 9692
16:31:03 <dtometzki> is in the same area
16:31:22 <nirik> mkonecny: it needs perms and following the sop, but otherwise sure.
16:32:12 * nirik waits for that ticket... hello pagure?
16:32:12 <mkonecny> He can at least shadow you :-)
16:32:28 <nirik> definitely.
16:32:32 <mkonecny> .ticket 5478
16:32:33 <zodbot> mkonecny: Issue #5478: Establishing a new FAS group for Python-SIG - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/5478
16:32:48 <nirik> oh yeah, I never got back to this. It did get unblocked I think.
16:33:27 <mkonecny> It seems like everything was solved, it was just forgotten
16:33:40 <nirik> so far. ;) But there's blockers on many other things.
16:33:57 <nirik> anyhow, I will try and just do this one too... it's the oldest open ticket we have I think.
16:34:06 <phsmoura> mkonecny: sure, Id like to work on something new and learn more about fedora infra. Just need to talk to you guys what I should prioritize cause I started working in another issue and still stuck on it and I yes need perms
16:34:07 <nirik> 5 years. sheesh
16:35:16 <mkonecny> It brings memories :-D
16:35:31 <dtometzki> mkonecny, which order do you use in the list ?
16:35:59 <mkonecny> Last modified date 1->9
16:36:23 <mkonecny> It should be described in the agenda
16:36:55 <nirik> 9892 is next...
16:37:08 <mkonecny> nirik:  Do you want to comment on 5478?
16:37:26 <nirik> yeah, I will do that.
16:37:48 <mkonecny> Ok
16:37:51 <mkonecny> .ticket 9892
16:37:52 <zodbot> mkonecny: Issue #9892: Centralise and update Fedora infra and releng SOPs - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9892
16:38:26 <mkonecny> Infrastructure documentation is already moved to docs.fedoraproject.org
16:38:33 <nirik> we did move the infra stuff, but releng isn't right?
16:38:36 <mkonecny> RelEng documentation is being migrated
16:38:51 <mkonecny> @pbokoc is working on that
16:38:55 <mkonecny> I will update the ticket
16:39:31 <nirik> ah, ok. Are mboddu and jednorozec aware? :)
16:40:33 <mkonecny> We were talking about this on planning meeting
16:40:55 <mkonecny> At least jednorozec should be aware
16:41:21 <mboddu> Yeah, I heard about it, but jednorozec was more involved than me in that
16:41:27 <nirik> ok. In any case I think it needs doing... ;)
16:41:56 <jednorozec> I need to ping Petr
16:42:14 <jednorozec> because there was bunch of updates to the SOPs in last few weeks
16:42:41 <mkonecny> jednorozec:  I pinged him around this two days ago and he replied that he is working on it
16:42:54 <mkonecny> s/around/about/
16:43:39 <mkonecny> Anything else regarding this ticket?
16:43:40 <jednorozec> mkonecny, that is fine, but we want to dropt some sops and update others
16:44:11 <mkonecny> He will probably ping you as well
16:44:30 <mkonecny> So next ticket
16:44:39 <mkonecny> .ticket 8213
16:44:40 <zodbot> mkonecny: Issue #8213: fedmsg -> fedora-messaging migration tracker - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8213
16:45:04 <nirik> datagrepper is done now right?
16:45:27 <mkonecny> Let me check
16:45:38 <nirik> but all the rest in the last comment are still to be done. ;(
16:46:49 <mkonecny> It seems like the badges-backend is done as well
16:47:19 <nirik> oh? when was that done?
16:47:21 <mkonecny> Hm, the datagrepper still has fedmsg/base in playbook
16:47:54 <mkonecny> Not sure, it's just not showing on the grep in playbooks/group
16:48:38 <nirik> yeah, I guess datagrepper is still fedmsg. I thought it was changed, but I guess that was just the db...
16:48:47 <mobrien> Datagrepper is in openshift now so that playbook may be out of date
16:49:02 <nirik> and badges-backend01 is still definitely fedmsg
16:49:13 <mkonecny> Ok, badges-backend is still there, it's just not on the top
16:49:14 <nirik> mobrien: oh right, we should nuke it then.
16:49:46 <mkonecny> I see resultsdb as well, which is not on the list
16:50:00 <mobrien> We might want to confim with Aurélien B just to be 100%
16:50:10 <nirik> yeah, thats moving to openshift really soon I hope.
16:50:14 <mkonecny> I will reply to the ticket then
16:50:16 <nirik> mobrien: yeah.
16:50:56 <mkonecny> We skipped resultsdb on purpose on the list?
16:51:24 <nirik> well at the time I think it wasn't clear if qa was going to keep running it or not...
16:51:27 <mobrien> datagrepper servers are commented out in the ansible inventory too
16:51:53 <mkonecny> I see that it was marked as done in the past
16:51:56 <nirik> leor is working on the openshift version...
16:52:36 <nirik> ah it is using fedora-messaging
16:52:42 <nirik> even the vm one thats live now.
16:53:02 <nirik> so, resultsdb doesn't need to be on the list.
16:53:25 <mkonecny> It has the fedmsg/base which just confused me
16:53:47 <nirik> it shouldn't need it...
16:54:01 <mkonecny> I see that it has both
16:54:27 <nirik> it's only running fedora-messaging.
16:54:47 <mkonecny> So it's just a leftover in the playbook
16:54:54 <nirik> yeah.
16:55:13 <mkonecny> I see that the datagrepper has rabbit/user role and fedmsg/base as well
16:55:59 <mkonecny> Commented on the ticket
16:56:19 <abompard> Hey folks!
16:56:30 <nirik> huh, where do you see it for datagrepper?
16:56:33 <abompard> Just finished a meeting, I can answer questions if needed
16:56:36 <mkonecny> I would say that we can go to open floor now
16:56:54 <nirik> abompard: datagrepper is fedora-messaging now right?
16:56:55 <dtometzki> yep for the last 4 minutes
16:57:03 <abompard> nirik: it is
16:57:09 <dtometzki> #topic Open Floor
16:57:12 <abompard> as well as datanommer
16:57:12 <mkonecny> nirik:  In playbooks/groups/datagrepper.yml
16:57:32 <mkonecny> Let me update the comment
16:57:35 <abompard> playbooks/groups/datagrepper is probably the old one, the new one is running in openshift
16:57:44 <abompard> as well as datanommer
16:58:00 <nirik> yeah, thats the old vm one, we need to clean it up.
16:59:16 <mkonecny> I updated the ticket
16:59:41 <mkonecny> We still have 16 services to migrate
17:00:10 <nirik> :(
17:00:28 <mkonecny> And I'm wondering why FMN isn't on the list
17:00:42 <nirik> it is... its just called notifs there.
17:01:27 <dtometzki> ok we have now 6:00 o´clock and i will close the meeting shortly. many thanks for joining
17:01:53 <mkonecny> This is why I'm struggling to find it each time :-D
17:02:11 <mkonecny> dtometzki: Thanks for running this :-)
17:02:17 <nirik> yeah, no idea why it was called that... ;)
17:02:22 <dtometzki> #endmeeting