2025-01-16 17:00:08 <@nirik:matrix.scrye.com> !startmeeting Infrastructure (2025-01-16)
2025-01-16 17:00:10 <@meetbot:fedora.im> Meeting started at 2025-01-16 17:00:08 UTC
2025-01-16 17:00:11 <@meetbot:fedora.im> The Meeting name is 'Infrastructure (2025-01-16)'
2025-01-16 17:00:18 <@nirik:matrix.scrye.com> !meetingname infrastructure
2025-01-16 17:00:18 <@nirik:matrix.scrye.com> !chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
2025-01-16 17:00:18 <@nirik:matrix.scrye.com> !info Agenda is at: https://board.net/p/fedora-infra
2025-01-16 17:00:18 <@nirik:matrix.scrye.com> !info About our team: https://docs.fedoraproject.org/en-US/cle/
2025-01-16 17:00:18 <@nirik:matrix.scrye.com> !info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
2025-01-16 17:00:18 <@nirik:matrix.scrye.com> !topic ahoy
2025-01-16 17:00:20 <@meetbot:fedora.im> The Meeting Name is now infrastructure
2025-01-16 17:00:58 <@humaton:fedora.im> !hi
2025-01-16 17:00:59 <@zodbot:fedora.im> Tomáš Hrčka (humaton) - he / him / his
2025-01-16 17:01:16 <@nirik:matrix.scrye.com> morning all.
2025-01-16 17:01:59 <@humaton:fedora.im> is it jsut the 2 of us :D
2025-01-16 17:02:13 <@humaton:fedora.im> is it just the 2 of us :D
2025-01-16 17:02:19 <@nirik:matrix.scrye.com> So far... I guess we can wait a few more and see who else makes it. ;)
2025-01-16 17:03:01 <@humaton:fedora.im> how is the winter over there?
2025-01-16 17:03:28 <@humaton:fedora.im> we have morning fog, during day snow and freeze over the night
2025-01-16 17:03:46 <@nirik:matrix.scrye.com> odd. No snow at all (just one day it snowed, but never stuck)... but yeah, last week we have had 'freezing fog'...
2025-01-16 17:04:14 <@humaton:fedora.im> oh yeah that happens more this winter than I have ever seen
2025-01-16 17:04:42 <@humaton:fedora.im> Fog than sudden -3C and it sticks to everything 
2025-01-16 17:04:43 <@nirik:matrix.scrye.com> not much rain lately either, which is weird. winter is out wet time here. :)
2025-01-16 17:07:31 <@nirik:matrix.scrye.com> well, shall we go ahead with the meeting? or just cancel today since it's only us two. ;)
2025-01-16 17:08:12 <@humaton:fedora.im> well I have just one thing to ponder, and that is how are we going to deploy forgejo
2025-01-16 17:08:43 <@humaton:fedora.im> we have AWX template from rocky guys and kubernetes helm from upstream
2025-01-16 17:09:18 <@nirik:matrix.scrye.com> well, I havent had a chance to look much, but I wonder if we couldn't just deploy like we do all our other openshift apps?
2025-01-16 17:09:33 <@nirik:matrix.scrye.com> via ansible, etc...
2025-01-16 17:09:41 <@humaton:fedora.im> well it will be the most complex openshift app we have
2025-01-16 17:09:41 <@nirik:matrix.scrye.com> but again, I don't know how complex it is.
2025-01-16 17:09:55 <@humaton:fedora.im> So I would say maybe we want some more sophisticated deployment
2025-01-16 17:10:35 <@humaton:fedora.im> I think we should aim for HA deployment https://code.forgejo.org/forgejo-helm/forgejo-helm/src/branch/main/docs/ha-setup.md
2025-01-16 17:10:42 <@james:fedora.im> I'm here, but slow as usual
2025-01-16 17:10:44 <@nirik:matrix.scrye.com> !topic forejo deployment
2025-01-16 17:10:49 <@james:fedora.im> I think Carl is still on vacation
2025-01-16 17:11:33 <@nirik:matrix.scrye.com> yeah, I would love a HA setup allowing us to update/etc without downtime...
2025-01-16 17:11:37 <@humaton:fedora.im> Running it all in openshift including DB will drop sing point of failure in the PGSQL 
2025-01-16 17:12:10 <@humaton:fedora.im> If you have time look at the helm thing. I dont want to use it but its nice inspiration
2025-01-16 17:12:26 <@nirik:matrix.scrye.com> I'll note there they say to use a cloud db serice. ;(
2025-01-16 17:12:27 <@humaton:fedora.im> having ansible operator doing that would be lovely...
2025-01-16 17:12:37 <@nirik:matrix.scrye.com> yeah, that would be nice.
2025-01-16 17:13:11 <@nirik:matrix.scrye.com> we could also do something like we did when we deployed osbs... have a thing in our ansible that checks out and runs the other complex thing (helm, whatever)
2025-01-16 17:14:17 <@humaton:fedora.im> I dont want to introduce yet another deployment method so avoiding the helm thing 
2025-01-16 17:14:29 <@nirik:matrix.scrye.com> we may have to try several things and see what works best. I don't think we should be afraid to try and scrap something that doesn't work out
2025-01-16 17:14:59 <@humaton:fedora.im> the helm is extra easy to run harder to modify
2025-01-16 17:15:27 <@humaton:fedora.im> I played with it in comushift and it was as easy as helm run ...
2025-01-16 17:15:54 <@humaton:fedora.im> But we will need our images, I would like to avoid using the upstream images because they use debian 
2025-01-16 17:16:28 <@humaton:fedora.im> so plugiing CI and image build together, pushing things to quay/maybe Konflux?/
2025-01-16 17:16:43 <@humaton:fedora.im> and deploying using operator
2025-01-16 17:17:31 <@james:fedora.im> When you say modify ... do you mean configuration or upgrades, or something more like "if you want to not use their specific version of postgresql you will have problems"
2025-01-16 17:17:52 <@Zlopez:matrix.org> !hi
2025-01-16 17:17:53 <@zodbot:fedora.im> Michal Konecny (zlopez)
2025-01-16 17:17:54 <@humaton:fedora.im> James Antill: the later
2025-01-16 17:18:33 <@humaton:fedora.im> configuration an UI modifications are done during imagebuild
2025-01-16 17:18:43 <@nirik:matrix.scrye.com> image building pipeline is one part, then deployment/config. If we build images right perhaps the deployment won't need that much complexity ?
2025-01-16 17:19:30 <@humaton:fedora.im> oh yeah image definitions and builds are the first thing we need to do
2025-01-16 17:20:47 <@nirik:matrix.scrye.com> there's some packaging attempts on copr I see: https://copr.fedorainfracloud.org/coprs/fulltext/?fulltext=forgejo
2025-01-16 17:21:26 <@nirik:matrix.scrye.com> oh, duh, it's already in
2025-01-16 17:21:44 <@nirik:matrix.scrye.com> oh, no, thats just the client lib
2025-01-16 17:21:47 <@nirik:matrix.scrye.com> https://bugzilla.redhat.com/show_bug.cgi?id=2268288
2025-01-16 17:22:16 <@nirik:matrix.scrye.com> anyhow. yeah, making a rhel/fedora image that has everything and works is probibly a good first step and can be done by anyone locally
2025-01-16 17:22:58 <@Zlopez:matrix.org> That doesn't sound too difficult
2025-01-16 17:23:15 <@humaton:fedora.im> well having the app is just small step
2025-01-16 17:23:27 <@humaton:fedora.im> there is redis there is DB there is minio
2025-01-16 17:23:43 <@humaton:fedora.im> btw iff you are unfamiliar with minio I recomend checking out
2025-01-16 17:23:57 <@humaton:fedora.im> its local s3 compatible free sw
2025-01-16 17:24:14 <@Zlopez:matrix.org> How much could we use from current test deployment?
2025-01-16 17:24:22 <@humaton:fedora.im> nothing
2025-01-16 17:24:30 <@humaton:fedora.im> the test deloyemnt should be nuked already 
2025-01-16 17:24:47 <@humaton:fedora.im> and it was the default from upstream without any real production features 
2025-01-16 17:24:52 <@nirik:matrix.scrye.com> FWIW, openshift has s3 compatible storage also...
2025-01-16 17:24:59 <@humaton:fedora.im> kind of devel deployment for hacking 
2025-01-16 17:25:10 <@Zlopez:matrix.org> OK
2025-01-16 17:25:45 <@humaton:fedora.im> oh minio is in openshift
2025-01-16 17:28:04 <@nirik:matrix.scrye.com> still a lot to investigate. ;)
2025-01-16 17:28:06 <@humaton:fedora.im> SO I am thinking about creating new project under fedora-infra group in pagure to track the deployment/migration
2025-01-16 17:28:47 <@Zlopez:matrix.org> +1
2025-01-16 17:29:01 <@humaton:fedora.im> So far I have few tickets in mind: 1 OCI artifacts definitions, Explore Konflux to build images, Make the operator
2025-01-16 17:29:13 <@humaton:fedora.im> So far I have few tickets in mind: OCI artifacts definitions, Explore Konflux to build images, Make the operator
2025-01-16 17:29:40 <@humaton:fedora.im> there will be more granularity in it but those topics are the first totackle
2025-01-16 17:29:42 <@nirik:matrix.scrye.com> sounds like a good start.
2025-01-16 17:29:49 <@humaton:fedora.im> there will be more granularity in it but those topics are the first to tackle
2025-01-16 17:30:35 <@humaton:fedora.im> That is all from the top of my head
2025-01-16 17:31:19 <@nirik:matrix.scrye.com> should we go thru the rest of the meeting now that more folks are here?
2025-01-16 17:31:53 <@nirik:matrix.scrye.com> lets go ahead then...
2025-01-16 17:31:56 <@nirik:matrix.scrye.com> !info chair 2025-01-30 - ???
2025-01-16 17:31:56 <@nirik:matrix.scrye.com> !topic Next chair
2025-01-16 17:31:56 <@nirik:matrix.scrye.com> !info chair 2025-01-23 - Yaash
2025-01-16 17:31:56 <@nirik:matrix.scrye.com> !info magic eight ball says:
2025-01-16 17:32:02 <@nirik:matrix.scrye.com> anyone want chair for the 30th?
2025-01-16 17:32:21 <@humaton:fedora.im> I will be on my way to fosdem
2025-01-16 17:32:58 <@nirik:matrix.scrye.com> I guess we can decide next week anyhow. ;)
2025-01-16 17:33:03 <@nirik:matrix.scrye.com> !topic announcements and information
2025-01-16 17:33:03 <@nirik:matrix.scrye.com> !info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2025-01-16 17:33:03 <@nirik:matrix.scrye.com> !info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2025-01-16 17:33:07 <@nirik:matrix.scrye.com> any other announcements?
2025-01-16 17:34:47 <@Zlopez:matrix.org> !info release-monitoring.org v2.0.0 is now available
2025-01-16 17:34:56 <@nirik:matrix.scrye.com> ok, moving along then...
2025-01-16 17:35:00 <@nirik:matrix.scrye.com> !info https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team
2025-01-16 17:35:00 <@nirik:matrix.scrye.com> !info ??? is on call from 2025-01-23 to 2025-01-30
2025-01-16 17:35:00 <@nirik:matrix.scrye.com> !info zlopez is on call from 2025-01-16 to 2025-01-23
2025-01-16 17:35:00 <@nirik:matrix.scrye.com> !topic Oncall
2025-01-16 17:35:18 <@Zlopez:matrix.org> Let me switch the oncall
2025-01-16 17:35:32 <@nirik:matrix.scrye.com> anyone want oncall that second week? If not, we can decide next week
2025-01-16 17:36:18 <@Zlopez:matrix.org> !oncall
2025-01-16 17:36:19 <@zodbot:fedora.im> ● @Zlopez:matrix.org (zlopez) Current Time for them: 18:36 (Europe/Prague)
2025-01-16 17:36:19 <@zodbot:fedora.im> If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)
2025-01-16 17:36:19 <@zodbot:fedora.im> The following people are oncall:
2025-01-16 17:36:19 <@zodbot:fedora.im> 
2025-01-16 17:36:25 <@Zlopez:matrix.org> Set
2025-01-16 17:36:26 <@nirik:matrix.scrye.com> cool
2025-01-16 17:36:30 <@nirik:matrix.scrye.com> !info Go over existing items and fix them
2025-01-16 17:36:30 <@nirik:matrix.scrye.com> !topic Monitoring discussion [nirik]
2025-01-16 17:36:30 <@nirik:matrix.scrye.com> !info https://nagios.fedoraproject.org/nagios 
2025-01-16 17:37:09 <@nirik:matrix.scrye.com> we have a builder I need to reinstall, the logdetective02 still showing up and...
2025-01-16 17:37:23 <@nirik:matrix.scrye.com> there's a rabbitmq queue that I don't think we use anymore, but it's alerting on it:
2025-01-16 17:37:27 <@nirik:matrix.scrye.com> RABBITMQ_QUEUE CRITICAL - Not Found : http://localhost:15672/api/queues/%2Fpubsub/toddlers-distgit-commit-processor
2025-01-16 17:37:48 <@Zlopez:matrix.org> That one is no longer running
2025-01-16 17:37:49 <@nirik:matrix.scrye.com> I think that moved to a poddlers queue or something? but we need to confirm and remove monitoring for it.
2025-01-16 17:37:58 <@nirik:matrix.scrye.com> yeah, it's alerting because it's not there...
2025-01-16 17:38:25 <@Zlopez:matrix.org> Do we have ticket for that?
2025-01-16 17:39:01 <@nirik:matrix.scrye.com> nope, it happened over the break I think. I can remove the monitoring, but I just wanted to make sure it was right that the queue no longer exists.
2025-01-16 17:39:38 <@nirik:matrix.scrye.com> so I can do that then if you are sure it's supposed to be gone
2025-01-16 17:39:41 <@Zlopez:matrix.org> Everything should be now moved to poddlers
2025-01-16 17:40:01 <@nirik:matrix.scrye.com> yeah, I wasn't sure why the queue name changed tho... but fine. ;)
2025-01-16 17:40:02 <@Zlopez:matrix.org> Better confirm with @abompard:fedora.im He did the move
2025-01-16 17:40:39 <@nirik:matrix.scrye.com> The last alert is disk space on bastion01. This is caused by https://pagure.io/fedora-infrastructure/issue/12361
2025-01-16 17:41:20 <@nirik:matrix.scrye.com> anything else on monitoring?
2025-01-16 17:43:08 <@nirik:matrix.scrye.com> ok, shall we just go to open floor?
2025-01-16 17:43:12 <@nirik:matrix.scrye.com> or triage some old tickets?
2025-01-16 17:43:24 <@Zlopez:matrix.org> I would go with open floor
2025-01-16 17:43:46 <@nirik:matrix.scrye.com> !topic open floor
2025-01-16 17:43:56 <@nirik:matrix.scrye.com> any other discussion items?
2025-01-16 17:43:57 <@Zlopez:matrix.org> I'm wondering if https://pagure.io/fedora-infrastructure/issue/12361 isn't caused by something that happened during mailman deployment
2025-01-16 17:44:29 <@nirik:matrix.scrye.com> shouldn't be... it's a completely different system
2025-01-16 17:44:41 <@abompard:fedora.im> Yeah you can remove the monitoring, it's for a disabled toddler
2025-01-16 17:45:06 <@nirik:matrix.scrye.com> hey Aurélien B! ok.
2025-01-16 17:45:16 <@abompard:fedora.im> hey :-)
2025-01-16 17:45:23 <@nirik:matrix.scrye.com> and to answer your query from eariler, no, I cannot login to accounts with my email address. ;(
2025-01-16 17:45:29 <@Zlopez:matrix.org> Didn't @nphilipp:fedora.im worked on that one?
2025-01-16 17:45:38 <@nirik:matrix.scrye.com> "Unauthorized: bad credentials."
2025-01-16 17:45:55 <@nirik:matrix.scrye.com> he worked on the ipa/ansible deployment... so perhaps he can see whats happening
2025-01-16 17:46:48 <@nirik:matrix.scrye.com> basically all users are there (even if they cannot login) so postfix sees them as users and the ones without a fedoraproject.org alias just get locally delivered email. ;( (at least from what I could see from a quick glance)
2025-01-16 17:48:02 <@Zlopez:matrix.org> I meant the distgit_commit_processor
2025-01-16 17:48:15 <@Zlopez:matrix.org> I meant the distgit\_commit\_processor
2025-01-16 17:48:27 <@Zlopez:matrix.org> I meant the distgit-commit-processor
2025-01-16 17:48:29 <@nirik:matrix.scrye.com> hum, not sure... I suppose possibly?
2025-01-16 17:48:51 <@Zlopez:matrix.org> I remember it was related to one open ticket
2025-01-16 17:49:29 <@Zlopez:matrix.org> I think this one https://pagure.io/fedora-infrastructure/issue/11641
2025-01-16 17:50:05 <@Zlopez:matrix.org> But I see that last update was 9 months ago https://pagure.io/fedora-infra/toddlers/issue/183
2025-01-16 17:50:14 <@Zlopez:matrix.org> So probably it was forgotten
2025-01-16 17:50:15 <@nirik:matrix.scrye.com> well, I don't see how that would affect this... but yeah
2025-01-16 17:50:28 <@abompard:fedora.im> Ah, right, it works in ipsilon but not in noggin
2025-01-16 17:50:30 <@nirik:matrix.scrye.com> if you could ping him on it that would be great. There's a PR related too.
2025-01-16 17:51:14 <@Zlopez:matrix.org> This is for the rabbitmq queue and disabled toddler, it's not related to the authentication problem
2025-01-16 17:51:17 <@nirik:matrix.scrye.com> https://pagure.io/fedora-infra/ansible/pull-request/2251
2025-01-16 17:51:37 <@nirik:matrix.scrye.com> ah, sorry, mixing things. ;)
2025-01-16 17:51:52 <@Zlopez:matrix.org> I'm probably just causing confusion :-D
2025-01-16 17:52:37 <@Zlopez:matrix.org> Should it work to login with e-mail? I always use username
2025-01-16 17:52:40 <@nirik:matrix.scrye.com> anyhow, we can work on all these out of meeting. ;) anything else for the meeting, or shall we close?
2025-01-16 17:53:00 <@nirik:matrix.scrye.com> it should...we did have it working everywhere with email at one point.
2025-01-16 17:53:24 <@Zlopez:matrix.org> I probably never tried it
2025-01-16 17:53:32 <@nirik:matrix.scrye.com> we have been getting a flow of users saying they cannot login recently due to this... they are using email...
2025-01-16 17:53:51 <@Zlopez:matrix.org> Which mail it's using the fedoraproject.org one?
2025-01-16 17:54:15 <@Zlopez:matrix.org> Or the user e-mail?
2025-01-16 17:55:15 <@nirik:matrix.scrye.com> The email associated with the account.
2025-01-16 17:55:55 <@Zlopez:matrix.org> Doesn't work for me as well
2025-01-16 17:56:44 <@Zlopez:matrix.org> Need to use username instead
2025-01-16 17:56:59 <@nirik:matrix.scrye.com> ok, lets continue over in #noc:fedoraproject.org and #admin:fedoraproject.org then.
2025-01-16 17:57:04 <@nirik:matrix.scrye.com> Thanks for coming everyone
2025-01-16 17:57:10 <@nirik:matrix.scrye.com> !endmeeting