2025-08-05 19:00:05 <@james:fedora.im> !startmeeting Fedora Infrastructure Ops Daily Standup Meeting

2025-08-05 19:00:07 <@meetbot:fedora.im> Meeting started at 2025-08-05 19:00:05 UTC
2025-08-05 19:00:07 <@meetbot:fedora.im> The Meeting name is 'Fedora Infrastructure Ops Daily Standup Meeting'
2025-08-05 19:00:15 <@james:fedora.im> !chair nirik  phsmoura  @james:fedora.im
2025-08-05 19:00:19 <@james:fedora.im> !info invitees Steve 
2025-08-05 19:00:24 <@nirik:matrix.scrye.com> morning
2025-08-05 19:00:24 <@james:fedora.im> !meetingname fedora_infrastructure_ops_daily_standup_meeting
2025-08-05 19:00:24 <@james:fedora.im> !info Meeting is 30 minutes MAX. At the end of 30, its stops
2025-08-05 19:00:24 <@james:fedora.im> !info Agenda is at https://board.net/p/fedora-infra-daily
2025-08-05 19:00:24 <@james:fedora.im> !info Reminder: Speak up if you want to work on a ticket!
2025-08-05 19:00:25 <@meetbot:fedora.im> The Meeting Name is now fedora_infrastructure_ops_daily_standup_meeting
2025-08-05 19:00:28 <@james:fedora.im> Hey
2025-08-05 19:02:09 <@james:fedora.im> !topic Planning, Upcoming work and Open floor
2025-08-05 19:02:18 <@james:fedora.im> No new tickets ... upgrades later today.
2025-08-05 19:02:21 <@smoliicek:fedora.im> hey everyone
2025-08-05 19:02:30 <@smoliicek:fedora.im> sorry for being late
2025-08-05 19:02:52 <@nirik:matrix.scrye.com> So I have been trying to catch up. I think I somewhat am...
2025-08-05 19:02:55 <@nirik:matrix.scrye.com> hey Vit Smolik
2025-08-05 19:03:50 <@nirik:matrix.scrye.com> James: let me know what things you would like me to help with on updates...
2025-08-05 19:05:03 <@james:fedora.im> One thing I did mean to ask ... what is the playbook you run to run the updates for all prod. hosts at the start of the outage.
2025-08-05 19:06:13 <@james:fedora.im> Vit Smolik: Hey, again ... it's common to be a couple of minutes late, so no big deal ;)
2025-08-05 19:06:19 <@nirik:matrix.scrye.com> No playbook. I do it with just a ansible command... something like: 'ansible -m shell -a 'dnf --refresh -y update; rkhunter --propupd' all:\!copr*:\!openqa\*:\!ocp\*:\!worker\*'
2025-08-05 19:06:38 <@nirik:matrix.scrye.com> it takes a while. especially if there's a selinux policy update.
2025-08-05 19:06:50 <@james:fedora.im> Ahh, I wondered if it was something like that.
2025-08-05 19:07:12 <@nirik:matrix.scrye.com> we could make a playbook for it I guess... should be pretty simple
2025-08-05 19:08:08 <@james:fedora.im> Will probably stick it in a comment or something at least ... I did think about try to tweak the update playbook so someone from sysadmin main can do more than a single host at once.
2025-08-05 19:08:49 <@james:fedora.im> Not that I want to run vhost_update_reboot all ... but doing a couple at once might be nice.
2025-08-05 19:09:01 <@nirik:matrix.scrye.com> well, that can be dangerous. If host A has something that needs a guest on host B and you reboot them both... or host A has a ocp control plain vm and host B does too... taking 2/3 of them down will make the cluster very unhappy.
2025-08-05 19:09:31 <@james:fedora.im> Ahh, yeh, fair enough. Better to be safe and slow.
2025-08-05 19:10:28 <@nirik:matrix.scrye.com> on that script you have been making... I wonder if it might make sense to run it in an automated way... but perhaps not...
2025-08-05 19:10:59 <@nirik:matrix.scrye.com> ie, like the vmdiff cron that runs every hour... (althought this could run every day or something) and makes a /var/log/ file you can get diffs of or see uptimes...
2025-08-05 19:11:04 <@nirik:matrix.scrye.com> but might be overkill.
2025-08-05 19:11:29 <@james:fedora.im> It's already built to do that ... just no cron job atm.
2025-08-05 19:11:34 <@james:fedora.im> Great minds etc. ;)
2025-08-05 19:11:45 <@nirik:matrix.scrye.com> I think zabbix might actually keep track of some of this.
2025-08-05 19:12:59 <@james:fedora.im> Yeh, everything is somewhere ... but it's easy to see now. I'd been looking at something that make all the ansible facts easier to view/query, but the only thing I saw externally is ansible-cmdb ... and that's python2 only.
2025-08-05 19:13:38 <@nirik:matrix.scrye.com> there was some REST thing, can't recall the name
2025-08-05 19:13:55 <@nirik:matrix.scrye.com> I need to look around in zabbix more.
2025-08-05 19:14:03 <@nirik:matrix.scrye.com> (once Gwmngilfen gets it all working)
2025-08-05 19:14:32 <@james:fedora.im> yeh, it has a lot of data ... if it was easy to query from somewhere that'd be great.
2025-08-05 19:14:46 <@nirik:matrix.scrye.com> it has a API, but no idea how easy it is
2025-08-05 19:15:40 <@james:fedora.im> After spending time fighting with jq on ansible fact dumps ... I'm not optimistic, but would be happy to be wrong.
2025-08-05 19:17:13 <@james:fedora.im> Anyway ... unless someone has something ... I'll end early and we'll have more time to get a hot beverage etc. before 21:00 UTC (~1h45m)
2025-08-05 19:17:48 <@nirik:matrix.scrye.com> Not sure I have much else...
2025-08-05 19:18:48 <@smoliicek:fedora.im> i wanted to ask if there is something i could help with
2025-08-05 19:18:53 <@smoliicek:fedora.im> i know the answear is prob no, but still worth a shot :)
2025-08-05 19:19:28 <@nirik:matrix.scrye.com> well, possibly? :) what sort of things are you interested in?
2025-08-05 19:19:54 <@nirik:matrix.scrye.com> I know thats a hard thing to quantify...
2025-08-05 19:19:56 <@james:fedora.im> For the updates+reboots ... you need more privs. than you have now, but in the future it's possible.
2025-08-05 19:20:07 <@smoliicek:fedora.im> yea, i know that
2025-08-05 19:20:17 <@smoliicek:fedora.im> i could try and write the playbook for the updates tho
2025-08-05 19:20:25 <@smoliicek:fedora.im> for this
2025-08-05 19:21:38 <@nirik:matrix.scrye.com> sure if you like...
2025-08-05 19:22:17 <@nirik:matrix.scrye.com> take a look at the manual/vhost_update playbooks to see how they work... and then a pr proposing a new 'update_all' or something?
2025-08-05 19:22:55 <@smoliicek:fedora.im> and "update_all" should also be in manual/ ??
2025-08-05 19:23:13 <@smoliicek:fedora.im> and "update_all" should also be in manual/ ?
2025-08-05 19:23:14 <@nirik:matrix.scrye.com> yep. It would be a manually run playbook...
2025-08-05 19:23:49 <@nirik:matrix.scrye.com> also, if you like ansible cleanups are always welcome... there's probibly places we still look for distribution_version 6 or 7 (rhel6 ir 7) that should be removed...
2025-08-05 19:24:09 <@smoliicek:fedora.im> okay, ill fork it on pagure and see how it goes
2025-08-05 19:24:24 <@smoliicek:fedora.im> alright, i can try that after
2025-08-05 19:24:39 <@nirik:matrix.scrye.com> Thanks!
2025-08-05 19:25:20 <@james:fedora.im> I have it on my TODO to get most (all?) of our when clauses to be simple variables ... so any help there would also be awesome.
2025-08-05 19:25:40 <@smoliicek:fedora.im> can i hit one of you up if i need help with something and i can't figure it out myself?
2025-08-05 19:26:01 <@james:fedora.im> Yeh, or generally ping the room in infra.
2025-08-05 19:26:02 <@nirik:matrix.scrye.com> we need to have a larger discussion later this year about ansible... if we want to move to some better frontend we may have to redo a lot of our playbooks.
2025-08-05 19:26:20 <@nirik:matrix.scrye.com> sure. yeah, asking in admin or noc someone is usually around.
2025-08-05 19:26:38 <@smoliicek:fedora.im> alright
2025-08-05 19:26:42 <@smoliicek:fedora.im> thanks guys :)
2025-08-05 19:27:00 <@james:fedora.im> Okay, see you in a bit for upgrades.
2025-08-05 19:27:02 <@james:fedora.im> !endmeeting