#meeting-3:fedoraproject.org log

<@james:fedora.im>

19:00:05

!startmeeting Fedora Infrastructure Ops Daily Standup Meeting

<@meetbot:fedora.im>

19:00:07

Meeting started at 2025-08-05 19:00:05 UTC

<@meetbot:fedora.im>

19:00:07

The Meeting name is 'Fedora Infrastructure Ops Daily Standup Meeting'

<@james:fedora.im>

19:00:15

!chair nirik phsmoura @james:fedora.im

<@james:fedora.im>

19:00:19

!info invitees Steve

<@nirik:matrix.scrye.com>

19:00:24

morning

<@james:fedora.im>

19:00:24

!meetingname fedora_infrastructure_ops_daily_standup_meeting

<@james:fedora.im>

19:00:24

!info Meeting is 30 minutes MAX. At the end of 30, its stops

<@james:fedora.im>

19:00:24

!info Agenda is at https://board.net/p/fedora-infra-daily

<@james:fedora.im>

19:00:24

!info Reminder: Speak up if you want to work on a ticket!

<@meetbot:fedora.im>

19:00:25

The Meeting Name is now fedora_infrastructure_ops_daily_standup_meeting

<@james:fedora.im>

19:00:28

Hey

<@james:fedora.im>

19:02:09

!topic Planning, Upcoming work and Open floor

<@james:fedora.im>

19:02:18

No new tickets ... upgrades later today.

<@smoliicek:fedora.im>

19:02:21

hey everyone

<@smoliicek:fedora.im>

19:02:30

sorry for being late

<@nirik:matrix.scrye.com>

19:02:52

So I have been trying to catch up. I think I somewhat am...

<@nirik:matrix.scrye.com>

19:02:55

hey Vit Smolik

<@nirik:matrix.scrye.com>

19:03:50

James: let me know what things you would like me to help with on updates...

<@james:fedora.im>

19:05:03

One thing I did mean to ask ... what is the playbook you run to run the updates for all prod. hosts at the start of the outage.

<@james:fedora.im>

19:06:13

Vit Smolik: Hey, again ... it's common to be a couple of minutes late, so no big deal ;)

<@nirik:matrix.scrye.com>

19:06:19

No playbook. I do it with just a ansible command... something like: 'ansible -m shell -a 'dnf --refresh -y update; rkhunter --propupd' all:\!copr*:\!openqa\*:\!ocp\*:\!worker\*'

<@nirik:matrix.scrye.com>

19:06:38

it takes a while. especially if there's a selinux policy update.

<@james:fedora.im>

19:06:50

Ahh, I wondered if it was something like that.

<@nirik:matrix.scrye.com>

19:07:12

we could make a playbook for it I guess... should be pretty simple

<@james:fedora.im>

19:08:08

Will probably stick it in a comment or something at least ... I did think about try to tweak the update playbook so someone from sysadmin main can do more than a single host at once.

<@james:fedora.im>

19:08:49

Not that I want to run vhost_update_reboot all ... but doing a couple at once might be nice.

<@nirik:matrix.scrye.com>

19:09:01

well, that can be dangerous. If host A has something that needs a guest on host B and you reboot them both... or host A has a ocp control plain vm and host B does too... taking 2/3 of them down will make the cluster very unhappy.

<@james:fedora.im>

19:09:31

Ahh, yeh, fair enough. Better to be safe and slow.

<@nirik:matrix.scrye.com>

19:10:28

on that script you have been making... I wonder if it might make sense to run it in an automated way... but perhaps not...

<@nirik:matrix.scrye.com>

19:10:59

ie, like the vmdiff cron that runs every hour... (althought this could run every day or something) and makes a /var/log/ file you can get diffs of or see uptimes...

<@nirik:matrix.scrye.com>

19:11:04

but might be overkill.

<@james:fedora.im>

19:11:29

It's already built to do that ... just no cron job atm.

<@james:fedora.im>

19:11:34

Great minds etc. ;)

<@nirik:matrix.scrye.com>

19:11:45

I think zabbix might actually keep track of some of this.

<@james:fedora.im>

19:12:59

Yeh, everything is somewhere ... but it's easy to see now. I'd been looking at something that make all the ansible facts easier to view/query, but the only thing I saw externally is ansible-cmdb ... and that's python2 only.

<@nirik:matrix.scrye.com>

19:13:38

there was some REST thing, can't recall the name

<@nirik:matrix.scrye.com>

19:13:55

I need to look around in zabbix more.

<@nirik:matrix.scrye.com>

19:14:03

(once Gwmngilfen gets it all working)

<@james:fedora.im>

19:14:32

yeh, it has a lot of data ... if it was easy to query from somewhere that'd be great.

<@nirik:matrix.scrye.com>

19:14:46

it has a API, but no idea how easy it is

<@james:fedora.im>

19:15:40

After spending time fighting with jq on ansible fact dumps ... I'm not optimistic, but would be happy to be wrong.

<@james:fedora.im>

19:17:13

Anyway ... unless someone has something ... I'll end early and we'll have more time to get a hot beverage etc. before 21:00 UTC (~1h45m)

<@nirik:matrix.scrye.com>

19:17:48

Not sure I have much else...

<@smoliicek:fedora.im>

19:18:48

i wanted to ask if there is something i could help with

<@smoliicek:fedora.im>

19:18:53

i know the answear is prob no, but still worth a shot :)

<@nirik:matrix.scrye.com>

19:19:28

well, possibly? :) what sort of things are you interested in?

<@nirik:matrix.scrye.com>

19:19:54

I know thats a hard thing to quantify...

<@james:fedora.im>

19:19:56

For the updates+reboots ... you need more privs. than you have now, but in the future it's possible.

<@smoliicek:fedora.im>

19:20:07

yea, i know that

<@smoliicek:fedora.im>

19:20:17

i could try and write the playbook for the updates tho

<@smoliicek:fedora.im>

19:20:25

for this

<@nirik:matrix.scrye.com>

19:21:38

sure if you like...

<@nirik:matrix.scrye.com>

19:22:17

take a look at the manual/vhost_update playbooks to see how they work... and then a pr proposing a new 'update_all' or something?

<@smoliicek:fedora.im>

19:22:55

and "update_all" should also be in manual/ ??

<@smoliicek:fedora.im>

19:23:13

and "update_all" should also be in manual/ ?

<@nirik:matrix.scrye.com>

19:23:14

yep. It would be a manually run playbook...

<@nirik:matrix.scrye.com>

19:23:49

also, if you like ansible cleanups are always welcome... there's probibly places we still look for distribution_version 6 or 7 (rhel6 ir 7) that should be removed...

<@smoliicek:fedora.im>

19:24:09

okay, ill fork it on pagure and see how it goes

<@smoliicek:fedora.im>

19:24:24

alright, i can try that after

<@nirik:matrix.scrye.com>

19:24:39

Thanks!

<@james:fedora.im>

19:25:20

I have it on my TODO to get most (all?) of our when clauses to be simple variables ... so any help there would also be awesome.

<@smoliicek:fedora.im>

19:25:40

can i hit one of you up if i need help with something and i can't figure it out myself?

<@james:fedora.im>

19:26:01

Yeh, or generally ping the room in infra.

<@nirik:matrix.scrye.com>

19:26:02

we need to have a larger discussion later this year about ansible... if we want to move to some better frontend we may have to redo a lot of our playbooks.

<@nirik:matrix.scrye.com>

19:26:20

sure. yeah, asking in admin or noc someone is usually around.

<@smoliicek:fedora.im>

19:26:38

alright

<@smoliicek:fedora.im>

19:26:42

thanks guys :)

<@james:fedora.im>

19:27:00

Okay, see you in a bit for upgrades.

<@james:fedora.im>

19:27:02

!endmeeting