19:00:24 <nirik> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting
19:00:24 <zodbot> Meeting started Thu Jan 13 19:00:24 2022 UTC.
19:00:24 <zodbot> This meeting is logged and archived in a public location.
19:00:24 <zodbot> The chair is nirik. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
19:00:24 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:24 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
19:00:25 <nirik> #chair mboddu nirik pingou mobrien nb
19:00:25 <zodbot> Current chairs: mboddu mobrien nb nirik pingou
19:00:25 <nirik> #meetingname fedora_infrastructure_ops_daily_standup_meeting
19:00:25 <nirik> #info meeting is 30 minutes MAX. At the end of 30, its stops
19:00:25 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
19:00:25 <nirik> #info agenda is at https://board.net/p/fedora-infra-daily
19:00:27 <nirik> #info reminder: speak up if you want to work on a ticket!
19:00:29 <nirik> #topic Tickets needing review
19:00:31 <nirik> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1
19:01:05 <nirik> .ticket 10474
19:01:06 <zodbot> nirik: Issue #10474: New 'legal' namespace in Fedora GitHub - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10474
19:01:22 <nirik> low/low/ops keep ryan assigned
19:01:46 <nirik> .ticket 10476
19:01:47 <zodbot> nirik: Issue #10476: Planned Outage - Datanommer Migration - 2022-01-17 11:00 UTC - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10476
19:01:57 <mobrien> Quick question. Is there any relation to fas and gitlab groups at the moment?
19:03:05 <nirik> med / med ops added comments
19:03:30 <nirik> mobrien: no. We could do that, but it's not great...
19:03:59 <mobrien> I must send that outage to the lists and have the planned status ready to push, just waiting on a review. Thanks nirik
19:04:01 <nirik> you need a specific group for every permission level... and it makes that match gitlab
19:04:20 <mobrien> That sounds sub optimal....
19:04:34 <nirik> so, if you say wanted to setup websites project like that. you would need 'websites-admins' 'websites-developers' 'websites-contibutors' etc.
19:04:51 <mobrien> Let's leave as is then
19:05:03 <nirik> and then if you added someone from the gitlab interface to some permissions like developer without adding them to the fas group, it would remove them when they logged in
19:05:27 <nirik> so yeah, we thought it best to just manage perms there and let subgroups do their own thing
19:05:47 <nirik> .ticket 10477
19:05:48 <zodbot> nirik: Issue #10477: proxy40 does not have enough memory - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10477
19:05:58 <nirik> low/low/ops I guess?
19:06:39 <mobrien> The machine type will need to be changed and a dns update after as the ip will be lost
19:06:42 <nirik> thats it for infra tickets triage
19:07:01 <nirik> mobrien: yeah. I don't know what time has 8GB mem, but thats all doable.
19:07:35 <mobrien> Should be straightforward. You can assign to me.
19:07:56 <nirik> sure. I think all those proxy3* have 4gb too? not sure tho
19:08:17 <mobrien> I'll have a look through
19:08:22 <nirik> #topic Upcoming work / Planning
19:08:57 <nirik> So, I am trying to finish builder upgrades, but I am waiting on talking to the mainframe folks so I can find out how much disk we really have in our s390x kvm lpar. ;( Hopefully will get that soon...
19:09:20 <mobrien> I did a rebuild of the osbs buildroot image today for f35 and everything built OK. I didn't get time to test do I'll test that tomorrow and push if it works
19:09:21 <nirik> mobrien: you didn't find anything further on container builds failures did you? ;(
19:09:44 <nirik> ah ok. worth a try... it seems like something deeper wrong tho.
19:10:18 <mobrien> I didn't find the root cause but something seems to be up with the the image tags on the registry. It looks as though some are missing
19:10:56 <mobrien> I might try drag David in tomorrow as well if he is available
19:10:57 <nirik> huh, nothing really should have changed there...
19:11:31 <nirik> the only thing that I know of that changed around that time was me reinstalling builders... but it doesn't even run any of that on builders, it's all on the osbs cluster by then.
19:12:19 <nirik> anyhow yeah, perhaps David can see whats going on too if you can catch him.
19:13:06 <nirik> The other current breakage is the IPA certs thing. ;( I dug into that for a few hours yesterday without fixing it... I'll try and look more today.
19:13:24 <nirik> perhaps I can try downgrading some things.
19:14:15 <nirik> #topic Discussion / AOB
19:14:24 <nirik> anything to discuss or bring up ?
19:15:31 <petebuffon> nirik: If you have a couple minutes sometime today I'd like to discuss next steps on https://pagure.io/fedora-infrastructure/issue/9695.
19:16:14 <nirik> ah ok.
19:16:26 <nirik> so, whats left there?
19:16:49 <nirik> a bunch of vm's I guess?
19:16:57 <petebuffon> yeah, a bunch of vms
19:17:40 <petebuffon> I guess my main question is if the vm macs still need to be dumped or if I can use the ansible mac variable?
19:18:05 <nirik> just the variable. don't need to be dumped.
19:18:13 <nirik> it will use the one from ansible facts
19:18:38 <petebuffon> alright so not necessarily the same for all vms
19:18:47 <petebuffon> how do I get the ansible facts output then?
19:19:03 <nirik> no no, you don't need it... use the variable and ansible will fill it in.
19:19:34 <petebuffon> ah okay that makes sense
19:19:44 <petebuffon> alright so it should be pretty straightforward then
19:19:56 <nirik> so take a look at proxy01.iad2.fedoraproject.org for example
19:20:21 <nirik> no wait, that one is wrong
19:21:17 <nirik> so I wonder...
19:21:24 <nb> Jo
19:21:38 <nirik> perhaps we should just define a default network_connections.
19:22:24 <nirik> hey nb! Hope you had a good holidays...
19:22:31 <petebuffon> alright, so we would define that one place and then use it as a variable?
19:22:57 <nirik> if we just define a default one in one place we only need to override it for hosts that need it/non vm's....
19:23:22 <petebuffon> ah
19:23:31 <nirik> no, just define it in group_vars/all and it will affect all hosts except those that override it in host_vars/hostname
19:23:51 <petebuffon> got it
19:24:12 <nirik> so I guess lets try that and clean up any of the vm's that are wrong...
19:24:35 <petebuffon> sounds good, what should I use as a basis for the default?
19:25:50 <nirik> so, looks like the only ones that are vm's that are wrong currently are proxies... I probibly did those before I considered things :)
19:26:31 <petebuffon> okay
19:26:32 <nirik> the one in proxy14 seems reasonably complete.
19:26:43 <nirik> but then there's ipv6 too... hum.
19:27:20 <nirik> I guess the ipv6 vm's will need to override, so we need the proxies the way they are anyhow.
19:27:49 <nirik> so, add to group_vars/all one that has just ipv4 and eth0... and uses {{ ansible_default_ipv4.macaddress }} for mac
19:28:43 <petebuffon> great, so the exceptions then will be vms with an interface other than eth0 and/or ipv6?
19:30:01 <nirik> yep. or hardware ones with bridges
19:30:09 <nirik> but those should all be done already
19:30:14 <petebuffon> right
19:30:37 <petebuffon> and finally should I stick with doing ~20 at a time or go for a more wholesale approach?
19:31:43 <nirik> Either way. I don't think there will be very many exceptions... just the ipv6 ones, and we really don't have too many of those.
19:31:51 <nirik> I bet there's under 20
19:32:08 <petebuffon> perfect
19:32:15 <petebuffon> thanks for hashing that out with me
19:32:28 <petebuffon> i should have time to work on it this week
19:32:43 <nirik> excellent. thanks for sticking it out, this has been a long one...
19:32:52 <nirik> but I think we will be in a much nicer place when it's done. :)
19:33:18 <nirik> ok, we are over time... thanks everyone!
19:33:21 <nirik> #endmeeting