19:00:24 #startmeeting Fedora Infrastructure Ops Daily Standup Meeting 19:00:24 Meeting started Thu Jan 13 19:00:24 2022 UTC. 19:00:24 This meeting is logged and archived in a public location. 19:00:24 The chair is nirik. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions. 19:00:24 Useful Commands: #action #agreed #halp #info #idea #link #topic. 19:00:24 The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 19:00:25 #chair mboddu nirik pingou mobrien nb 19:00:25 Current chairs: mboddu mobrien nb nirik pingou 19:00:25 #meetingname fedora_infrastructure_ops_daily_standup_meeting 19:00:25 #info meeting is 30 minutes MAX. At the end of 30, its stops 19:00:25 The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 19:00:25 #info agenda is at https://board.net/p/fedora-infra-daily 19:00:27 #info reminder: speak up if you want to work on a ticket! 19:00:29 #topic Tickets needing review 19:00:31 #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1 19:01:05 .ticket 10474 19:01:06 nirik: Issue #10474: New 'legal' namespace in Fedora GitHub - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10474 19:01:22 low/low/ops keep ryan assigned 19:01:46 .ticket 10476 19:01:47 nirik: Issue #10476: Planned Outage - Datanommer Migration - 2022-01-17 11:00 UTC - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10476 19:01:57 Quick question. Is there any relation to fas and gitlab groups at the moment? 19:03:05 med / med ops added comments 19:03:30 mobrien: no. We could do that, but it's not great... 19:03:59 I must send that outage to the lists and have the planned status ready to push, just waiting on a review. Thanks nirik 19:04:01 you need a specific group for every permission level... and it makes that match gitlab 19:04:20 That sounds sub optimal.... 19:04:34 so, if you say wanted to setup websites project like that. you would need 'websites-admins' 'websites-developers' 'websites-contibutors' etc. 19:04:51 Let's leave as is then 19:05:03 and then if you added someone from the gitlab interface to some permissions like developer without adding them to the fas group, it would remove them when they logged in 19:05:27 so yeah, we thought it best to just manage perms there and let subgroups do their own thing 19:05:47 .ticket 10477 19:05:48 nirik: Issue #10477: proxy40 does not have enough memory - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10477 19:05:58 low/low/ops I guess? 19:06:39 The machine type will need to be changed and a dns update after as the ip will be lost 19:06:42 thats it for infra tickets triage 19:07:01 mobrien: yeah. I don't know what time has 8GB mem, but thats all doable. 19:07:35 Should be straightforward. You can assign to me. 19:07:56 sure. I think all those proxy3* have 4gb too? not sure tho 19:08:17 I'll have a look through 19:08:22 #topic Upcoming work / Planning 19:08:57 So, I am trying to finish builder upgrades, but I am waiting on talking to the mainframe folks so I can find out how much disk we really have in our s390x kvm lpar. ;( Hopefully will get that soon... 19:09:20 I did a rebuild of the osbs buildroot image today for f35 and everything built OK. I didn't get time to test do I'll test that tomorrow and push if it works 19:09:21 mobrien: you didn't find anything further on container builds failures did you? ;( 19:09:44 ah ok. worth a try... it seems like something deeper wrong tho. 19:10:18 I didn't find the root cause but something seems to be up with the the image tags on the registry. It looks as though some are missing 19:10:56 I might try drag David in tomorrow as well if he is available 19:10:57 huh, nothing really should have changed there... 19:11:31 the only thing that I know of that changed around that time was me reinstalling builders... but it doesn't even run any of that on builders, it's all on the osbs cluster by then. 19:12:19 anyhow yeah, perhaps David can see whats going on too if you can catch him. 19:13:06 The other current breakage is the IPA certs thing. ;( I dug into that for a few hours yesterday without fixing it... I'll try and look more today. 19:13:24 perhaps I can try downgrading some things. 19:14:15 #topic Discussion / AOB 19:14:24 anything to discuss or bring up ? 19:15:31 nirik: If you have a couple minutes sometime today I'd like to discuss next steps on https://pagure.io/fedora-infrastructure/issue/9695. 19:16:14 ah ok. 19:16:26 so, whats left there? 19:16:49 a bunch of vm's I guess? 19:16:57 yeah, a bunch of vms 19:17:40 I guess my main question is if the vm macs still need to be dumped or if I can use the ansible mac variable? 19:18:05 just the variable. don't need to be dumped. 19:18:13 it will use the one from ansible facts 19:18:38 alright so not necessarily the same for all vms 19:18:47 how do I get the ansible facts output then? 19:19:03 no no, you don't need it... use the variable and ansible will fill it in. 19:19:34 ah okay that makes sense 19:19:44 alright so it should be pretty straightforward then 19:19:56 so take a look at proxy01.iad2.fedoraproject.org for example 19:20:21 no wait, that one is wrong 19:21:17 so I wonder... 19:21:24 Jo 19:21:38 perhaps we should just define a default network_connections. 19:22:24 hey nb! Hope you had a good holidays... 19:22:31 alright, so we would define that one place and then use it as a variable? 19:22:57 if we just define a default one in one place we only need to override it for hosts that need it/non vm's.... 19:23:22 ah 19:23:31 no, just define it in group_vars/all and it will affect all hosts except those that override it in host_vars/hostname 19:23:51 got it 19:24:12 so I guess lets try that and clean up any of the vm's that are wrong... 19:24:35 sounds good, what should I use as a basis for the default? 19:25:50 so, looks like the only ones that are vm's that are wrong currently are proxies... I probibly did those before I considered things :) 19:26:31 okay 19:26:32 the one in proxy14 seems reasonably complete. 19:26:43 but then there's ipv6 too... hum. 19:27:20 I guess the ipv6 vm's will need to override, so we need the proxies the way they are anyhow. 19:27:49 so, add to group_vars/all one that has just ipv4 and eth0... and uses {{ ansible_default_ipv4.macaddress }} for mac 19:28:43 great, so the exceptions then will be vms with an interface other than eth0 and/or ipv6? 19:30:01 yep. or hardware ones with bridges 19:30:09 but those should all be done already 19:30:14 right 19:30:37 and finally should I stick with doing ~20 at a time or go for a more wholesale approach? 19:31:43 Either way. I don't think there will be very many exceptions... just the ipv6 ones, and we really don't have too many of those. 19:31:51 I bet there's under 20 19:32:08 perfect 19:32:15 thanks for hashing that out with me 19:32:28 i should have time to work on it this week 19:32:43 excellent. thanks for sticking it out, this has been a long one... 19:32:52 but I think we will be in a much nicer place when it's done. :) 19:33:18 ok, we are over time... thanks everyone! 19:33:21 #endmeeting