18:00:01 #startmeeting Fedora Infrastructure Ops Daily Standup Meeting 18:00:01 Meeting started Mon Jun 1 18:00:01 2020 UTC. 18:00:01 This meeting is logged and archived in a public location. 18:00:01 The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:01 Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:01 The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 18:00:02 #chair cverna mboddu nirik smooge 18:00:02 #meetingname fedora_infrastructure_ops_daily_standup_meeting 18:00:02 #info meeting is 30 minutes MAX. At the end of 30, its stops 18:00:02 Current chairs: cverna mboddu nirik smooge 18:00:02 The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 18:00:02 #info agenda is at https://board.net/p/fedora-infra-daily 18:00:03 #topic Tickets needing review 18:00:04 #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1 18:00:06 here 18:00:10 * pingou 18:00:14 .ticket 8957 18:00:15 nirik: Issue #8957: create a jbangdev group - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8957 18:00:21 * mboddu is here 18:00:40 anyone want to be ticket updater today? 18:00:45 i will 18:00:50 I've this one on my to-process list, but the long and short of it is that I don't know how copr works with groups 18:00:54 so I can't quite help there 18:01:08 it's the same as pagure... on login. 18:01:13 so, ask them to logout/login? 18:01:20 sounds fine 18:01:34 smooge: ask them that and close? 18:01:44 pagure.issue.tag.added -- smooge tagged ticket fedora-infrastructure#8957: low-gain and medium-trouble https://pagure.io/fedora-infrastructure/issue/8957 18:01:45 pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8957 https://pagure.io/fedora-infrastructure/issue/8957 18:01:53 o/ 18:02:09 .ticket 8965 18:02:10 nirik: Issue #8965: Bodhi-backend is trying to connect directly to pagure.io fedmsg - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8965 18:02:12 .ticket 8964 18:02:14 nirik: Issue #8964: docs too big to fit on proxies anymore - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8964 18:02:15 sorry... missed one 18:02:27 pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#8957: "create a jbangdev group" https://pagure.io/fedora-infrastructure/issue/8957#comment-655673 18:02:33 pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8957 https://pagure.io/fedora-infrastructure/issue/8957 18:02:34 pagure.issue.edit -- pingou edited the priority fields of ticket fedora-infrastructure#8957 https://pagure.io/fedora-infrastructure/issue/8957 18:02:35 pagure.issue.comment.added -- pingou commented on ticket fedora-infrastructure#8957: "create a jbangdev group" https://pagure.io/fedora-infrastructure/issue/8957#comment-655675 18:02:46 so... if this could use a hardlink copy it might help a lot? 18:03:00 and/or our sync could just exclude all those build dirs 18:03:15 and/or we could just serve it from openshift and stop trying to put it on proxies. 18:03:33 well part of it is that their build script seems to repeat the entire build for ALL the zones and then copies out one zone, then removes that, then builds for all the zones, and copies out the next one, and 18:03:38 perhaps this would be good foer a list discussion? 18:04:11 unless theres some urgent doc change, I would prefer to just leave it off for now and revist after next week... 18:04:35 thoughts? 18:04:44 ok high-trouble high-gain groomed waiting on assignee 18:04:49 sounds good to me, we could ask asamalik and misc to also have a look at the build script 18:05:18 smooge: +1 ack 18:05:24 pagure.issue.tag.added -- smooge tagged ticket fedora-infrastructure#8964: groomed, high-gain, and 2 others https://pagure.io/fedora-infrastructure/issue/8964 18:05:25 pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8964 https://pagure.io/fedora-infrastructure/issue/8964 18:05:27 cverna: yeah. might be some easy wins 18:05:40 .ticket 8965 18:05:41 nirik: Issue #8965: Bodhi-backend is trying to connect directly to pagure.io fedmsg - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8965 18:06:09 so... is pagure.io still on it's on message bus? I can't recall 18:06:25 so, there's 1 last thing using fedmsg on bodhi-backend01 18:06:30 not anymore since fedora-messaging 18:06:44 and when we upgrade to rhel8 we should clean things up 18:06:58 it's the koji owner sync thing. 18:07:24 ie, watches for new packages and adds them to koji tags (I think still incorrectly as 'releng' owned too) 18:07:36 so could be turned off? 18:07:49 only if we don't want anyone to build any new packges. ;) 18:07:58 nicolas17: can you open a ticket so we can port it? 18:08:01 nirik: ^ 18:08:04 sorry nicolas17 :) 18:08:09 * nicolas17 hides 18:08:15 sure. I can... I am not sure how it works in phx2 either. 18:08:50 We can turn it off in IAD2 until we have a fedora-messaging version ? 18:08:55 well, we could reuse this ticket? or close this one and do a new one? 18:09:03 cverna: we can... 18:09:19 ticket wise, either way wfm 18:09:45 nirik, I should have been clearer: if pagure didn't use fedmsg anymore then having it defined in the configs that bodhi is using to connect to it could go away? 18:09:46 not sure how much effort there is to port it to fedora-messaging, if it is just a consumer that should be fairly straight forward 18:10:30 pagure.issue.new -- kevin opened a new ticket fedora-infrastructure#8967: "port koji-owner-sync script to fedora-messaging and make it work better" https://pagure.io/fedora-infrastructure/issue/8967 18:10:52 nirik, close this one as with that as the replacement? 18:10:55 smooge: bodhi doesn't.;. but this script that runs there does 18:11:03 smooge: ack +1 18:11:22 the script runs there because it needs koji admin perms, and it just reuses bodhi's keytab... 18:11:25 pagure.issue.edit -- smooge edited the close_status and status fields of ticket fedora-infrastructure#8965 https://pagure.io/fedora-infrastructure/issue/8965 18:11:25 pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#8965: "Bodhi-backend is trying to connect directly to pagure.io fedmsg" https://pagure.io/fedora-infrastructure/issue/8965#comment-655680 18:11:35 couldn't we just point fedmsg to the gateway? 18:11:52 instead of pagure01 18:11:54 thats not how it works is it? 18:12:13 for fedmsg, w/ the bridge it should I think 18:12:34 oh, I see what you are saying... point it to our busgateway 18:12:38 yes 18:12:39 yeah, we could. 18:12:52 pagure01> fedora-messaging -> bugateway -> fedmsg -> bodhi 18:12:54 I really want to finish porting away from fedmsg tho. it's really a headache having both 18:12:59 +! 18:13:01 +1* 18:13:15 but we could just punt and do that if we can't port in a timely manner 18:13:24 .ticket 8966 18:13:24 just as a temporary fix until we get the fedora-messaging client, which should be straight forward 18:13:25 nirik: Issue #8966: Staging Anitya is unable to send fedora messaging messages - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8966 18:13:44 I have a fix for this 18:13:45 Michal is ok w/ this waiting post move 18:13:51 that was my fix 18:14:17 yeah, odd that it broke now tho 18:15:09 pagure.issue.tag.added -- smooge tagged ticket fedora-infrastructure#8966: backlog, groomed, and 3 others https://pagure.io/fedora-infrastructure/issue/8966 18:15:10 pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8966 https://pagure.io/fedora-infrastructure/issue/8966 18:16:25 ok... thats all the infra needs review. 18:16:33 anything on releng side mboddu ? 18:16:43 or any other tickets we want to discuss? 18:16:44 nirik: Yes 18:17:00 .releng 9497 18:17:02 mboddu: Issue #9497: Package stuck in f33-build-side-23466-testing-pending - releng - Pagure.io - https://pagure.io/releng/issue/9497 18:17:14 brb 18:17:25 pingou: ^ on that, I can untag the builds, but the tag should be removed as well after that? 18:17:33 side tag* 18:18:02 * pingou checks the ticket 18:18:12 The doc doesn't say anything, I can look up history of other side tags or the code, but thought it might be easier to ask you :) 18:18:31 mboddu: I don't think they need up 18:18:33 us* 18:18:45 they can just do fedpkg remove-side-tag 18:18:56 it'll remove it w/ merging the packages 18:19:12 then they can just re-tag the build to the proper tag if they wish to 18:19:13 The builds are already merged 18:19:33 I will try fedpkg remove-side-tag and see what happens 18:19:43 mboddu: BTW, if you have a document for what to do at branching can you add 'add new rawhide tag to koji-gc' to it? 18:19:45 mboddu: I'd let them do it 18:20:05 nirik: Sure, it should be the releng doc as I have updated it very recently 18:20:08 pingou: Okay 18:20:20 Other tickets, I already worked on them 18:20:25 Thats it on releng side 18:20:33 I just noticed the other day it was missing and.... fedora_koji is painfully near full. 18:21:25 ok, quickly on the dc move... I updated https://hackmd.io/Eqsf5wFoQRGYhAVwSKJvIA?edit with our plans. 18:21:37 nirik: Sure, and sorry I missed it 18:21:42 so thats the place to look for up to date info... 18:21:44 basically: 18:22:10 today/tomorrow/wed we are going to be testing iad2 stuff more, making sure playbooks complete, making sure everything is installed, etc. 18:22:28 thursday we are gonna cut over zodbot/value 18:22:38 friday we are gonna cut over batacave 18:23:03 poor zodbot 18:23:11 * mboddu weeps for batcave and zodbot 18:23:14 then next week: monday is taking down all *stg.phx2 and other stuff we won't be moving, 18:23:33 tuesday is fedora-messaging cluster, mirrormanager, pdc and the auth stuff 18:23:47 sorry, monday ^ 18:24:02 tuesday is build stack... src, koji, etc 18:24:26 wed is openshift apps. mailman/lists, datagrepper/datanommer 18:24:58 thursday is the build openshift stuff (websites, docs, etc), blockerbugs, and elections (after the current elections close) 18:25:08 friday is fix things day. 18:25:10 When does batcave will be brought back up? 18:25:23 in IAD2 18:25:38 mboddu: it's already there 18:25:48 mboddu: this friday... the plan is to do a cut over friday and then everyone should start using it instead of the phx2 one (which will be turned off) 18:26:08 pingou: ^ thats the question, but worded poorly :D 18:26:10 we have to switch the fedora_app volume then to be rw in iad2. 18:26:30 yeah, it's there now, but it's got ro storage mounts and is manually synced. 18:26:42 nirik: Okay, thanks 18:27:02 I plan to send a message to devel-announce and infra lists later today with the above ^ ie, general days when things move. 18:27:13 we still need to isolate more exact times for things. 18:27:36 (because we need storage folks to flip over the storage and need to tell them when to be around) 18:27:54 does that all make sense? any problems or issues? things I forgot? 18:28:20 * mboddu reading back about compose boxes for rawhide 18:28:34 so, some things will be stopped while we move. 18:28:40 Right 18:28:52 rawhide composer will be brought back up on Monday or ... ? 18:29:18 so, tuesday morning we disable rawhide and bodhi. 18:29:37 tuesday evening if all goes well we try those manually and confirm they work 18:29:50 Oh okay 18:30:01 also, sunday night we ask people to stop processing unretirements or adding new packages 18:30:24 because thats going to potentially fail if done at the wrong time. So that holds until we say "ok, try doing one now" 18:30:40 Okay, makes sense 18:31:02 we might want to disable rawhide monday morning... since it's been taking a long time to finish 18:31:29 Tues morning or Mon morning? 18:31:46 Well, we could disable after our Mon compose 18:32:25 yep. 18:32:45 if you see things that need doing at a specific time on this plan: please add a checkbox for them! 18:33:07 ok, thats all I had and we are over time... 18:33:11 anything else? 18:33:23 Nothing from me 18:33:59 #endmeeting