22:11:16 <smooge> #startmeeting Fedora Infrastructure Ops Triage Meeting 22:11:16 <zodbot> Meeting started Thu Dec 10 22:11:16 2020 UTC. 22:11:16 <zodbot> This meeting is logged and archived in a public location. 22:11:16 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot. 22:11:16 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 22:11:16 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_triage_meeting' 22:11:25 <smooge> #chair mboddu nirik smooge pingou mobrien 22:11:25 <zodbot> Current chairs: mboddu mobrien nirik pingou smooge 22:11:35 <nirik> hey 22:11:52 <nirik> lets see 22:12:00 <smooge> #info https://pagure.io/fedora-infrastructure/issues?status=Open 22:12:18 <nirik> .ticket 8087 22:12:19 <zodbot> nirik: Issue #8087: Update bugzilla2fedmsg in production - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8087 22:13:06 * nirik re-reads the saga 22:14:15 <nirik> so, the stg one was working, but not the openshift one. 22:14:35 <nirik> I see however now that the stg one is broken... not in access, but in our python 22:14:47 <nirik> Dec 10 22:13:25 bugzilla2fedmsg01.stg.iad2.fedoraproject.org moksha-hub[26087]: KeyError: 'topic_prefix_re' 22:15:13 <smooge> whee. did these work after the move to IAD2? 22:15:27 <smooge> the dates on the ticket seemed all pre move 22:16:00 <nirik> the prod one does... I am not sure about stg 22:16:16 <smooge> ah that may need other firewall routing rules and is probably not allowed 22:16:32 <nirik> bugzilla2fedmsg is not deployed even in stg openshift 22:16:50 <nirik> well, I am not sure, because prod worked without any changes... 22:17:13 <nirik> but how about we mark it med/low/dev ? 22:17:17 <smooge> ok 22:17:34 <nirik> and ask pingou to see if he can bring it up and get to a point where it's not connecting or not? 22:17:36 <smooge> low gain or low trouble? 22:17:37 <nirik> or abompard 22:17:48 <nirik> low gain, since it's just stg... 22:17:59 <nirik> but perhaps it's med since we would like to move it to openshift. 22:18:00 <nirik> whatever 22:18:27 <nirik> and I guess say we have no idea if it's still not able to connect. 22:19:08 <nirik> .ticket 7973 22:19:09 <zodbot> nirik: Issue #7973: Finding a replacement for epylog - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/7973 22:19:21 <smooge> i would like to rename it bugzilla2fedoramessaging but that is probably too long 22:19:23 <nirik> how about we close this and fold it into the logging initative? 22:19:32 <smooge> was going to ask 22:19:39 <nirik> smooge: don't you know how much trouble you get into renaming things? ;) 22:20:08 <smooge> bugzilla-stream? 22:20:52 <smooge> do the same with 7588 ? 22:21:07 <nirik> sure! 22:21:26 <nirik> .ticket 7787 22:21:28 <zodbot> nirik: Issue #7787: Intermittent build failures due to git clone failure - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/7787 22:21:54 <nirik> close and if we see it again we can reopen 22:22:25 <nirik> .ticket 7826 22:22:26 <zodbot> nirik: Issue #7826: Get teams.fp.o to allow access to projects at a default level for anyone with a FAS - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/7826 22:22:38 <nirik> I need to go thru my emails and see if we asked them about this already. 22:22:49 <nirik> but low/low/ops? 22:23:36 <smooge> well he said he hasn't seen it since the move 22:23:47 <nirik> right. so closing seems ok? 22:23:47 <smooge> so I was going to say closed 'fixed itself 22:23:51 <nirik> ok, sure. 22:24:19 <smooge> miracles do happen 22:24:39 <nirik> indeed. 22:25:05 <smooge> 8167 get moved into an initiative :) 22:25:25 * smooge sees if he can get us down to 45 open 22:25:36 <nirik> I don't think thats in an initative? which one? 22:26:04 <smooge> oh wait never mind.. the rabbitmq schema inititive was dropped 22:26:19 <nirik> did you get 7826? 22:26:26 <nirik> but low/low/ops? 22:26:42 <nirik> .ticket 7361 22:26:43 <zodbot> nirik: Issue #7361: Signing infrastructure for aarch64 - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/7361 22:26:51 <smooge> sorry missed that one.. done 22:27:00 <nirik> need to get a hold of peter jones on this one I think? 22:27:09 <smooge> need a couple of things 22:27:30 <nirik> and I think it's pending the new shim and some other things... 22:27:35 <smooge> 1. how is it to be done.. that was still in the air each time I asked. 2. a system to do it on 3. pjones time 22:27:39 <nirik> high high ops? or med? 22:27:46 <smooge> high/high/ops 22:28:13 <nirik> well, yeah, yubikeys or smartcard... 2 is one or two of our existing builders, and yeah. 22:28:21 <smooge> 7880 should be ops? 22:28:51 <nirik> I am not sure. reading 22:29:32 <nirik> might be dev actually... 22:29:49 <smooge> good enough fro me 22:29:56 <nirik> and it will need to be dealt with somehow in the messaging schemas initative... 22:30:27 <nirik> .ticket 7115 22:30:28 <zodbot> nirik: Issue #7115: We need application monitoring against openshift routers - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/7115 22:30:38 <nirik> punt to monitoring iniative? 22:31:02 <smooge> done 22:31:15 <nirik> .ticket 6801 22:31:16 <zodbot> nirik: Issue #6801: run hardlink on fedora-secondary - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/6801 22:31:17 <nirik> should be ops? 22:31:24 <smooge> I closed it 22:31:33 <smooge> oh sorry damn it 22:31:42 <smooge> I keep missing you have moved to a new ticket 22:31:43 <nirik> we should still fix it. 22:31:47 <nirik> sorry. 22:32:30 <smooge> ok updated 22:32:46 <smooge> down to 7 tickets to label 22:33:00 <nirik> .ticket 6750 22:33:01 <zodbot> nirik: Issue #6750: Help! We need some kind of search solution for docs - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/6750 22:33:21 <nirik> med/med/ops? 22:33:27 <nirik> sorry, dev... ? 22:33:41 <smooge> sounds lik eit 22:34:26 <nirik> .ticket 6572 22:34:27 <zodbot> nirik: Issue #6572: Feature Request: Github Bages - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/6572 22:34:30 <nirik> low/low/dev 22:34:50 <smooge> done 22:35:06 <nirik> .ticket 6441 22:35:07 <zodbot> nirik: Issue #6441: Monitoring the worker's queue of pagure - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/6441 22:35:23 <nirik> punt for monitoring ? or should we do this now in nagios? 22:35:56 <smooge> dev ? 22:36:09 <nirik> well, it's just nagios config, could be ops... 22:36:17 <nirik> I think those are all just simple processes to check for 22:36:24 <smooge> is it though? it is tieing nagios into redis or fedora-messaging 22:36:40 <smooge> I guess I can look at nagios again 22:36:42 <nirik> oh wait. 22:36:53 <nirik> yeah, it's redis 22:36:59 <nirik> but I bet nagios has a redis plugin 22:37:00 <smooge> the necronomicom is getting boring 22:37:27 <nirik> .ticket 6397 22:37:28 <zodbot> nirik: Issue #6397: F28 timeframe: move to having user-facing sites live under a single domain - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/6397 22:37:47 <nirik> how about asking them to redo this as an initative? or is that too heavy... but I think it would be a lot of work. 22:37:48 <smooge> hey it says f28 and that is EOL.. so close it? 22:38:06 <smooge> no it needs to be an initiative this is a LOT of work 22:38:20 <nirik> yeah, thus why we have not gotten anywhere on it 22:38:35 <smooge> because it needs a web team and it needs us to look at moving magazine and other things back into Fedora Infra 22:39:23 <nirik> well, I don't think we need to do that... but that scoping needs to be done as part of it anyhow 22:39:47 <smooge> Kevin and I discussed this and we do not think we can do this as a ticket as it will need a lot of outside work and capital. We think this is an initiative and needs to go through the CPE planning process. 22:40:20 <smooge> nirik, looking at misc's last comment.. to put fedoraproject.org/magazine as one item would require it to be out of wordpress external. 22:40:36 <smooge> sorry "Kevin and I" was what I was going to put in the ticket 22:40:43 <nirik> sure. +1 22:40:44 <smooge> does that sound ok or different wording? 22:41:00 <smooge> done 22:41:09 <nirik> .ticket 5478 22:41:10 <zodbot> nirik: Issue #5478: Establishing a new FAS group for Python-SIG - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/5478 22:41:21 <nirik> so this has been waiting for abompard to deal with the mailing list... 22:41:50 <nirik> they wanted to rename the list 22:42:15 <smooge> oh gooood. 22:42:17 <nirik> so we could ask him to look again, punt it until after the mailman upgrade, or ask if there is some other way they would be willing to do this 22:42:36 <smooge> i think it has to be punted until mailman upgrade 22:42:47 <smooge> which I think I am going to make my main task next month 22:42:58 <smooge> well outside of all my other main tasks 22:43:06 <nirik> ok, so we should keep it around then... low/low/ops? 22:44:04 <smooge> low/low/ops/lists 22:44:06 <smooge> done 22:44:25 <nirik> .ticket 5372 22:44:26 <zodbot> nirik: Issue #5372: maintained cert-pinned location for Atomic Host (and other) content - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/5372 22:44:27 <smooge> I think 5372 is not going to happen 22:44:36 <nirik> I really don't want to do this. :( but I hate saying no... 22:44:56 <nirik> it's fragile and a lot of work 22:45:07 <smooge> I can say no 22:45:13 <nirik> ok. :) 22:45:18 <smooge> I have a lot of pent up no this week 22:45:43 <nirik> refer him to our management if he wishes to escalate. 22:46:25 <nirik> .ticket 5166 22:46:28 <zodbot> nirik: Issue #5166: RFR: Copr - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/5166 22:46:31 <nirik> (last ticket I think) 22:46:50 <nirik> huh 22:47:00 <nirik> ah, I see. 22:47:34 <nirik> so, I tried to close this a while back 22:48:08 <nirik> I think he and leigh were going to work out resposibilities. I am not sure if that ever happened. 22:48:47 <smooge> https://paste.centos.org/view/aab94c6c 22:48:54 <nirik> So, IMHO, close with "Copr _is_ fully supported now, we need to work out responsabilities out of band" 22:49:32 <nirik> smooge: well, I would say RFR's are for onboarding new apps... copr has long since been onboarded. 22:49:40 <smooge> I won't use fully supported.. I will say "The work on reaching a Service Level Expectations and COPR need to be done out of band" 22:49:52 <nirik> it's in the 'we provide platform, they provide code and fix their app' 22:49:59 <smooge> nirik, sorry that paste was for 5372 22:50:06 <smooge> i am slooow here 22:50:08 <nirik> oh, sorry 22:50:24 <nirik> sure. +1 for that for 5372 22:51:35 <smooge> last ticket is closed 22:51:54 <nirik> cool. :) 22:52:17 <smooge> so one thing we may have to do tomorrow/break. the pagure.io cert has not been approved by RHIT and it goes EOL soon 22:52:56 <nirik> can we poke someone? or they wanted to go to letsencrypt? 22:53:15 <smooge> I have sent several pokes but no reply etc Tuesday, January 5, 2021 at 7:00:00 AM 22:53:34 <smooge> my guess is that the piggy bank is dryt 22:53:41 <smooge> as it is EOQ 22:53:46 <smooge> EOY 22:54:02 <nirik> could be. 22:54:12 <nirik> well, we can see... we have a bit more time 22:54:16 <nirik> any others to discuss? 22:54:20 <smooge> we can wait until Jan 4th and see and move to LetsEncrypt 22:54:36 <smooge> nope that was it since you were out til Jan 4th and I was out til 11th 22:54:56 <smooge> I see a couple with no ops/dev 22:54:57 <nirik> cool. thanks for the triage. ;) we can start on releng in the new year 22:55:10 <nirik> the board is kinda unwealdy... 22:55:14 <nirik> so many on backlog 22:55:39 <smooge> .ticket 9170 22:55:40 <zodbot> smooge: Issue #9170: Cleaning up temporary queues in /bodhi? - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9170 22:56:38 <smooge> dev for 9170? 22:56:39 <nirik> this is kinda both dev and ops... ops to delete the existing queues, dev to fix it so it doesn't make them moving forward? 22:56:59 <smooge> added to both 22:57:48 <smooge> .ticket 9048 22:57:49 <zodbot> smooge: Issue #9048: Set sidetag_debuginfo_allowed/sidetag_rpm_macros_allowed on fXX-build - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9048 22:58:06 <smooge> it has been next-meeting for a while I think 22:58:43 <nirik> I think close it... debuginfo is fine, the other I think we need an ok from fesco. 22:58:51 <nirik> and I am not really in favor. 22:59:09 <nirik> unless we get koji to make 'unmergeable' side-tags for things like that 23:01:03 <smooge> close fixed or what 23:01:15 <nirik> close won'tfix... 23:01:47 <smooge> done 23:01:55 <nirik> any others? 23:02:09 <smooge> i think the one below it.. have to scrooll again 23:02:21 <nirik> .ticket 8989 23:02:22 <zodbot> nirik: Issue #8989: add queued and running as additional supported outcomes to resultsdb - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8989 23:02:27 <nirik> dev ? 23:02:38 <smooge> done 23:02:41 <nirik> .ticket 8931 23:02:42 <zodbot> nirik: Issue #8931: Ansible-review doesn't catch syntax errors - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8931 23:03:09 <nirik> I'm unclear on what we need to do here 23:03:35 <nirik> I guess add a ansible-playbook --syntax-check 23:03:38 <nirik> so dev? 23:04:00 <smooge> done 23:04:09 <nirik> and by the way, zuul doesn't seem to be checking our PR's at all. I guess thats another new ticket? 23:04:37 <smooge> .ticket 8729 23:04:38 <zodbot> smooge: Issue #8729: Do we want to have a copy of translate.fp.o backups? - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8729 23:04:46 <smooge> yeah sounds like another ticket 23:04:51 <nirik> https://pagure.io/fedora-infra/ansible/pull-requests shows the last zuul run 3months ago 23:04:57 * nirik can file that 23:05:14 <smooge> find out zuul is no longer supported we moved onto a new deamon 23:05:44 <nirik> so on 8729... I guess I just need to talk with pingou sometime on it... I think they should go on /fedora_backups on backup01 23:06:05 <smooge> ops 23:07:08 <nirik> sure 23:07:38 <smooge> 7095 ? 23:07:44 <nirik> .ticket 7095 23:07:50 <zodbot> nirik: An error has occurred and has been logged. Check the logs for more information. 23:07:54 <nirik> .ticket 7095 23:07:59 <smooge> sorry its priv.. 23:07:59 <zodbot> nirik: An error has occurred and has been logged. Check the logs for more information. 23:08:14 <nirik> ah, priv 23:08:29 <nirik> yeah, this should still be done. ops med/med? 23:08:30 <smooge> i have labeled it 23:08:51 <smooge> .ticket 8903 23:08:52 <zodbot> smooge: Issue #8903: AWS: additional permission requirements for Testing Farm - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8903 23:09:28 <nirik> med med ops and ping mobrien ? 23:09:33 <nirik> I just keep never getting to it... 23:09:34 <smooge> i think this is the last one I can deal with 23:09:47 <smooge> we still have about 8 or so needing to be put into a or b 23:10:11 <nirik> 8415 should be dev 23:10:43 <smooge> done 23:10:57 <smooge> .ticket 7826 23:10:58 <zodbot> smooge: Issue #7826: Get teams.fp.o to allow access to projects at a default level for anyone with a FAS - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/7826 23:11:11 <smooge> will not / can not fix ? 23:11:21 <smooge> oh wait sorry just updated 23:11:56 <smooge> i didn't put ops i 23:12:04 <nirik> I need to check my emails on it. 23:13:20 <smooge> ok I think 7880 was the only one 23:13:37 <nirik> .ticket 7880 23:13:38 <zodbot> nirik: Issue #7880: email2fas not working in production (fedmsg/datanommer, fedmsg/irc) - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/7880 23:13:57 <smooge> i marked it but did not save it 23:14:00 <smooge> it is done now 23:14:06 <nirik> ah, ok. 23:14:14 <smooge> typing dev and space does not make it 'dev' 23:14:22 <smooge> ok we can start on releng 23:14:32 <smooge> not tonight... 23:14:36 <smooge> but we can do so 23:14:49 <nirik> well, and 9524 that I just filed. ;) 23:14:58 <nirik> yeah, I think releng we should start on in the new year... 23:15:21 <smooge> yeah 23:15:24 <smooge> ending meeting? 23:15:56 <nirik> +1 23:15:56 <smooge> #endmeeting