22:11:16 <smooge> #startmeeting Fedora Infrastructure Ops Triage Meeting
22:11:16 <zodbot> Meeting started Thu Dec 10 22:11:16 2020 UTC.
22:11:16 <zodbot> This meeting is logged and archived in a public location.
22:11:16 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
22:11:16 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
22:11:16 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_triage_meeting'
22:11:25 <smooge> #chair mboddu nirik smooge pingou mobrien
22:11:25 <zodbot> Current chairs: mboddu mobrien nirik pingou smooge
22:11:35 <nirik> hey
22:11:52 <nirik> lets see
22:12:00 <smooge> #info https://pagure.io/fedora-infrastructure/issues?status=Open
22:12:18 <nirik> .ticket 8087
22:12:19 <zodbot> nirik: Issue #8087: Update bugzilla2fedmsg in production - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8087
22:13:06 * nirik re-reads the saga
22:14:15 <nirik> so, the stg one was working, but not the openshift one.
22:14:35 <nirik> I see however now that the stg one is broken... not in access, but in our python
22:14:47 <nirik> Dec 10 22:13:25 bugzilla2fedmsg01.stg.iad2.fedoraproject.org moksha-hub[26087]: KeyError: 'topic_prefix_re'
22:15:13 <smooge> whee. did these work after the move to IAD2?
22:15:27 <smooge> the dates on the ticket seemed all pre move
22:16:00 <nirik> the prod one does... I am not sure about stg
22:16:16 <smooge> ah that may need other firewall routing rules and is probably not allowed
22:16:32 <nirik> bugzilla2fedmsg is not deployed even in stg openshift
22:16:50 <nirik> well, I am not sure, because prod worked without any changes...
22:17:13 <nirik> but how about we mark it med/low/dev ?
22:17:17 <smooge> ok
22:17:34 <nirik> and ask pingou to see if he can bring it up and get to a point where it's not connecting or not?
22:17:36 <smooge> low gain or low trouble?
22:17:37 <nirik> or abompard
22:17:48 <nirik> low gain, since it's just stg...
22:17:59 <nirik> but perhaps it's med since we would like to move it to openshift.
22:18:00 <nirik> whatever
22:18:27 <nirik> and I guess say we have no idea if it's still not able to connect.
22:19:08 <nirik> .ticket 7973
22:19:09 <zodbot> nirik: Issue #7973: Finding a replacement for epylog - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/7973
22:19:21 <smooge> i would like to rename it bugzilla2fedoramessaging but that is probably too long
22:19:23 <nirik> how about we close this and fold it into the logging initative?
22:19:32 <smooge> was going to ask
22:19:39 <nirik> smooge: don't you know how much trouble you get into renaming things? ;)
22:20:08 <smooge> bugzilla-stream?
22:20:52 <smooge> do the same with 7588 ?
22:21:07 <nirik> sure!
22:21:26 <nirik> .ticket 7787
22:21:28 <zodbot> nirik: Issue #7787: Intermittent build failures due to git clone failure - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/7787
22:21:54 <nirik> close and if we see it again we can reopen
22:22:25 <nirik> .ticket 7826
22:22:26 <zodbot> nirik: Issue #7826: Get teams.fp.o to allow access to projects at a default level for anyone with a FAS - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/7826
22:22:38 <nirik> I need to go thru my emails and see if we asked them about this already.
22:22:49 <nirik> but low/low/ops?
22:23:36 <smooge> well he said he hasn't seen it since the move
22:23:47 <nirik> right. so closing seems ok?
22:23:47 <smooge> so I was going to say closed 'fixed itself
22:23:51 <nirik> ok, sure.
22:24:19 <smooge> miracles do happen
22:24:39 <nirik> indeed.
22:25:05 <smooge> 8167 get moved into an initiative :)
22:25:25 * smooge sees if he can get us down to 45 open
22:25:36 <nirik> I don't think thats in an initative? which one?
22:26:04 <smooge> oh wait never mind.. the rabbitmq schema inititive was dropped
22:26:19 <nirik> did you get 7826?
22:26:26 <nirik> but low/low/ops?
22:26:42 <nirik> .ticket 7361
22:26:43 <zodbot> nirik: Issue #7361: Signing infrastructure for aarch64 - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/7361
22:26:51 <smooge> sorry missed that one.. done
22:27:00 <nirik> need to get a hold of peter jones on this one I think?
22:27:09 <smooge> need a couple of things
22:27:30 <nirik> and I think it's pending the new shim and some other things...
22:27:35 <smooge> 1. how is it to be done.. that was still in the air each time I asked. 2. a system to do it on 3. pjones time
22:27:39 <nirik> high high ops? or med?
22:27:46 <smooge> high/high/ops
22:28:13 <nirik> well, yeah, yubikeys or smartcard... 2 is one or two of our existing builders, and yeah.
22:28:21 <smooge> 7880 should be ops?
22:28:51 <nirik> I am not sure. reading
22:29:32 <nirik> might be dev actually...
22:29:49 <smooge> good enough fro me
22:29:56 <nirik> and it will need to be dealt with somehow in the messaging schemas initative...
22:30:27 <nirik> .ticket 7115
22:30:28 <zodbot> nirik: Issue #7115: We need application monitoring against openshift routers - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/7115
22:30:38 <nirik> punt to monitoring iniative?
22:31:02 <smooge> done
22:31:15 <nirik> .ticket 6801
22:31:16 <zodbot> nirik: Issue #6801: run hardlink on fedora-secondary - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/6801
22:31:17 <nirik> should be ops?
22:31:24 <smooge> I closed it
22:31:33 <smooge> oh sorry damn it
22:31:42 <smooge> I keep missing you have moved to a new ticket
22:31:43 <nirik> we should still fix it.
22:31:47 <nirik> sorry.
22:32:30 <smooge> ok updated
22:32:46 <smooge> down to 7 tickets to label
22:33:00 <nirik> .ticket 6750
22:33:01 <zodbot> nirik: Issue #6750: Help! We need some kind of search solution for docs - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/6750
22:33:21 <nirik> med/med/ops?
22:33:27 <nirik> sorry, dev... ?
22:33:41 <smooge> sounds lik eit
22:34:26 <nirik> .ticket 6572
22:34:27 <zodbot> nirik: Issue #6572: Feature Request: Github Bages - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/6572
22:34:30 <nirik> low/low/dev
22:34:50 <smooge> done
22:35:06 <nirik> .ticket 6441
22:35:07 <zodbot> nirik: Issue #6441: Monitoring the worker's queue of pagure - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/6441
22:35:23 <nirik> punt for monitoring ? or should we do this now in nagios?
22:35:56 <smooge> dev ?
22:36:09 <nirik> well, it's just nagios config, could be ops...
22:36:17 <nirik> I think those are all just simple processes to check for
22:36:24 <smooge> is it though? it is tieing nagios into redis or fedora-messaging
22:36:40 <smooge> I guess I can look at nagios again
22:36:42 <nirik> oh wait.
22:36:53 <nirik> yeah, it's redis
22:36:59 <nirik> but I bet nagios has a redis plugin
22:37:00 <smooge> the necronomicom is getting boring
22:37:27 <nirik> .ticket 6397
22:37:28 <zodbot> nirik: Issue #6397: F28 timeframe: move to having user-facing sites live under a single domain - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/6397
22:37:47 <nirik> how about asking them to redo this as an initative? or is that too heavy... but I think it would be a lot of work.
22:37:48 <smooge> hey it says f28 and that is EOL.. so close it?
22:38:06 <smooge> no it needs to be an initiative this is a LOT of work
22:38:20 <nirik> yeah, thus why we have not gotten anywhere on it
22:38:35 <smooge> because it needs a web team and it needs us to look at moving magazine and other things back into Fedora Infra
22:39:23 <nirik> well, I don't think we need to do that... but that scoping needs to be done as part of it anyhow
22:39:47 <smooge> Kevin and I discussed this and we do not think we can do this as a ticket as it will need a lot of outside work and capital. We think this is an initiative and needs to go through the CPE planning process.
22:40:20 <smooge> nirik, looking at misc's last comment.. to put fedoraproject.org/magazine as one item would require it to be out of wordpress external.
22:40:36 <smooge> sorry "Kevin and I" was what I was going to put in the ticket
22:40:43 <nirik> sure. +1
22:40:44 <smooge> does that sound ok or different wording?
22:41:00 <smooge> done
22:41:09 <nirik> .ticket 5478
22:41:10 <zodbot> nirik: Issue #5478: Establishing a new FAS group for Python-SIG - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/5478
22:41:21 <nirik> so this has been waiting for abompard to deal with the mailing list...
22:41:50 <nirik> they wanted to rename the list
22:42:15 <smooge> oh gooood.
22:42:17 <nirik> so we could ask him to look again, punt it until after the mailman upgrade, or ask if there is some other way they would be willing to do this
22:42:36 <smooge> i think it has to be punted until mailman upgrade
22:42:47 <smooge> which I think I am going to make my main task next month
22:42:58 <smooge> well outside of all my other main tasks
22:43:06 <nirik> ok, so we should keep it around then... low/low/ops?
22:44:04 <smooge> low/low/ops/lists
22:44:06 <smooge> done
22:44:25 <nirik> .ticket 5372
22:44:26 <zodbot> nirik: Issue #5372: maintained cert-pinned location for Atomic Host (and other) content - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/5372
22:44:27 <smooge> I think 5372 is not going to happen
22:44:36 <nirik> I really don't want to do this. :( but I hate saying no...
22:44:56 <nirik> it's fragile and a lot of work
22:45:07 <smooge> I can say no
22:45:13 <nirik> ok. :)
22:45:18 <smooge> I have a lot of pent up no this week
22:45:43 <nirik> refer him to our management if he wishes to escalate.
22:46:25 <nirik> .ticket 5166
22:46:28 <zodbot> nirik: Issue #5166: RFR: Copr - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/5166
22:46:31 <nirik> (last ticket I think)
22:46:50 <nirik> huh
22:47:00 <nirik> ah, I see.
22:47:34 <nirik> so, I tried to close this a while back
22:48:08 <nirik> I think he and leigh were going to work out resposibilities. I am not sure if that ever happened.
22:48:47 <smooge> https://paste.centos.org/view/aab94c6c
22:48:54 <nirik> So, IMHO, close with "Copr _is_ fully supported now, we need to work out responsabilities out of band"
22:49:32 <nirik> smooge: well, I would say RFR's are for onboarding new apps... copr has long since been onboarded.
22:49:40 <smooge> I won't use fully supported.. I will say "The work on reaching a Service Level Expectations and COPR need to be done out of band"
22:49:52 <nirik> it's in the 'we provide platform, they provide code and fix their app'
22:49:59 <smooge> nirik, sorry that paste was for 5372
22:50:06 <smooge> i am slooow here
22:50:08 <nirik> oh, sorry
22:50:24 <nirik> sure. +1 for that for 5372
22:51:35 <smooge> last ticket is closed
22:51:54 <nirik> cool. :)
22:52:17 <smooge> so one thing we may have to do tomorrow/break. the pagure.io cert has not been approved by RHIT and it goes EOL soon
22:52:56 <nirik> can we poke someone? or they wanted to go to letsencrypt?
22:53:15 <smooge> I have sent several pokes but no reply etc Tuesday, January 5, 2021 at 7:00:00 AM
22:53:34 <smooge> my guess is that the piggy bank is dryt
22:53:41 <smooge> as it is EOQ
22:53:46 <smooge> EOY
22:54:02 <nirik> could be.
22:54:12 <nirik> well, we can see... we have a bit more time
22:54:16 <nirik> any others to discuss?
22:54:20 <smooge> we can wait until Jan 4th and see and move to LetsEncrypt
22:54:36 <smooge> nope that was it since you were out til Jan 4th and I was out til 11th
22:54:56 <smooge> I see a couple with no ops/dev
22:54:57 <nirik> cool. thanks for the triage. ;) we can start on releng in the new year
22:55:10 <nirik> the board is kinda unwealdy...
22:55:14 <nirik> so many on backlog
22:55:39 <smooge> .ticket 9170
22:55:40 <zodbot> smooge: Issue #9170: Cleaning up temporary queues in /bodhi? - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9170
22:56:38 <smooge> dev for 9170?
22:56:39 <nirik> this is kinda both dev and ops... ops to delete the existing queues, dev to fix it so it doesn't make them moving forward?
22:56:59 <smooge> added to both
22:57:48 <smooge> .ticket 9048
22:57:49 <zodbot> smooge: Issue #9048: Set sidetag_debuginfo_allowed/sidetag_rpm_macros_allowed on fXX-build - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9048
22:58:06 <smooge> it has been next-meeting for a while I think
22:58:43 <nirik> I think close it... debuginfo is fine, the other I think we need an ok from fesco.
22:58:51 <nirik> and I am not really in favor.
22:59:09 <nirik> unless we get koji to make 'unmergeable' side-tags for things like that
23:01:03 <smooge> close fixed or what
23:01:15 <nirik> close won'tfix...
23:01:47 <smooge> done
23:01:55 <nirik> any others?
23:02:09 <smooge> i think the one below it.. have to scrooll again
23:02:21 <nirik> .ticket 8989
23:02:22 <zodbot> nirik: Issue #8989: add queued and running as additional supported outcomes to resultsdb - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8989
23:02:27 <nirik> dev ?
23:02:38 <smooge> done
23:02:41 <nirik> .ticket 8931
23:02:42 <zodbot> nirik: Issue #8931: Ansible-review doesn't catch syntax errors - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8931
23:03:09 <nirik> I'm unclear on what we need to do here
23:03:35 <nirik> I guess add a ansible-playbook --syntax-check
23:03:38 <nirik> so dev?
23:04:00 <smooge> done
23:04:09 <nirik> and by the way, zuul doesn't seem to be checking our PR's at all. I guess thats another new ticket?
23:04:37 <smooge> .ticket 8729
23:04:38 <zodbot> smooge: Issue #8729: Do we want to have a copy of translate.fp.o backups? - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8729
23:04:46 <smooge> yeah sounds like another ticket
23:04:51 <nirik> https://pagure.io/fedora-infra/ansible/pull-requests shows the last zuul run 3months ago
23:04:57 * nirik can file that
23:05:14 <smooge> find out zuul is no longer supported we moved onto a new deamon
23:05:44 <nirik> so on 8729... I guess I just need to talk with pingou sometime on it... I think they should go on /fedora_backups on backup01
23:06:05 <smooge> ops
23:07:08 <nirik> sure
23:07:38 <smooge> 7095 ?
23:07:44 <nirik> .ticket 7095
23:07:50 <zodbot> nirik: An error has occurred and has been logged. Check the logs for more information.
23:07:54 <nirik> .ticket 7095
23:07:59 <smooge> sorry its priv..
23:07:59 <zodbot> nirik: An error has occurred and has been logged. Check the logs for more information.
23:08:14 <nirik> ah, priv
23:08:29 <nirik> yeah, this should still be done. ops med/med?
23:08:30 <smooge> i have labeled it
23:08:51 <smooge> .ticket 8903
23:08:52 <zodbot> smooge: Issue #8903: AWS: additional permission requirements for Testing Farm - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8903
23:09:28 <nirik> med med ops and ping mobrien ?
23:09:33 <nirik> I just keep never getting to it...
23:09:34 <smooge> i think this is the last one I can deal with
23:09:47 <smooge> we still have about 8 or so needing to be put into a or b
23:10:11 <nirik> 8415 should be dev
23:10:43 <smooge> done
23:10:57 <smooge> .ticket 7826
23:10:58 <zodbot> smooge: Issue #7826: Get teams.fp.o to allow access to projects at a default level for anyone with a FAS - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/7826
23:11:11 <smooge> will not / can not fix ?
23:11:21 <smooge> oh wait sorry just updated
23:11:56 <smooge> i didn't put ops i
23:12:04 <nirik> I need to check my emails on it.
23:13:20 <smooge> ok I think 7880 was the only one
23:13:37 <nirik> .ticket 7880
23:13:38 <zodbot> nirik: Issue #7880: email2fas not working in production (fedmsg/datanommer, fedmsg/irc) - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/7880
23:13:57 <smooge> i marked it but did not save it
23:14:00 <smooge> it is done now
23:14:06 <nirik> ah, ok.
23:14:14 <smooge> typing dev and space does not make it 'dev'
23:14:22 <smooge> ok we can start on releng
23:14:32 <smooge> not tonight...
23:14:36 <smooge> but we can do so
23:14:49 <nirik> well, and 9524 that I just filed. ;)
23:14:58 <nirik> yeah, I think releng we should start on in the new year...
23:15:21 <smooge> yeah
23:15:24 <smooge> ending meeting?
23:15:56 <nirik> +1
23:15:56 <smooge> #endmeeting