15:00:40 <smooge> #startmeeting Infrastructure (2019-12-05)
15:00:40 <zodbot> Meeting started Thu Dec  5 15:00:40 2019 UTC.
15:00:40 <zodbot> This meeting is logged and archived in a public location.
15:00:40 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:40 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:40 <zodbot> The meeting name has been set to 'infrastructure_(2019-12-05)'
15:00:41 <smooge> #meetingname infrastructure
15:00:41 <zodbot> The meeting name has been set to 'infrastructure'
15:00:41 <smooge> #chair nirik pingou relrod smooge tflink cverna mizdebsk mkonecny abompard bowlofeggs
15:00:41 <zodbot> Current chairs: abompard bowlofeggs cverna mizdebsk mkonecny nirik pingou relrod smooge tflink
15:00:41 <smooge> #info Agenda is at: https://board.net/p/fedora-infra
15:00:41 <smooge> #topic aloha
15:00:48 * pingou says: hi
15:00:53 <tflink> morning
15:00:54 <mizdebsk> hello
15:01:29 <smooge> morning all
15:01:43 <smooge> nirik, will be a bit late
15:01:49 <smooge> #topic Next chair
15:01:49 <smooge> #info magic eight ball says:
15:01:49 <smooge> #info    2019-12-05 -smooge
15:01:49 <smooge> #info    2019-12-12 -nirik
15:01:49 <smooge> #info    2019-12-19 - ??
15:01:50 <smooge> #info    2019-12-26 - NO MEETING
15:01:52 <smooge> #info    2020-01-02 - NO MEETING
15:01:54 <smooge> #info    2020-01-09 - ??
15:02:39 * cverna waves
15:03:10 * relrod waves
15:03:10 <lgriffin> .hello2
15:03:11 <zodbot> lgriffin: lgriffin 'Leigh Griffin' <lgriffin@redhat.com>
15:03:28 <lgriffin> Howdy all
15:03:30 <smooge> we need someone to do the 12-19 meeting
15:03:33 <smooge> I will be out then
15:03:49 <mizdebsk> i can do it
15:04:52 <cverna> mizdebsk++
15:05:15 <cverna> no cookies :(
15:05:30 <smooge> mizdebsk++ thanks
15:05:30 <zodbot> smooge: Karma for mizdebsk changed to 5 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:05:50 <smooge> #topic New folks introductions
15:05:50 <smooge> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:05:50 <smooge> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
15:06:09 <mkonecny> .hello zlopez
15:06:10 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@packetseekers.eu>
15:06:39 <smooge> do we have any new people? if not we can move on
15:07:10 <smooge> #topic announcements and information
15:07:11 <smooge> #info No meetings coming up due to various holidays. Many Fedora people will be on time-off til parts of january
15:07:11 <smooge> #info work in progress in improving the script syncing assignee and CC from dist-git to bugzilla
15:07:11 <smooge> #info work done on improving the error messages sent by fedscm-admin when processing scm requests
15:07:11 <smooge> #info builder reinstalls/uprgades to f31 continuing. Should be done before holiday break - kevin
15:07:12 <smooge> #info ops folks are trying a 30min ticket triage every day at 19UTC in #fedora-admin
15:07:57 <nirik> morning
15:10:18 <smooge> any other announcements for this week?
15:12:58 <smooge> #topic Oncall
15:12:58 <smooge> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
15:12:58 <smooge> #info smooge is on call 2019-11-28->2019-12-05
15:12:58 <smooge> #info kevin is oncall 2019-12-05 -> 2019-12-12
15:12:58 <smooge> ??? is oncall 2019-12-12 -> 2019-12-19
15:12:59 <smooge> NOONE is oncall 2019-12-19 -> 2020-01-02
15:13:01 <smooge> ??? is oncall 2019-01-02 -> 2019-01-09
15:13:33 <nirik> .takoncallus
15:13:38 <nirik> .takeoncallus
15:13:49 <smooge> #info On-call summary: mirrorlist outages still happening.. most other pings have been on 'do my ticket' tpes
15:13:49 <pingou> .oncall
15:13:49 <zodbot> smooge is oncall. My normal hours are 13:00 UTC to 21:00 UTC Monday through Friday. If I do not answer or it is outside those hours, please file a ticket (https://pagure.io/fedora-infrastructure/issues)
15:13:51 <nirik> never remember the right one. ;)
15:13:59 <pingou> nirik: it's on the board
15:14:09 <pingou> .oncalltakeus
15:14:09 <zodbot> pingou: Error: You don't have the alias.add capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified.
15:14:09 <smooge> .oncalltakeus
15:14:20 <nirik> .oncalltakeus
15:14:20 <zodbot> nirik: Kneel before zod!
15:14:25 <pingou> .oncall
15:14:25 <zodbot> nirik is oncall. My normal hours are 13:00 UTC to 21:00 UTC Monday through Friday. If I do not answer or it is outside those hours, please file a ticket (https://pagure.io/fedora-infrastructure/issues)
15:14:29 <pingou> working :)
15:14:36 <nirik> yeah
15:14:46 <pingou> so who volunteers to take over next week?
15:15:42 <cverna> I can take next week
15:16:34 <mizdebsk> in december i can take any week too
15:17:21 <smooge> cverna said it first so I will put him there
15:17:51 <smooge> #info cverna is oncall 2019-12-12 -> 2019-12-19
15:17:51 <smooge> #info NOONE is oncall 2019-12-19 -> 2020-01-02
15:18:09 <smooge> #topic Monitoring discussion
15:18:09 <smooge> #info https://nagios.fedoraproject.org/nagios
15:18:09 <smooge> #info Go over existing out items and fix
15:19:35 * nirik is having firefox issues. ;(
15:19:42 <cverna> osbs-master01.stg.phx2.fedoraproject.org is up now should I restart the nrpe service ?
15:19:49 <smooge> proxy11 is due to a problem with podman not killing processes cleanly so one will keep running locking port 81 or 82 until kill -9 is send
15:20:02 <smooge> cverna, check the firewall allows is
15:20:11 <nirik> I am not sure at all why buildvmhost-aarch64-01 is showing down. The machine is up and seems fine.
15:20:34 <smooge> cverna, openshift things do weird things with firewalls so it may not allow it
15:21:12 <cverna> smooge: oh right, checking that now
15:23:21 <nirik> ok, that was firewall, I think I fixed it.
15:23:41 <nirik> the swap low thing smooge filed a bug on
15:23:43 <smooge> any other items for this or do we need to go to the next topic?
15:23:57 <smooge> yeah.. it is an interesting kernel bug
15:24:13 <nirik> odcs Check for fedmsg-hub-3 proc just needs removed...
15:24:19 <nirik> that one moved to fedora-messaging I think?
15:24:47 <nirik> planet I am not sure of, it's updating, but not sending fedmsgs. ;(
15:25:41 <smooge> is it one where someone set it up for fedora-messaging but because it isn't in PHX2 amqp doesn't work?
15:27:02 <smooge> ok next itme
15:27:05 <smooge> #topic Tickets discussion
15:27:05 <smooge> #info https://pagure.io/fedora-infrastructure/report/Meetings%20ticket
15:27:34 <nirik> no, it's in phx2...
15:27:49 <smooge> isn't planet on people?
15:28:10 <nirik> oh yes, I thought you meant odcs
15:30:15 <smooge> ok for tickets we are doing a daily 30 minute standup in #fedora-admin to go over tickets needing to be itemized and to assign out any tickets so they are getting worked on
15:30:52 <smooge> #info Fedora CoreOS Team requests
15:30:52 <smooge> #link https://hackmd.io/5uB7hOJKSjGUt65iLgPnbA#Existing-requests-for-Fedora-Infra
15:33:46 <smooge> #topic ResultsDB Ownership Going Forward
15:33:46 <smooge> #link https://pagure.io/fedora-infrastructure/issue/8415
15:34:02 <smooge> not sure who is running this question
15:34:22 <nirik> tflink was asking.
15:34:40 <nirik> I think we need to bring it to the CPE team/applicaations folks...
15:34:47 <tflink> yep
15:34:48 <nirik> unless the CI people would take it
15:35:33 <tflink> either way works for me
15:36:24 <cverna> I would be -1 for the CPE team :)
15:36:27 <tflink> we've upgraded the instance to F31 so it'll be OK at least until the move but I think that the HW running that VM is due to be retired
15:36:52 <pingou> internally it's run on openshift, so moving to this sounds like a good idea
15:36:59 <tflink> in my mind, resultsdb is more gating than CI so it made sense for the folks doing the gating to be responsible for it. don't know if that's CPE or CI
15:37:06 <pingou> tflink: do you know who will be taking the lead on this internally?
15:37:36 <tflink> pingou: the internal deployments have been another group. it was pnt devops at one point
15:37:51 <pingou> would be nice to piggy back on them
15:37:52 <tflink> they're the ones who handled the "port" to openshift
15:38:34 <pingou> I'll take point on this
15:38:48 <nirik> thanks pingou!
15:39:13 <nirik> it would be good to decide even if we run it but ci 'owns' it upstream...
15:39:14 <pingou> tflink: this is as much about the maintenance of the deployed instance as well as development right?
15:39:14 <cverna> can we approch the Factory 2.0 team since they own greenwave, that might be a logical choice
15:39:37 <nirik> cverna: might be an option too... yeah.
15:40:34 <tflink> pingou: at a minimum, it's about the deployed instance. I honestly don't know if jskladan is planning to keep up with development
15:40:34 <smooge> #topic backlog discussion
15:40:34 <smooge> #info go over our backlog and discuss and determine priority
15:40:34 <smooge> #link https://pagure.io/fedora-infrastructure/issue/8178
15:40:34 <smooge> #info topic: provision new aarch64 builders
15:40:49 <pingou> I'll reach out to Aleksandra, Ralph and Luiz and see with them
15:41:39 <pingou> I'll ping Josef for the dev question
15:41:48 <tflink> pingou: thanks
15:41:51 <tflink> pingou++
15:41:51 <zodbot> tflink: Karma for pingou changed to 8 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:42:29 <smooge> so we have 7 aarch64 vmhosts going. We are waiting for cables ot bring up replacements in qa and cloud
15:43:01 <nirik> yeah, I am going to move more buildvm's to them in the coming days
15:43:16 <smooge> any other backlog for today?
15:43:27 <nirik> this has taken a lot longer than we hoped... but hardware... ;(
15:43:39 <smooge> #topic Community Fires Team [pingou?]
15:45:12 <smooge> ok i am not sure if I am on a netsplit or nothing to be said.. so I am going to move to openfloor
15:45:25 <smooge> #topic Open Floor
15:45:39 <smooge> ok if I am netsplit.. zodbot is still with me
15:45:40 * tflink has one hopefully quick thing for openfloor
15:45:51 <smooge> go tflink
15:46:01 <nirik> smooge: no, you are fine... people are just quiet today. ;)
15:46:03 <tflink> for https://pagure.io/fedora-infrastructure/issue/8438 I was hoping for some clarification on what you all would prefer
15:46:20 <pingou> sorry smooge was watching another window for a minute :(
15:46:23 <relrod> Just one quick note from me, I'm out next week for moving, so if anyone needs anything urgent from me, ping me by tomorrow, otherwise file tickets and assign them to me as needed ;)
15:46:27 <nirik> tflink: well, I am not sure. ;)
15:46:28 <tflink> sharing certs, consolidating instances etc.
15:47:08 <nirik> perhaps someone knows the fedora-messaging setup there and could tell us?
15:47:12 <cverna> relrod: good luck with the moving
15:47:17 * tflink is asking the folks who are responsible for the ci pipleline jenkins instance what they would prefer as well
15:47:19 <nirik> if it's not easy to share, happy to issue new ones
15:47:46 <tflink> I suspect that it comes down to more of that there are different people responsible for the different instances
15:48:55 <nirik> cverna / pingou ^ any thoughts?
15:49:06 <relrod> cverna: thanks. I suspect very cramped legs after hours and hours of driving ;)
15:49:11 <nirik> relrod: yeah, safe travels and hope the move goes well. :)
15:51:21 * cverna checks the ticket
15:51:42 <pingou> nirik: iirc, we'll still have to regenerate the certs once we've put in place the sending restrictions, right?
15:52:09 <pingou> if so, with the current setup it doesn't make much difference
15:52:18 <pingou> and we can just take the path of least resistance
15:52:38 <tflink> AFAIK, the two instances will be sending out messages on pretty much the same topics
15:52:42 <nirik> uh, I have no idea on that
15:53:20 <smooge> ok I think several of us have 0 idea on amqp needs and when things need to be regenerated etc
15:54:17 <pingou> I expect we will have to expire the current ones and generate new ones
15:54:54 <nirik> that will be a pain.
15:55:21 <nirik> ok, so I guess lets just go ahead and issue certs for this now and consolidate later if we can.
15:55:24 <pingou> a bit, but we don't have so many out of our hands
15:55:50 <smooge> ok coming up to the top of the hour. can we move the amqp needs to #fedora-admin?
15:56:06 <smooge> or we can just issue certs and close this :)
15:56:09 <smooge> alright then
15:56:28 <smooge> anything else for today?
15:56:36 <smooge> thank you all for coming this week
15:57:09 <smooge> #endmeeting