15:00:22 <cverna> #startmeeting Infrastructure (2019-09-12)
15:00:22 <zodbot> Meeting started Thu Sep 12 15:00:22 2019 UTC.
15:00:22 <zodbot> This meeting is logged and archived in a public location.
15:00:22 <zodbot> The chair is cverna. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:22 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:22 <zodbot> The meeting name has been set to 'infrastructure_(2019-09-12)'
15:00:31 <cverna> #meetingname infrastructure
15:00:31 <zodbot> The meeting name has been set to 'infrastructure'
15:00:40 <cverna> #topic aloha
15:00:50 <cverna> #chair nirik pingou puiterwijk relrod smooge tflink cverna mizdebsk mkonecny abompard bowlofeggs
15:00:50 <zodbot> Current chairs: abompard bowlofeggs cverna mizdebsk mkonecny nirik pingou puiterwijk relrod smooge tflink
15:00:51 <mizdebsk> .hello2
15:00:52 <zodbot> mizdebsk: mizdebsk 'Mikolaj Izdebski' <mizdebsk@redhat.com>
15:00:53 <nirik> morning.
15:01:02 <cverna> morning all
15:01:31 <cverna> let me try to get this right :-)
15:02:02 <cverna> #topic New folks introductions
15:02:13 <cverna> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:02:22 <cverna> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
15:02:55 <cverna> anyone new this week that would like to say hi ? :-)
15:03:43 <cverna> I ll give it couple more minutes before moving to the next topic
15:05:04 <cverna> ok let's move on
15:05:08 <cverna> #topic announcements and information
15:05:23 <cverna> #info smooge is on PTO 2019-09-07 -> 2019-09-15
15:05:23 <cverna> #info we are in F31 beta freeze
15:05:23 <cverna> #info We are looking for people to maintain Fedocal and Nuancier - mkonecny
15:06:12 <cverna> do we have anything else to add in this section ?
15:06:36 * nirik ponders
15:07:34 <nirik> probibly, but I can't think of it. :)
15:07:59 <cverna> :-)
15:08:05 <cverna> moving on then
15:08:21 <cverna> #topic Oncall
15:08:35 <cverna> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
15:08:35 <cverna> #info nirik is on call from 2019-09-05 -> 2019-09-12
15:08:35 <cverna> #info ?????? is on call from 2019-09-12 -> 2019-09-19
15:08:35 <cverna> #info Summary of last week: (from kevin )
15:08:55 <cverna> Any major fire nirik you would like to share ?
15:09:17 <nirik> Nothing too major, lots of oncall pings...
15:09:24 <nirik> we also grew a lot of tickets this week.
15:09:49 <nirik> who would like to take oncall for the next week? ;) any takers?
15:10:10 <relrod> I can if needed
15:10:20 <nirik> thanks relrod!
15:10:41 <cverna> #info relord is on call from 2019-09-12 -> 2019-09-19
15:11:20 <cverna> oh I should probably have mentioned in the info section, that pingou and I will mostly afk next week. We are both going to a training in Ireland
15:11:48 <cverna> too late :P
15:11:58 <relrod> .alias add oncall "echo relrod (Rick Elrod) is oncall. If not around, please file a ticket: https://pagure.io/fedora-infrastructure/issues"
15:12:00 <zodbot> relrod: Kneel before zod!
15:12:09 <cverna> #topic Monitoring discussion
15:12:09 <cverna> #info https://nagios.fedoraproject.org/nagios
15:12:10 <cverna> #info Go over existing out items and fix
15:12:41 <cverna> osbs is still on me, I had no time to work on it this week :(
15:13:21 * nirik looks
15:13:53 <nirik> oh yeah, I meant to look at the arm SOCs... they all just dropped off. Might be that chassis is just history now finally.
15:14:22 <nirik> we still need to fix fas fedmsgs.
15:14:50 <nirik> and I still don't know why notifs has those alerts. :)
15:15:45 <cverna> do we have a ticket for the fas fedmsgs ? we could tag that with the backlog tag and try to prioritize it
15:16:16 <nirik> already done and done. :)
15:16:22 <nirik> .ticket 7919
15:16:23 <zodbot> nirik: Issue #7919: Fix fas fedmsg sending in openshift - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/7919
15:16:48 <bowlofeggs> .hello2
15:16:49 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <rbarlow@redhat.com>
15:16:54 <bowlofeggs> (but i might disappear again)
15:17:11 <cverna> nirik: cool :-)
15:17:17 <cverna> bowlofeggs: o/
15:18:02 <bowlofeggs> wooo
15:18:07 <dustymabe> .hello2
15:18:08 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
15:18:09 <cverna> ok moving on to the next topic ? or do we have more for monitoring ?
15:18:24 <cverna> dustymabe: o/
15:18:24 <nirik> move on.
15:18:28 <cverna> #topic Tickets discussion
15:18:28 <cverna> #info https://pagure.io/fedora-infrastructure/report/Meetings%20ticket
15:19:11 <cverna> We don't have anything tagged but feel free to bring something up
15:20:52 * nirik has nothing specific. Lots of things to do...
15:21:35 <dustymabe> is this open floor ?
15:21:53 <cverna> i will be in a sec
15:22:20 <cverna> #topic Open Floor
15:22:29 <cverna> dustymabe floor is yours
15:22:44 * dustymabe waves
15:22:55 <nirik> if the floor is open, does that mean we all fall through it? :)
15:23:11 <dustymabe> nirik and I (but mostly nirik) have been trying to figure out the invalid cert setup on communishift
15:23:16 <dustymabe> https://pagure.io/fedora-infrastructure/issue/8124
15:23:24 <dustymabe> it's causing various issues throughout the cluster
15:23:40 <nirik> the problem there is that the cert-manager project is in 'terminating'
15:23:48 <dustymabe> is there anyone with some openshift expertise that might be able to help us figure out why a certain project won't finishing terminating?
15:23:52 <nirik> I haven't had time to dig more... but nothing I have tired has fixed it.
15:24:01 <nirik> tried
15:24:19 <nirik> yeah, if anyone can assist, that would be lovely.
15:25:16 <cverna> we could reach out to lgriffin, I think he mentioned that a few OpenShift SRE are sitting not far from him
15:25:29 <nirik> oh yeah! good idea.
15:25:33 <dustymabe> nirik: if we can't fix it do we have to hose the hole cluster?
15:25:39 <dustymabe> whole*
15:26:10 <nirik> probibly re-install yeah... which wouldn't take too long, but would mean everyone would have to setup their stuff again.
15:26:11 <cverna> oc apply -f nuke.yaml
15:26:50 <dustymabe> do we have a process for notifying people of maintenance on communishift ?
15:27:11 <dustymabe> i.e. send direct email to anyone with access
15:27:37 <nirik> yeah, I was planning on anytime we do that sending to communishift-members@ which should get everyone with access.
15:27:53 <dustymabe> +1
15:27:55 * nirik will reach out to leigh and see if he has anyone who can help us.
15:28:40 * dustymabe really wants to start using communishift
15:28:49 <dustymabe> but hasn't been able to get it to work so far
15:29:40 <nirik> due to the cert? or ?
15:30:02 <dustymabe> yeah cert
15:30:10 <dustymabe> at least i'm having some issues related to the cert
15:30:16 <dustymabe> and some that may or may not be related to that
15:30:28 <dustymabe> only one way to find out on that last part
15:30:29 <dustymabe> :)
15:30:43 <dustymabe> any other open floor topics?
15:30:48 <cverna> one problem at the time :)
15:31:10 <cverna> yeah we will reach out for help and update the ticket with any progress on that
15:32:53 <cverna> ok I guess the next topic is who runs the meeting next week :-)
15:32:53 <nirik> email away
15:33:17 <cverna> #topic next week meeting host
15:33:29 <dustymabe> I vote bowlofeggs
15:33:42 <dustymabe> he's my favorite leader
15:33:49 <cverna> bowlofeggs once
15:33:51 <nirik> follow the mullet!
15:33:55 <cverna> bowlofeggs twice
15:34:10 <cverna> #info bowlofeggs to run the meeting next week :P
15:34:18 <relrod> +1
15:34:27 <cverna> #undo
15:34:27 <zodbot> Removing item from minutes: INFO by cverna at 15:34:10 : bowlofeggs to run the meeting next week :P
15:34:36 <dustymabe> haha - bowlofeggs might get a vote
15:34:39 <cverna> not sure if he is available tho :P
15:35:26 <cverna> I will not be around next week, otherwise I would have been happy to run it
15:35:51 <nirik> I can do it if bowlofeggs is not available.
15:36:16 <dustymabe> either way I think we should give a shoutout to smooge for running the meeting for so long
15:36:28 <cverna> #info bowlofeggs to run the meeting next week, if not available nirik will cover :-)
15:36:46 <cverna> yeah and nirik before that :-)
15:37:02 <nirik> indeed. props to smooge!
15:37:12 <cverna> we should let you guys rest a little :-)
15:37:57 <cverna> ok any last word ? before I close this ?
15:38:08 <tflink> I have 2 questions for open floor
15:38:08 <nirik> I think it''s great to rotate things... people learn and it's new and fun...
15:38:23 <cverna> tflink go ahead
15:38:36 <cverna> #topic Open Floor the return
15:38:51 <tflink> there was talk about upgrading the centos ci openshift cluster at flock. is there a ticket tracking this work or an ETA on when we might see it happen?
15:39:07 * tflink couldn't find one in the fedora-infrastructure pagure
15:39:24 <cverna> I think siddharthvipul is going to start looking at that
15:39:28 * dustymabe would like to follow that ticket as well
15:39:58 <cverna> I don't think we have a ticket in fedora-infra but maybe there is something in the centos world of trackers
15:40:11 <nirik> yeah, not sure... siddharthvipul or bstinson might know more.
15:40:24 <tflink> I was also hoping for some clarification on communishift - is kubevirt going to be supported and is that the only method for running VMs in communishift?
15:40:48 <nirik> tflink: yes and yes...
15:40:52 <tflink> we're looking to start getting some new stuff running and would prefer to avoid reworking some base tools for managing VMs
15:41:36 <tflink> the current bits we have use qemu-kvm directly and I'm not sure if that's possible outside of running privileged containers
15:41:46 <nirik> There's a bug in the 2.0.0 version that makes networking not work with masq (the default)
15:41:56 <bstinson> wrt apps.ci.centos.org, we will open a ticket soon. we're still gathering some information
15:42:11 <tflink> bstinson: could you cc me on the ticket once it's created?
15:42:19 <nirik> I would have thought they would have a 2.0.1 out by now, but I haven't seen mention of it.
15:42:37 <tflink> nirik: is there a ticket I can follow for that issue?
15:43:17 <nirik> well, lets see... there's a bug on it, but it already got fixed upstream. let me see if there's one on the product...
15:43:23 <bstinson> tflink: sure
15:43:46 <tflink> bstinson: thanks. any guesses on when we could expect seeing changes?
15:43:55 <nirik> tflink: I did get it to boot and appear to work, just couldn't ssh into the instance. ;(
15:44:29 <tflink> nirik: yeah, that's what I recall from your presentation at flock. I was just hoping it had been fixed or that there was another option that wouldn't require large changes to our tooling
15:46:15 <nirik> I am digging on it... could also file a infra ticket for tracking the issue if you like.
15:46:51 <tflink> nirik: that works, I just figured that you'd prefer to have something other than me poking people from time to time to see if there's been a status change :)
15:46:51 <bstinson> tflink: there are some management decisions involved, so at this point i can't guess
15:47:00 <bstinson> once those are resolved, i'll be in a better place
15:47:24 <tflink> bstinson: ok, thanks for the explanation
15:47:32 <nirik> tflink: this is for taskotron? or ?
15:47:42 <tflink> nirik: some of the bits to replace taskotron
15:47:49 <nirik> ok
15:48:26 <nirik> https://bugzilla.redhat.com/show_bug.cgi?id=1725848 is the upstream bug but it's kinda a mess...
15:48:40 <nirik> (multiple issues in it, and they used it for documentation changes too)
15:48:42 <tflink> I'm proxying some concerns and issues with the apps.ci.centos.org instance, we figure that starting the new stuff on communishift would be a better place for us
15:49:07 <tflink> knowing that we'd have to move it eventually
15:49:50 <nirik> I'll try and find out latest status out of meeting...
15:49:55 <tflink> thanks
15:51:11 <tflink> nirik++ bstinson++
15:51:12 <zodbot> tflink: Karma for kevin changed to 32 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:51:15 <zodbot> tflink: Karma for bstinson changed to 6 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:51:23 <tflink> since I tend to forget about cookies :)
15:52:04 <tflink> those were my questions, they sound like they've been resolved for the meeting
15:52:49 <cverna> ok going to close the meeting then
15:52:57 <cverna> thanks all for joining
15:53:19 <cverna> #endmeeting