15:00:48 <smooge> #startmeeting Infrastructure (2020-01-16)
15:00:48 <smooge> #meetingname infrastructure
15:00:48 <zodbot> Meeting started Thu Jan 16 15:00:48 2020 UTC.
15:00:48 <zodbot> This meeting is logged and archived in a public location.
15:00:48 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:48 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:48 <zodbot> The meeting name has been set to 'infrastructure_(2020-01-16)'
15:00:48 <zodbot> The meeting name has been set to 'infrastructure'
15:00:48 <smooge> #chair nirik pingou relrod smooge tflink cverna mizdebsk mkonecny abompard bowlofeggs
15:00:48 <smooge> #info Agenda is at: https://board.net/p/fedora-infra
15:00:48 <zodbot> Current chairs: abompard bowlofeggs cverna mizdebsk mkonecny nirik pingou relrod smooge tflink
15:00:49 <smooge> #topic aloha
15:01:00 <nils> .hello nphilipp
15:01:01 <zodbot> nils: nphilipp 'Nils Philippsen' <nphilipp@redhat.com>
15:01:04 <nirik> morning all
15:01:32 <nils> \o
15:02:02 <ks3> .hello2
15:02:03 <zodbot> ks3: ks3 'Kevin Sandy' <kevin@digitallotus.com>
15:02:28 <cverna> mornin
15:02:35 <cverna> morning*
15:02:50 <tflink> morning
15:03:38 <karsten> .hello2
15:03:38 <zodbot> karsten: karsten 'Karsten Hopp' <karsten@bitnap.de>
15:04:22 <smooge> #topic Next chair
15:04:22 <smooge> #info magic eight ball says:
15:04:22 <smooge> #info  2020-01-16 - smooge
15:04:22 <smooge> #info  2020-01-23 - ???
15:04:23 <smooge> #info  2020-01-30 - no meeting?
15:04:52 <bowlofeggs> .hello2
15:04:53 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <rbarlow@redhat.com>
15:04:56 <smooge> who can take next weeks meeting? Or should we cancel due to everyone on travel and such
15:05:09 <bowlofeggs> i can do it if it is going to happen
15:05:18 * nirik will be out traveling
15:05:28 <cverna> I ll be around
15:05:41 <smooge> i think nirik, relrod, smooge will be out next week.
15:05:45 <cverna> but yeah I think we will have many people travelling
15:05:54 <smooge> I will put bowlofeggs down for the meeting and it can be a record short one
15:06:05 <cverna> +1
15:06:22 <smooge> thanks bowlofeggs
15:06:27 <smooge> #topic New folks introductions
15:06:27 <smooge> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:06:27 <smooge> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
15:06:48 <ernestask> .hello ekulik
15:06:49 <zodbot> ernestask: ekulik 'None' <ekulik@redhat.com>
15:06:55 <ernestask> welp, no name
15:07:11 <ernestask> in any case, hi, I’m one of the ABRT devs
15:07:11 <abompard> .hello2
15:07:13 <zodbot> abompard: abompard 'Aurelien Bompard' <aurelien@bompard.org>
15:07:21 <ernestask> I’m looking towards becoming an apprentice of yours
15:07:26 <ernestask> and working together more closely
15:07:32 <ernestask> as I need to take care of our part of infra sometimes
15:07:51 <nirik> welcome ernestask!
15:07:59 <cverna> welcome ernestask
15:08:10 * nirik notes the "None" is there if you set the 'privacy' setting in fas so it doesn't share your real name.
15:08:52 <smooge> ernestask, I will add you after the meeting
15:09:07 <smooge> #topic announcements and information
15:09:07 <smooge> #info builder reinstalls/uprgades to f31 continuing. Should be done before holiday break - kevin
15:09:07 <smooge> #info ops folks are trying a 30min ticket triage every day at 19UTC in #fedora-admin
15:09:07 <smooge> #info smooge is on PTO 2020-01-13 -> 2020-01-23
15:09:08 <smooge> #info the following people will be AFK for Devconf: cverna, pingou, mkonecny, mboddu, nirik, relrod, nils
15:09:09 <smooge> #info the following people will be AFK for FOSDEM: pingou, mboddu
15:10:00 <nirik> builders are all done now. ;)
15:10:20 * nirik forgot to update the board
15:11:04 <nirik> smooge: wait.. you are already on pto?
15:11:19 <smooge> nope
15:11:40 <nirik> 2020-01-13 was monday?
15:11:54 <smooge> #info smooge is on PTO 2020-01-18 -> 2020-01-26
15:12:01 <smooge> sorry about that
15:12:21 <smooge> 8, 6 and 3 look too much alike :P
15:12:40 <cverna> :)
15:12:55 <nirik> ok, just confused me. ;)
15:13:31 <smooge> #info all of infrastructure was updated/rebooted in the last 72 hours
15:13:40 <smooge> #info pkgs01.stg was rebuilt
15:15:06 <bowlofeggs> oh wow, we rebooted all the world's roads and bridges too?
15:15:24 <nirik> yeah, the bridges were tricky.
15:15:30 <smooge> #info Fedora Infrastructure will be moving datacenters this year. This is your only notice beyond the one we have filed in Alpha Centauri filing cabinet
15:16:08 <smooge> since we are being silly
15:16:30 <mkonecny> .hello zlopez
15:16:30 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@packetseekers.eu>
15:16:52 <smooge> I don't have any other announcements
15:16:55 <smooge> #topic Oncall
15:16:55 <smooge> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
15:16:55 <smooge> #info smooge is oncall 2020-01-09 -> 2020-01-16
15:16:55 <smooge> #info nirik is oncall 2020-01-16 -> 2020-01-23
15:16:56 <smooge> #info ??? is oncall 2020-01-23 -> 2020-01-30
15:17:11 <smooge> cverna, or bowlofeggs can you take that last week?
15:17:26 <cverna> I can
15:18:01 <nirik> .oncalltakeus
15:18:01 <zodbot> nirik: Kneel before zod!
15:18:25 <cverna> oh I will be at devconf but only the weekend so that should be ok
15:18:42 <cverna> also my travel time is much shorter
15:19:15 <smooge> ok I have updated you in the form.
15:19:20 <cverna> thanks
15:19:23 <smooge> #info ???? is oncall 2020-01-30 -> 2020-02-06
15:19:33 <smooge> if anyone wants to plan out ahead
15:19:50 <smooge> #info Summary of last week: (from current oncall )
15:20:06 <smooge> #info dedicated-solutions (proxy11) has been having network problems a lot
15:20:16 <smooge> #info smooge broke composes for EPEL
15:20:45 <cverna> what is dedicated-solutions ? what it is doing ?
15:20:57 <smooge> it is a colocation we get a server from
15:21:13 <nirik> they are in germany? I think?
15:21:15 <smooge> i think the data center is in europe
15:21:19 <cverna> ha ok :)
15:21:32 <smooge> and it has been up and down since Sunday though today it has been down the longest
15:22:20 <smooge> #info Most .oncall were from 06:00 UTC -> 07:00 UTC when smooge is not available
15:22:25 <smooge> and that is all I have
15:22:48 <smooge> #topic Monitoring discussion [nirik]
15:22:48 <smooge> #info https://nagios.fedoraproject.org/nagios
15:22:48 <smooge> #info Go over existing out items and fix
15:23:14 <nirik> lets see.
15:23:27 <nirik> the fas messages missing... we still need to fix that
15:23:44 <nirik> no fedora planet messages - we need to fix that
15:24:02 <nirik> dbgserver we need to remove from monitoring... not sure why it's still showing up
15:24:20 <nirik> qa-stg01 disk space... we need to ping tflink to look I think?
15:24:21 <smooge> ok will do so
15:24:39 * tflink will look
15:24:49 <cverna> for fas should we just wait for the new fas ?
15:24:50 <nirik> and the check fedmsg gateway ones that relrod was going to fix.
15:24:55 <tflink> IIRC, that host has needed to be deleted for a while now
15:25:18 <smooge> dbgserver still shows up in monitoring because inventory/inventory:[dbgserver_stg] defines it
15:25:23 <nirik> cverna: well, we could... but it's anoying as people change ssh keys and then our trigger to update them never fires.
15:25:39 <nirik> smooge: can you fix that/remove it?
15:25:43 <smooge> yep
15:25:44 <cverna> nirik: ok I was not sure how it was used
15:26:08 <cverna> I have added the koji wellness monitoring plugin from https://pagure.io/fedora-infrastructure/issue/6505
15:26:24 <nirik> oh cool. I had that on my backlog a long time... thanks much!
15:26:46 <cverna> it is checking if koji.fp.o works well, but nagios will report on koji01.fp.o
15:27:00 <nirik> ok
15:27:38 <nirik> anyhow I think thats it for nagios... would be nice to get it all clean someday again
15:28:09 <smooge> #topic Tickets discussion [nirik]
15:28:09 <smooge> #info https://pagure.io/fedora-infrastructure/report/Meetings%20ticket
15:29:27 <smooge> ok we are doing daily meetings on tickets so I think this section might be covered?
15:29:46 <nirik> well, we could still talk about any folks want. ;)
15:29:46 <smooge> #topic backlog discussion
15:29:46 <smooge> #info go over our backlog and discuss and determine priority
15:29:46 <smooge> #link https://pagure.io/fedora-infrastructure/issue/8178
15:29:46 <smooge> #info topic: provision new aarch64 builders
15:29:52 <smooge> oops sorry
15:29:55 <nirik> thats fine.
15:30:26 <smooge> aarch64 boxes will get cables today. I will put in a ticket for networking after that
15:30:44 <smooge> aarch64 boxes will then be able to get get built/run after that
15:31:06 <smooge> oh wait its closed
15:31:07 <nirik> cool.
15:31:25 <cverna> I would like to try and tackle https://pagure.io/fedora-infrastructure/issue/6397
15:31:27 <nirik> so those are... how many, I lost track?
15:31:48 <cverna> oops sorry I thought you were done
15:31:58 <smooge> nirik, no idea myself..
15:32:05 <nirik> cverna: thats going to be a big one. ;) needing lot of planning, but sure!
15:32:08 <smooge> sorry my brain has swapped that out
15:32:20 <nirik> I think it's 7 of them, but I can check my notes.
15:33:02 <cverna> nirik: I can start looking at the planning bits, like what needs to be done
15:33:18 <cverna> nirik: then we can probably do that in an incremental fashion :)
15:33:22 <nirik> cverna: it would be great to get it done sometime for sure.
15:33:55 <nirik> ideally if we redirect things we set a sunset on that and remove the redirects in a year or two or something (I hate having tons of redirects)
15:34:15 <cverna> yes that sounds good to me
15:34:42 <cverna> I ll try to reference what needs to move first
15:34:49 <nirik> cool
15:35:41 <smooge> cool
15:35:43 <smooge> any others?
15:36:23 <mkonecny> https://pagure.io/fedora-infrastructure/issue/8544
15:36:35 <mkonecny> I just want to announce that this one is fixed
15:36:54 <smooge> cool
15:36:57 <smooge> I can close it
15:37:05 <mkonecny> Quick fix was applied and longterm fix will be deployed next week
15:37:25 <nirik> mkonecny: cool. Thanks for looking at it.
15:37:31 <smooge> There is a topic on resultsdb ownership in the doc, but it also says to revisit after devconf
15:37:48 <smooge> so otherwise.. I think we are at open floor
15:38:02 <smooge> oh and thanks nirik for taking oncall
15:39:27 <smooge> #topic open floor
15:39:33 * tflink has one thing
15:39:36 <tflink> for open floor
15:39:42 <smooge> its your floor
15:40:15 <tflink> long story short, the keypairs for rabbitmq generated by infra can't work with jenkins. mostly because Java
15:40:32 <smooge> hahahahaha
15:40:40 * tflink was wondering if there were any preferences on where to store the things we need to convert those keypairs into
15:41:00 <cverna> is it for fedora messaging ?
15:41:02 <tflink> to be clear, it's not difficult to convert the keypairs into something that Jenkins can use
15:41:20 <tflink> cverna: yeah, to auth with the rabbitmq instances
15:41:33 <cverna> there is a jenkins plugin for fedora-messaging
15:41:36 <nirik> ah, bummer.
15:41:43 <cverna> mkonecny: do you have the link for it ?
15:41:44 <tflink> cverna: jms-messaging?
15:41:46 <mkonecny> tflink: I updated jms-messaging-plugin for rabbitmq
15:41:52 <cverna> ha yes
15:42:08 <tflink> yeah. it can't use the keypairs generated by infra without modification
15:42:11 <mkonecny> https://github.com/jenkinsci/jms-messaging-plugin RabbitMQ support is added in latest verson
15:42:26 <tflink> cool, does that include the ability to read from rabbitmq queues?
15:42:38 <mkonecny> tflink: We used this plugin for CentOS CI
15:42:48 <tflink> yeah, I'm using it as well
15:42:51 <mkonecny> tflink: It can read and publish
15:43:04 <tflink> great, the current fork I'm using can't do that
15:43:34 <mkonecny> If there will be any issue with the plugin, just let me know
15:43:38 <tflink> but we digress. should the modified keypairs be stored in the ansible private repo?
15:43:59 <smooge> i think so
15:44:37 <tflink> ok, I can add them and their passwords to the repo so they're not just sitting on my laptop
15:44:39 <nirik> we could sure.
15:44:50 <smooge> if the unmodified ones are in private then the modified ones should be
15:45:13 * tflink also needs to write up the process of getting that all to work in a public place
15:45:24 <smooge> I do need to add one more topic
15:46:14 <smooge> #topic Hardware budget season
15:46:54 <smooge> We are at the time of the year where we put in budget requests to Red Hat for replacing servers or additional capacity.
15:47:38 <cverna> bowlofeggs needs a talos II workstation to build a CI system for the infra
15:47:44 <smooge> If you know of project capacity you need additional capacity between 2020-03->2020-12 please let me know so I can put in requests
15:48:31 <smooge> ok will put that in
15:48:47 <nirik> speaking of capacity, we might want to add some nodes to our prod/stg openshift.
15:48:59 <smooge> I have a hard deadline of 2020-02-01
15:50:06 <smooge> so please give me some ideas of what and how many by then. [i just got given the deadline so my apologies for its shortness.]
15:50:10 <smooge> EOF
15:50:33 <smooge> nirik, yeah I will do so
15:50:40 <smooge> #topic Back to open floor
15:50:47 <smooge> ok anything else?
15:51:03 <nirik> well, we will need to work out how to do that. ;) I'm not sure off hand, I guess it takes a redeploy?
15:51:20 * nirik has nothing else off hand
15:51:35 <smooge> nirik, well I figure we will be redeploying to the new datacenter so setting up a OS4 there makes sense
15:52:13 <smooge> ok eol
15:52:16 <smooge> #endmeeting