15:00:04 <pingou> #startmeeting Infrastructure (2019-11-21)
15:00:05 <zodbot> Meeting started Thu Nov 21 15:00:04 2019 UTC.
15:00:05 <zodbot> This meeting is logged and archived in a public location.
15:00:05 <zodbot> The chair is pingou. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:05 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:05 <zodbot> The meeting name has been set to 'infrastructure_(2019-11-21)'
15:00:07 <pingou> #meetingname infrastructure
15:00:07 <zodbot> The meeting name has been set to 'infrastructure'
15:00:09 <pingou> #chair nirik pingou relrod smooge tflink cverna mizdebsk mkonecny abompard bowlofeggs
15:00:09 <zodbot> Current chairs: abompard bowlofeggs cverna mizdebsk mkonecny nirik pingou relrod smooge tflink
15:00:11 <pingou> #info Agenda is at: https://board.net/p/fedora-infra
15:00:13 <pingou> #topic aloha
15:00:23 <pingou> .hello2
15:00:24 <zodbot> pingou: pingou 'Pierre-YvesChibon' <pingou@pingoured.fr>
15:00:32 <nils> .hello nphilipp
15:00:33 <zodbot> nils: nphilipp 'Nils Philippsen' <nphilipp@redhat.com>
15:00:43 <ks3> .hello2
15:00:44 <zodbot> ks3: ks3 'Kevin Sandy' <kevin@digitallotus.com>
15:00:54 <cverna> hello
15:00:58 * mboddu sorta here
15:01:05 <scoady> hi all
15:01:25 <nirik> morning
15:01:42 * jlanda waves
15:02:29 <tflink> morning
15:02:49 <pyratebeard> hello o/
15:02:52 <pingou> (giving it until 3 past the hour)
15:03:06 <pingou> #topic Next chair
15:03:09 <pingou> #info magic eight ball says:
15:03:11 <pingou> 2019-11-28 - holiday, NO MEETING!
15:03:13 <pingou> 2019-12-05 -smooge
15:03:20 <pingou> so no meeting next week, smooge is chairing the week after that
15:03:29 <pingou> anyone for the following week? December 12th?
15:03:51 <nirik> I can do it... haven't in a long while.
15:03:57 <pingou> okido
15:04:03 <pingou> 2019-12-12 -nirik
15:04:19 <pingou> so we have one meeting w/o chairman before the holiday shutdown
15:04:23 <pingou> December 19th
15:04:36 * nirik will be on pto by then. :)
15:04:53 <pingou> any volunteer? (Otherwise we'll find one later)
15:04:55 <smooge> magic 8 ball says bowlofeggs
15:05:09 <pingou> wfm
15:05:25 <pingou> putting a (to be confirmed) next to his name
15:05:34 <pingou> in case he cannot make the meeting
15:05:40 <pingou> #topic New folks introductions
15:05:43 <pingou> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:05:45 <pingou> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
15:05:49 <pingou> any new comers this week? :)
15:07:19 <pingou> taking this as a 'no' :)
15:07:23 <pingou> #topic announcements and information
15:07:26 <pingou> #info No meeting next week.
15:07:28 <pingou> #info work in progress in improving the script syncing assignee and CC from dist-git to bugzilla
15:07:30 <pingou> #info work done on improving the error messages sent by fedscm-admin when processing scm requests
15:07:32 <pingou> any other news for this week?
15:08:00 <jlanda> you updated stg pagure instances to 5.8 :)
15:08:00 <cverna> #info bodhi 5.1 deployed in stg
15:08:10 <nirik> #info holidays in the us next thursday and friday
15:08:21 <nirik> #info koji updated to 1.19.1 in prod
15:08:23 <pingou> #info pagure 5.8 deployed in stg for pagure.io and src.fp.o
15:09:06 <tflink> #info taskotron machines may be turned off on Tuesday unless our upgrade experiment works
15:09:27 <pingou> tflink: want to ellaborate?
15:09:50 <nirik> #info next tuesday is f29 eol day
15:10:04 <tflink> taskotron requires some features in imagefactory that nobody else uses, those features broke when imagefactory was ported to python3
15:10:53 <tflink> since we're not getting any new servers for taskotron, have been directed to not put any significant effort into it and are not running anything gating, we can't justify much effort to keep the systems alive
15:11:00 <tflink> hence turning them off on F29 EOL day
15:11:33 <tflink> there are python2 builds of imagefactory in F30, we're trying a low-effort method for upgrading in the hopes that it might allow taskotron to function until the datacenter move next year
15:11:36 <nirik> :(
15:12:06 <pingou> alright, that'll be a sad day :( but thanks for the heads-up and explanations
15:12:26 <tflink> ATM, it's kinda working and we're trying to sort out the remaining issues in our dev instance
15:12:30 <nirik> resultsdb will move anyhow right?
15:12:38 <nirik> move to f30 that is
15:12:46 <pingou> fingers crossed the upgrade worked
15:12:47 <tflink> yeah, resultsdb is not affected
15:12:49 <pingou> works*
15:13:01 <tflink> we still need to figure out who's going to own those systems going forward, though
15:13:06 <bowlofeggs> pingou: i won't be around during all of december to chair the meeting
15:13:15 <pingou> bowlofeggs: thanks for the info
15:13:25 <smooge> tflink, own what systems?
15:13:26 <tflink> AFAIK, resultsdb is running on systems that won't be moved and are running out of warranty
15:13:39 <tflink> virthost-comm03.qa IIRC
15:13:49 <smooge> ah ok.
15:13:53 * tflink checks ansible to make sure he's remembering correctly
15:14:12 <smooge> no that is correct.. most of the virthost-comm boxes are eol in Dec 2020 anyway
15:14:27 <tflink> yeah, 03
15:14:48 <nirik> we are going to replace that one soon I think.
15:15:28 <nirik> where 'soon' is q4.
15:15:31 <smooge> nirik, no. the boxes which are getting replaced are qa0x ones
15:15:57 <nirik> oh? ok, I see it's marked replace on the spreadsheet...oh... thats at move time.
15:16:14 <smooge> yeah.. if we do the proper cargo-cult procurement dance
15:16:15 <tflink> AFAIK, the internal RH resultsdb instances are running on openshift so that could be an option going forward
15:16:40 <nirik> tflink: do they need anything local? or just db for persistent?
15:17:05 <pingou> we have the ci-resultsdb-listener running on the same host
15:17:06 <tflink> there are some local settings in a single file but everything important is in the db
15:17:47 <pingou> we could move ci-resultsdb-listener to openshift but that'll mean figuring out auth to upload results to resultsdb
15:17:50 <nirik> +1 for openshift then, but will need someone to move it. ;)
15:18:18 <pingou> backlog :)
15:18:52 <tflink> the checks that are currently running in taskotron will eventually run in testing farm which is kinda part of CI
15:19:17 <tflink> that process is just taking longer than we hoped
15:19:49 <mkonecny> .hello zlopez
15:19:50 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@packetseekers.eu>
15:20:40 <pingou> anything else?
15:21:23 <pingou> #topic Oncall
15:21:26 <pingou> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
15:21:28 <pingou> #info pingou is on call 2019-11-14->2019-11-21
15:21:30 <pingou> #info kevin is on call 2019-11-21->2019-11-28
15:21:32 <pingou> #info smooge is on call 2019-11-28->2019-12-05
15:21:34 <pingou> do we have any volunteers for Dec 5th to Dec 12th?
15:22:48 <pingou> okido, we'll have to look for one later
15:23:04 <pingou> #info Summary of last week: (from current oncall )
15:23:10 <pingou> it was pretty quiet
15:23:22 <pingou> I've had one request on IRC about the fesco logs on mote
15:23:42 <pingou> the meetingname command got mangle in the same line as the startmeeting command
15:23:58 <pingou> so the file were wrongly named and not linked to the FESCo team
15:24:19 <pingou> I could fix the name and the link but I did not manage to fix the front page link (which by now should be gone)
15:24:20 <nirik> fun
15:24:31 <pingou> and we had one issue with the CI pipeline
15:24:54 <pingou> when they upgraded the plugin to send the bus notifications, they finally added the timestamp field
15:25:11 <pingou> except that they provided the timestamp in the wrong unit (miliseconds instead of seconds)
15:25:25 <pingou> and postgresql did not like timestamp corresponding to the year 51480
15:25:50 <pingou> so we'll likely have to patch postgresql to accept these timestamp sometime before we reach that year
15:25:51 <jlanda> there is time to fill the bug upstream then
15:26:10 <pingou> other than that, it was quiet for me
15:26:15 <pingou> #topic Monitoring discussion
15:26:18 <pingou> #info https://nagios.fedoraproject.org/nagios
15:26:20 <pingou> #info Go over existing out items and fix
15:26:59 * nirik is trying to recover autosign01....
15:27:09 <pingou> that's one :)
15:27:24 <pingou> nirik: do we want to skip for this week then?
15:27:43 <nirik> sure
15:27:52 <pingou> #topic Tickets discussion
15:27:55 <pingou> #info https://pagure.io/fedora-infrastructure/report/Meetings%20ticket
15:27:57 <pingou> #infohttps://pagure.io/fedora-infrastructure/issue/8397
15:28:03 <pingou> Move flocktofedora.org certs to letsencrypt
15:28:19 <pingou> smooge tagged this ticket
15:28:51 <pingou> I'm not sure what he wanted us to discuss though
15:28:54 <pingou> someone has an idea?
15:29:15 <cverna> nope
15:29:19 <nirik> dunno
15:29:22 <nirik> smooge: ^ ?
15:29:49 <smooge> i tagged it because it needs to be done soon so wanted it at least rememebred
15:30:04 <smooge> it needs to be done in the next 70 days
15:30:25 <pingou> let's move it to the backlog section then
15:30:31 <cverna> is that something an apprentice could do ?
15:30:35 <smooge> nope
15:30:39 <cverna> ok
15:30:46 <smooge> someone in sysadmin-main has to be the person I believe
15:31:15 <pingou> we also have:
15:31:16 <smooge> and yeah it would have been better as backlog
15:31:18 <pingou> #info Fedora CoreOS Team requests
15:31:20 <pingou> #link https://hackmd.io/5uB7hOJKSjGUt65iLgPnbA#Existing-requests-for-Fedora-Infra
15:31:44 <pingou> #8142 source of truth for AWS IAM permissions
15:32:00 <pingou> #8218 non-critical fedora coreos apps in communishift
15:32:07 <pingou> #8208 communishift: allow exposing /dev/kvm to containers
15:32:15 <pingou> #8281 Create fedora-cloud and fedora-testing Google Cloud projects
15:32:24 <pingou> and #8370 Fedora CoreOS + multi-arch
15:32:52 <pingou> dustymabe: anything else?
15:32:57 <pingou> (if you're around :))
15:34:31 <nirik> 8142 should be done. we talked about 8370 yesterday in releng meeting. The rest we have had no time for yet
15:35:19 <pingou> okido, thanks nirik
15:35:28 <pingou> #topic backlog discussion
15:35:31 <pingou> #info go over our backlog and discuss and determine priority
15:35:33 <pingou> #link https://pagure.io/fedora-infrastructure/issue/8178
15:35:35 <pingou> #info topic: provision new aarch64 builders
15:35:37 <pingou> #link https://pagure.io/fedora-infrastructure/issue/8157
15:35:39 <pingou> #info topic: ansible: enable ansible-report as a hook
15:35:41 <pingou> #infohttps://pagure.io/fedora-infrastructure/issue/8397
15:35:43 <pingou> #info topic: Move flocktofedora.org certs to letsencrypt
15:35:45 <pingou> (due before January 2020)
15:35:47 <pingou> the last one we just spoke about
15:35:48 <pingou> I believe there was some work on the aarch64 builders
15:36:02 <jlanda> before feb
15:36:12 <pingou> and I'm wondering if we should drop the ansible-report hook from the backlog for now
15:36:23 <nirik> thats been ongoing for weeks. It's proved fun and challenging. :) we have 1 in service... 1 ready and others in various states of install.
15:37:02 <pingou> that does sound fun :)
15:37:24 <cverna> the ansible-report can be dropped, there is enough info in the ticket for someone to run the tool and start fixing warnings. I can send an email on the list to seek volunteers to do that :)
15:37:47 <pingou> dropped
15:37:56 <pingou> #topic Community Fires Team
15:38:07 <pingou> I wanted to present the CFT a little
15:38:26 <pingou> basically, it's a team of four people currently: nils, karsten, mkonecny and I
15:38:45 <pingou> and we aim at working at small changes (less than a week) on all and any of our apps
15:39:09 <pingou> I have on my todo to send an email to the infra list presenting the CFT, what we do and how we work
15:39:23 <pingou> so hopefully we'll have something more to discuss next week :)
15:39:29 <pingou> #topic Open Floor
15:39:51 <pingou> any topic or question anyone has?
15:41:17 * nirik looks forward to fixes from the team. ;)
15:41:30 <pingou> in that case :)
15:41:33 <pingou> #endmeeting