15:00:04 #startmeeting Infrastructure (2019-11-21) 15:00:05 Meeting started Thu Nov 21 15:00:04 2019 UTC. 15:00:05 This meeting is logged and archived in a public location. 15:00:05 The chair is pingou. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:05 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:00:05 The meeting name has been set to 'infrastructure_(2019-11-21)' 15:00:07 #meetingname infrastructure 15:00:07 The meeting name has been set to 'infrastructure' 15:00:09 #chair nirik pingou relrod smooge tflink cverna mizdebsk mkonecny abompard bowlofeggs 15:00:09 Current chairs: abompard bowlofeggs cverna mizdebsk mkonecny nirik pingou relrod smooge tflink 15:00:11 #info Agenda is at: https://board.net/p/fedora-infra 15:00:13 #topic aloha 15:00:23 .hello2 15:00:24 pingou: pingou 'Pierre-YvesChibon' 15:00:32 .hello nphilipp 15:00:33 nils: nphilipp 'Nils Philippsen' 15:00:43 .hello2 15:00:44 ks3: ks3 'Kevin Sandy' 15:00:54 hello 15:00:58 * mboddu sorta here 15:01:05 hi all 15:01:25 morning 15:01:42 * jlanda waves 15:02:29 morning 15:02:49 hello o/ 15:02:52 (giving it until 3 past the hour) 15:03:06 #topic Next chair 15:03:09 #info magic eight ball says: 15:03:11 2019-11-28 - holiday, NO MEETING! 15:03:13 2019-12-05 -smooge 15:03:20 so no meeting next week, smooge is chairing the week after that 15:03:29 anyone for the following week? December 12th? 15:03:51 I can do it... haven't in a long while. 15:03:57 okido 15:04:03 2019-12-12 -nirik 15:04:19 so we have one meeting w/o chairman before the holiday shutdown 15:04:23 December 19th 15:04:36 * nirik will be on pto by then. :) 15:04:53 any volunteer? (Otherwise we'll find one later) 15:04:55 magic 8 ball says bowlofeggs 15:05:09 wfm 15:05:25 putting a (to be confirmed) next to his name 15:05:34 in case he cannot make the meeting 15:05:40 #topic New folks introductions 15:05:43 #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves 15:05:45 #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted 15:05:49 any new comers this week? :) 15:07:19 taking this as a 'no' :) 15:07:23 #topic announcements and information 15:07:26 #info No meeting next week. 15:07:28 #info work in progress in improving the script syncing assignee and CC from dist-git to bugzilla 15:07:30 #info work done on improving the error messages sent by fedscm-admin when processing scm requests 15:07:32 any other news for this week? 15:08:00 you updated stg pagure instances to 5.8 :) 15:08:00 #info bodhi 5.1 deployed in stg 15:08:10 #info holidays in the us next thursday and friday 15:08:21 #info koji updated to 1.19.1 in prod 15:08:23 #info pagure 5.8 deployed in stg for pagure.io and src.fp.o 15:09:06 #info taskotron machines may be turned off on Tuesday unless our upgrade experiment works 15:09:27 tflink: want to ellaborate? 15:09:50 #info next tuesday is f29 eol day 15:10:04 taskotron requires some features in imagefactory that nobody else uses, those features broke when imagefactory was ported to python3 15:10:53 since we're not getting any new servers for taskotron, have been directed to not put any significant effort into it and are not running anything gating, we can't justify much effort to keep the systems alive 15:11:00 hence turning them off on F29 EOL day 15:11:33 there are python2 builds of imagefactory in F30, we're trying a low-effort method for upgrading in the hopes that it might allow taskotron to function until the datacenter move next year 15:11:36 :( 15:12:06 alright, that'll be a sad day :( but thanks for the heads-up and explanations 15:12:26 ATM, it's kinda working and we're trying to sort out the remaining issues in our dev instance 15:12:30 resultsdb will move anyhow right? 15:12:38 move to f30 that is 15:12:46 fingers crossed the upgrade worked 15:12:47 yeah, resultsdb is not affected 15:12:49 works* 15:13:01 we still need to figure out who's going to own those systems going forward, though 15:13:06 pingou: i won't be around during all of december to chair the meeting 15:13:15 bowlofeggs: thanks for the info 15:13:25 tflink, own what systems? 15:13:26 AFAIK, resultsdb is running on systems that won't be moved and are running out of warranty 15:13:39 virthost-comm03.qa IIRC 15:13:49 ah ok. 15:13:53 * tflink checks ansible to make sure he's remembering correctly 15:14:12 no that is correct.. most of the virthost-comm boxes are eol in Dec 2020 anyway 15:14:27 yeah, 03 15:14:48 we are going to replace that one soon I think. 15:15:28 where 'soon' is q4. 15:15:31 nirik, no. the boxes which are getting replaced are qa0x ones 15:15:57 oh? ok, I see it's marked replace on the spreadsheet...oh... thats at move time. 15:16:14 yeah.. if we do the proper cargo-cult procurement dance 15:16:15 AFAIK, the internal RH resultsdb instances are running on openshift so that could be an option going forward 15:16:40 tflink: do they need anything local? or just db for persistent? 15:17:05 we have the ci-resultsdb-listener running on the same host 15:17:06 there are some local settings in a single file but everything important is in the db 15:17:47 we could move ci-resultsdb-listener to openshift but that'll mean figuring out auth to upload results to resultsdb 15:17:50 +1 for openshift then, but will need someone to move it. ;) 15:18:18 backlog :) 15:18:52 the checks that are currently running in taskotron will eventually run in testing farm which is kinda part of CI 15:19:17 that process is just taking longer than we hoped 15:19:49 .hello zlopez 15:19:50 mkonecny: zlopez 'Michal Konečný' 15:20:40 anything else? 15:21:23 #topic Oncall 15:21:26 #info https://fedoraproject.org/wiki/Infrastructure/Oncall 15:21:28 #info pingou is on call 2019-11-14->2019-11-21 15:21:30 #info kevin is on call 2019-11-21->2019-11-28 15:21:32 #info smooge is on call 2019-11-28->2019-12-05 15:21:34 do we have any volunteers for Dec 5th to Dec 12th? 15:22:48 okido, we'll have to look for one later 15:23:04 #info Summary of last week: (from current oncall ) 15:23:10 it was pretty quiet 15:23:22 I've had one request on IRC about the fesco logs on mote 15:23:42 the meetingname command got mangle in the same line as the startmeeting command 15:23:58 so the file were wrongly named and not linked to the FESCo team 15:24:19 I could fix the name and the link but I did not manage to fix the front page link (which by now should be gone) 15:24:20 fun 15:24:31 and we had one issue with the CI pipeline 15:24:54 when they upgraded the plugin to send the bus notifications, they finally added the timestamp field 15:25:11 except that they provided the timestamp in the wrong unit (miliseconds instead of seconds) 15:25:25 and postgresql did not like timestamp corresponding to the year 51480 15:25:50 so we'll likely have to patch postgresql to accept these timestamp sometime before we reach that year 15:25:51 there is time to fill the bug upstream then 15:26:10 other than that, it was quiet for me 15:26:15 #topic Monitoring discussion 15:26:18 #info https://nagios.fedoraproject.org/nagios 15:26:20 #info Go over existing out items and fix 15:26:59 * nirik is trying to recover autosign01.... 15:27:09 that's one :) 15:27:24 nirik: do we want to skip for this week then? 15:27:43 sure 15:27:52 #topic Tickets discussion 15:27:55 #info https://pagure.io/fedora-infrastructure/report/Meetings%20ticket 15:27:57 #infohttps://pagure.io/fedora-infrastructure/issue/8397 15:28:03 Move flocktofedora.org certs to letsencrypt 15:28:19 smooge tagged this ticket 15:28:51 I'm not sure what he wanted us to discuss though 15:28:54 someone has an idea? 15:29:15 nope 15:29:19 dunno 15:29:22 smooge: ^ ? 15:29:49 i tagged it because it needs to be done soon so wanted it at least rememebred 15:30:04 it needs to be done in the next 70 days 15:30:25 let's move it to the backlog section then 15:30:31 is that something an apprentice could do ? 15:30:35 nope 15:30:39 ok 15:30:46 someone in sysadmin-main has to be the person I believe 15:31:15 we also have: 15:31:16 and yeah it would have been better as backlog 15:31:18 #info Fedora CoreOS Team requests 15:31:20 #link https://hackmd.io/5uB7hOJKSjGUt65iLgPnbA#Existing-requests-for-Fedora-Infra 15:31:44 #8142 source of truth for AWS IAM permissions 15:32:00 #8218 non-critical fedora coreos apps in communishift 15:32:07 #8208 communishift: allow exposing /dev/kvm to containers 15:32:15 #8281 Create fedora-cloud and fedora-testing Google Cloud projects 15:32:24 and #8370 Fedora CoreOS + multi-arch 15:32:52 dustymabe: anything else? 15:32:57 (if you're around :)) 15:34:31 8142 should be done. we talked about 8370 yesterday in releng meeting. The rest we have had no time for yet 15:35:19 okido, thanks nirik 15:35:28 #topic backlog discussion 15:35:31 #info go over our backlog and discuss and determine priority 15:35:33 #link https://pagure.io/fedora-infrastructure/issue/8178 15:35:35 #info topic: provision new aarch64 builders 15:35:37 #link https://pagure.io/fedora-infrastructure/issue/8157 15:35:39 #info topic: ansible: enable ansible-report as a hook 15:35:41 #infohttps://pagure.io/fedora-infrastructure/issue/8397 15:35:43 #info topic: Move flocktofedora.org certs to letsencrypt 15:35:45 (due before January 2020) 15:35:47 the last one we just spoke about 15:35:48 I believe there was some work on the aarch64 builders 15:36:02 before feb 15:36:12 and I'm wondering if we should drop the ansible-report hook from the backlog for now 15:36:23 thats been ongoing for weeks. It's proved fun and challenging. :) we have 1 in service... 1 ready and others in various states of install. 15:37:02 that does sound fun :) 15:37:24 the ansible-report can be dropped, there is enough info in the ticket for someone to run the tool and start fixing warnings. I can send an email on the list to seek volunteers to do that :) 15:37:47 dropped 15:37:56 #topic Community Fires Team 15:38:07 I wanted to present the CFT a little 15:38:26 basically, it's a team of four people currently: nils, karsten, mkonecny and I 15:38:45 and we aim at working at small changes (less than a week) on all and any of our apps 15:39:09 I have on my todo to send an email to the infra list presenting the CFT, what we do and how we work 15:39:23 so hopefully we'll have something more to discuss next week :) 15:39:29 #topic Open Floor 15:39:51 any topic or question anyone has? 15:41:17 * nirik looks forward to fixes from the team. ;) 15:41:30 in that case :) 15:41:33 #endmeeting