15:00:23 <mkonecny> #startmeeting Infrastructure (2020-04-09)
15:00:23 <zodbot> Meeting started Thu Apr  9 15:00:23 2020 UTC.
15:00:23 <zodbot> This meeting is logged and archived in a public location.
15:00:23 <zodbot> The chair is mkonecny. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:23 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:23 <zodbot> The meeting name has been set to 'infrastructure_(2020-04-09)'
15:00:23 <mkonecny> #meetingname infrastructure
15:00:23 <zodbot> The meeting name has been set to 'infrastructure'
15:00:23 <mkonecny> #chair nirik pingou smooge cverna mizdebsk mkonecny abompard
15:00:23 <mkonecny> #info Agenda is at: https://board.net/p/fedora-infra
15:00:24 <zodbot> Current chairs: abompard cverna mizdebsk mkonecny nirik pingou smooge
15:00:24 <mkonecny> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
15:00:25 <mkonecny> #topic aloha
15:00:30 <clime> hello!
15:00:42 <pingou> Morning!
15:00:44 <Saffronique> Afternoon :)
15:00:48 <nirik> morning
15:01:02 <mkonecny> evening :-)
15:01:32 <mymindstorm> hi
15:01:41 <mkonecny> #topic New folks introductions
15:01:41 <mkonecny> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:01:41 <mkonecny> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
15:01:45 <cverna> afternoon :)
15:01:54 <mkonecny> Anybody new, who wants to say hi?
15:01:58 <tflink> morning
15:02:12 <smooge> hello
15:02:23 <austinpowered> good morning
15:02:51 <pingou> hi
15:02:59 <Saffronique> Hi all, joined this week o/
15:03:03 <pingou> welcome Saffronique
15:03:08 <mkonecny> Welcome Saffronique
15:03:17 <Saffronique> Just getting setup on systems, currently trying to get 2FA working
15:03:20 <nirik> welcome Saffronique
15:03:23 <nils> .hello nphilipp
15:03:24 <zodbot> nils: nphilipp 'Nils Philippsen' <nphilipp@redhat.com>
15:03:47 <nils> welcome!
15:04:34 <mkonecny> Saffronique: Do you need some help with 2FA?
15:04:55 <Saffronique> umm yeah Im getting an error when I try visit the production site: https://admin.fedoraproject.org:8080/totpcgiprovision/
15:05:03 <Saffronique> The staging one worked fine
15:05:30 * nirik can try and look at logs
15:05:49 <mkonecny> nirik++
15:05:51 <zodbot> mkonecny: Karma for kevin changed to 30 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:05:54 <mkonecny> Moving on
15:05:55 <mkonecny> #topic Next chair
15:05:55 <mkonecny> #info magic eight ball says:
15:05:55 <mkonecny> #info 2020-04-09 - mkonecny
15:05:55 <mkonecny> #info 2020-04-16 - ???
15:06:13 <mkonecny> Who wants to chair next meeting?
15:06:18 <mkonecny> Any volunteer?
15:07:02 <smooge> I will probably be unable to do so for the next 2 weeks
15:07:31 <mkonecny> I see nirik behind me in current chairs :-)
15:07:54 <smooge> #info 2020-04-30 - smooge
15:08:30 <mkonecny> Thanks smooge
15:08:44 <mkonecny> Still looking for volunteer for next week
15:09:09 <pingou> is the oncall taken for next week?
15:09:19 <mkonecny> No
15:09:34 <pingou> then I take both
15:09:42 <mkonecny> pingou++
15:10:01 <mkonecny> #info 2020-04-16 - pingou
15:10:19 <mkonecny> #topic announcements and information
15:10:19 <mkonecny> #info CPE Sustaining team has standups on Tuesday and Thursday at 1400 UTC in #fedora-admin - please join
15:10:19 <mkonecny> #info Fedora Infrastructure will be moving in 2020-06 from its Phoenix Az datacenter to one near Herndon Va. A lot of planning will be involved on this. Please watch out for announcements on changes.
15:10:19 <mkonecny> #info Fedora Communishift will be moving to new datacentre in April. Current downtime is expected to be from 2020-04-10 -> 2020-05-01. Please watch out for announcements on changes.
15:10:22 <mkonecny> #info Taskotron will EOL in 2020-05
15:10:55 <nirik> #info openshift stg and prod updated
15:10:58 <mkonecny> Anything else?
15:11:15 <nirik> #info final f32 freeze starts later today!
15:11:25 <mkonecny> Nice work nirik, so we are running 4.* now
15:11:37 <pingou> #info rpmautospec deployed in stg, please test it :)
15:11:40 <nirik> no. latest 3.11... had a bunch of security updates
15:11:42 <nirik> 3.11.200
15:11:49 <nirik> was 3.11.117
15:11:53 <mkonecny> Ok
15:13:12 <mkonecny> #topic Oncall
15:13:12 <mkonecny> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
15:13:12 <mkonecny> #info cverna is oncall 2020-03-26 -> 2020-04-02
15:13:12 <mkonecny> #info siddharthvipul is oncall 2020-04-02 -> 2020-04-09
15:13:12 <mkonecny> #info pingou is oncall 2020-04-09 -> 2020-04-16
15:13:13 <mkonecny> ## .oncalltakeeu .oncalltakeus
15:13:15 <mkonecny> #info Summary of last week: (from current oncall )
15:14:13 <mkonecny> siddharthvipul: The floor is yours
15:15:52 <mkonecny> It looks like he isn't here :-(
15:16:01 <cverna> I was actually on call this week
15:16:08 <siddharthvipul> mkonecny: hey, I am in a call right now :(
15:16:30 <mkonecny> cverna: Oh, so there was some mistake in the agenda then
15:16:48 <cverna> We had a problem with Openshift that required to restart the cluster thanks nirki
15:16:58 <cverna> nirik even
15:17:18 <cverna> And a lot of bodhi baby sitting
15:18:26 <cverna> Bodhi should be ok now, I have pushed a few patches that are mostly merge upstream already
15:18:36 <cverna> Eof
15:18:40 <nirik> I hope the upgrade also made the cluster more stable
15:19:30 <mkonecny> #topic Monitoring discussion [nirik]
15:19:30 <mkonecny> #info https://nagios.fedoraproject.org/nagios
15:19:30 <mkonecny> #info Go over existing out items and fix
15:22:13 <nirik> ok, lets see
15:22:35 <nirik> buildhw-aarch64-07.arm.fedoraproject.org  is down.
15:22:47 <nirik> needs a power cycle.. smooge: did we need some access issue fixed there or something?
15:23:12 <smooge> we could not get into one of the apc
15:23:15 <nirik> there are various machines with low swap... but thats a rhel8 bug that should be fixed in 8.2
15:23:39 <nirik> proxy31 registry is down, that needs us to run the oci-registry playbook to update it's firewall
15:23:55 <nirik> it would be nice to get proxy31 all done... whats left on it?
15:23:57 <smooge> ah I was rnning the wrong playbook
15:24:16 <smooge> i have been waiting to find out what was needed. arrfab fixed the ipv6
15:24:31 <nirik> what was the magic incantation there?
15:24:35 <smooge> no idea
15:24:57 <smooge> he went into aws, said.. oh , and it started working
15:25:04 <nirik> ok, will have to ask.
15:25:09 <smooge> I looked at the things I could see and didn't see anything different
15:25:18 <smooge> but he did and he got it working
15:25:19 <nirik> I'd love to just put that in before freeze if we cna.
15:25:29 <nirik> thats all on nagios.
15:25:39 <mkonecny> Thanks nirik
15:25:41 <mkonecny> #topic backlog discussion [cverna]
15:25:41 <mkonecny> #info go over our backlog and discuss and determine priority
15:25:46 <smooge> which playbook do I need to run?
15:26:29 <nirik> smooge: playbooks/groups/oci-registry.yml
15:26:34 <cverna> backlog is on the mailing list 😊
15:27:21 <smooge> i am guessing I also need to update something so it knows about proxy31
15:27:44 <mkonecny> cverna: Ok, should we remove it from the agenda?
15:28:45 <cverna> +1
15:28:49 <nirik> smooge: no it gets that from ansible vars.
15:28:54 <nirik> mkonecny: +1
15:29:34 <mkonecny> Ok, it's gone
15:29:49 <mkonecny> And we are reaching the end
15:29:50 <mkonecny> #topic Open Floor
15:30:30 <mkonecny> Anything you want to discuss?
15:30:44 <nirik> smooge: so, did we find that the dc move doc was fine to share now?
15:30:59 <pingou> nils: do you want to jump in now?
15:31:08 <nils> yup
15:31:15 <smooge> nirik, I don't know. I have removed all the things people brought up
15:31:26 <nirik> smooge: if so, can you share the link (I don't seem to have the url with the group in it?)
15:31:46 <nirik> ok, I will say it's ok to share. :)
15:32:18 <smooge> https://hackmd.io/@fedorainfra2020/rJpsA4FLL
15:32:35 <smooge> #info document covering known tasks and status https://hackmd.io/@fedorainfra2020/rJpsA4FLL
15:32:36 <nirik> ^ so thats the document smooge and I are working from on the datacenter move.
15:32:44 <nirik> it's broken out by weeks.
15:32:55 <nirik> and theres likely to be more things added to it as we go
15:33:11 <nirik> There are definitely going to be things we can use help on.
15:33:25 <nirik> testing things, deploying apps, etc
15:34:16 <mkonecny> The move is always much fun
15:34:32 <nirik> yeah, it's gonna be a 'fun' one. ;(
15:34:45 <nirik> our first shipment is next week (unless it gets canceled)
15:34:50 <nils> Has there ever been a not "fun" one?
15:34:58 <nirik> communishift and other machines that are mostly not in use go then
15:35:13 <nirik> nils: no. :)
15:35:25 <pingou> nils: yes one
15:35:31 <pingou> nils: naah, I was just kidding!
15:35:59 <nirik> anyhow, thats all I had, if anyone has questions on that, ask smooge or i.
15:37:24 <nirik> nils: you had something?
15:37:46 <nils> yeah, we had something, just not sure if this is the right forum or if this is more of a releng question
15:38:15 <nils> Anyway, we announced rpmautospec (automatic release tags and changelogs in RPMs) for staging today
15:38:24 <nils> https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/LWE4URIRWVTEZKXKP7QOK5JXFNVJRUNW/ <-- for context
15:39:07 <nils> And there's this bit that people can only use it to build for Rawhide, but not any of the stable Fedora releases because we don't have composes in staging, ergo no updates. At least that's how I understood it.
15:39:14 <nirik> great work (I haven't looked at it yet much :)
15:39:40 <nils> And the question is: is there another way to have this available in the DNF repos for stabled Fedora versions in stg?
15:39:58 <nirik> well, you could still build fine, just there's no handy repo
15:40:03 <pingou> can we build it and make it skip bodhi?
15:40:19 <pingou> (once signed)
15:40:19 <nirik> they can't just download from koji?
15:40:28 <pingou> we need it in the buildroot
15:40:42 <nils> Yeah, it needs to be mock-installable in koji.stg
15:40:57 <nirik> ah. buildroot overrides? side tags?
15:41:06 <nils> nirik, it's the mock running in Koji that wants to install it
15:42:11 <nils> i.e. we hook into the buildSRPMFromSCM task, and half of our work is done in the mock chroot by our CLI tool
15:42:25 <pingou> buildroot overrides may work
15:42:38 <nils> That may work
15:42:39 <nirik> I don't know if they work in stg, but they should...
15:42:45 <nils> we can try
15:42:52 <nirik> cool
15:42:58 <nils> is there an upper limit for how long a BRO may live?
15:43:32 <nirik> don't think so, unless bodhi has some limit hard coded
15:44:15 <nils> cool
15:44:25 <nils> we'll try that then I guess
15:44:29 <nils> thanks!
15:45:49 <mkonecny> If there is nothing else I will give you a few minutes of your time back :-)
15:45:55 <nils> :)
15:46:01 <nils> thanks for hosting, mkonecny!
15:46:02 <cverna> Thanks all
15:46:31 <clime> thasnk you mkonecny
15:46:35 <clime> *thanks
15:46:52 <mkonecny> Thanks everybody for joining this meeting
15:46:58 <mkonecny> #endmeeting