15:02:29 <mobrien> #startmeeting Infrastructure (2020-11-12)
15:02:29 <zodbot> Meeting started Thu Nov 12 15:02:29 2020 UTC.
15:02:29 <zodbot> This meeting is logged and archived in a public location.
15:02:29 <zodbot> The chair is mobrien. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:29 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:02:29 <zodbot> The meeting name has been set to 'infrastructure_(2020-11-12)'
15:02:29 <mobrien> #meetingname infrastructure
15:02:29 <zodbot> The meeting name has been set to 'infrastructure'
15:02:29 <mobrien> #chair nirik pingou smooge cverna mizdebsk mkonecny abompard siddharthvipul mobrien
15:02:29 <zodbot> Current chairs: abompard cverna mizdebsk mkonecny mobrien nirik pingou siddharthvipul smooge
15:02:30 <mobrien> #info Agenda is at: https://board.net/p/fedora-infra
15:02:30 <mobrien> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
15:02:30 <mobrien> #topic aloha
15:02:33 <pingou> ó/ folks!
15:02:46 <mobrien> .hi
15:02:47 <zodbot> mobrien: mobrien 'Mark O'Brien' <markobri@redhat.com>
15:02:50 <smooge> hello
15:02:55 <siddharthvipul> good morning (ugt) y'all
15:02:59 <siddharthvipul> .hello siddharthvipul1
15:03:01 <zodbot> siddharthvipul: siddharthvipul1 'Vipul Siddharth' <siddharthvipul1@gmail.com>
15:03:51 <mkonecny> .hello zlopez
15:03:52 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@psmail.xyz>
15:04:51 <mobrien> #topic New folks introductions
15:04:51 <mobrien> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:04:51 <mobrien> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
15:05:01 <mobrien> Any new folks around?
15:05:04 <cverna> .hello
15:05:05 <zodbot> cverna: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1".
15:05:15 <cverna> .hello2
15:05:15 <zodbot> cverna: cverna 'Clement Verna' <clems.verna@gmail.com>
15:05:21 <cverna> I am a bit rusty :P
15:05:40 <siddharthvipul> cverna, enroll as new, introduce yourself :P
15:05:47 <siddharthvipul> :p
15:06:12 <mobrien> Give us an interesting fact about yourself, such as right now I'm listening to some relaxing Celine Dion
15:06:45 <pingou> :)
15:06:48 <siddharthvipul> I am using laptop on the kitchen counter and sooo looking forward to closing the laptop for a couple of years
15:07:03 <mkonecny> I just got a package from UK with new Warhammer 40k miniatures :-)
15:07:14 <pingou> siddharthvipul: years?
15:07:18 <mobrien> Oh I finally got my Fedora in the post today
15:07:23 <siddharthvipul> mkonecny, oh nice, will you be painting those or they are all set?
15:07:26 <pingou> mobrien: congrats!
15:07:35 <siddharthvipul> pingou, now that I said it., :}
15:07:43 <siddharthvipul> nov 12: no typos were made
15:07:55 <mkonecny> siddharthvipul: I love to assembly them, but most of my collection is uncolored
15:08:01 <cverna> I like celine dion too :P
15:08:27 <cverna> over wise nothing else really interesting
15:08:46 <mobrien> cverna were you disappointed when she chose to represent Switzerland rather than France in eurovision :P
15:09:03 <mobrien> anyway lets get back on track
15:09:04 <pingou> did she win?
15:09:34 <mobrien> Yep in 1988
15:09:42 <mobrien> In Ireland
15:09:44 <mobrien> ### Determine who the next chair is
15:09:44 <mobrien> #info magic eight ball says:
15:09:44 <mobrien> #info 2020-11-12 - mobrien
15:09:44 <mobrien> #info 2020-11-19 - siddharthvipul
15:09:44 <mobrien> #info 2020-11-26 - no meeting
15:09:45 <mobrien> #info 2020-12-03 - nirik
15:09:54 <nb> hi
15:09:57 <cverna> no but I have to say that I am not really a big patriot ( a bit ashamed about it)
15:09:57 <pingou> mobrien: had she sing for France, she would have lost :D
15:10:10 <cverna> pingou: lol
15:10:14 <mobrien> :D
15:10:14 <siddharthvipul> cverna, I am more french than you now :")
15:10:22 <mkonecny> I think we are pretty covered for the next few weeks
15:10:32 <mobrien> yep mkonecny
15:10:33 <siddharthvipul> actually, I might be Irish (watching father ted)
15:10:40 <siddharthvipul> mobrien, +1
15:10:55 <cverna> lol siddharthvipul you are a citizen of the world :)
15:11:02 <mobrien> I think we can skip this section for this week
15:11:06 <siddharthvipul> cverna, oh of course B)
15:11:11 <siddharthvipul> mobrien, yess
15:11:19 <mkonecny> mobrien: +1
15:11:21 <mobrien> #topic announcements and information
15:11:22 <mobrien> #info CPE Sustaining EU-hours team has a Monday through Friday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
15:11:22 <mobrien> #info CPE Sustaining NA-hours team has a Monday through Friday 30 minute meeting going through tickets at 1800 UTC in #fedora-admin
15:11:22 <mobrien> #info Datacenter move is over, but some items still need to be done: see https://fedoraproject.org/wiki/Infrastructure/2020-post-datacenter-move-known-issues
15:11:25 <mobrien> #info F33 is out! Update/install today! https://getfedora.org/
15:11:27 <mobrien> #info f33 final freeze is over!
15:11:29 <mobrien> #info no meeting on November 26th
15:11:31 <mobrien> Anyone have any announcements?
15:11:56 <smooge> #info Updates and Outages completed yesterday
15:12:08 <mkonecny> #info New version of the-new-hotness 0.13.2 deployed in production
15:12:19 <mobrien> smooge++
15:12:22 <mobrien> mkonecny++
15:12:22 <zodbot> mobrien: Karma for zlopez changed to 2 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:12:25 <mobrien> nirik++
15:12:32 <smooge> nothing else on my part
15:13:18 <siddharthvipul> #info Fedora CoreOS happening right, participate : https://fedoramagazine.org/fedora-coreos-test-day/
15:13:24 <siddharthvipul> #undo
15:13:24 <zodbot> Removing item from minutes: INFO by siddharthvipul at 15:13:18 : Fedora CoreOS happening right, participate : https://fedoramagazine.org/fedora-coreos-test-day/
15:13:30 <siddharthvipul> #info Fedora CoreOS test day happening right, participate : https://fedoramagazine.org/fedora-coreos-test-day/
15:13:51 <nb> #info there are badges for people who participate in the CoreOS test day!
15:14:04 <cverna> \o\
15:14:16 <siddharthvipul> I like me some badges :}
15:14:52 <cverna> and cookies :)
15:15:05 <siddharthvipul> yess
15:15:06 <mobrien> it finishes today right?
15:15:12 <pingou> siddharthvipul++
15:15:13 <siddharthvipul> mobrien, yeah, I think so :(
15:15:14 <zodbot> pingou: Karma for siddharthvipul1 changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:15:28 <siddharthvipul> wooww, here is my first cookie of this release
15:15:39 <siddharthvipul> thank you pingou :D
15:16:00 <mobrien> ok if no more announcements lets move on to ....
15:16:11 <mobrien> #topic Oncall
15:16:11 <mobrien> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
15:16:11 <mobrien> #info mkonecny  is oncall for 2020-11-06 to 2020-11-12
15:16:11 <mobrien> #info mobrien  is oncall for 2020-11-12 to 2020-11-19
15:16:11 <mobrien> #info pingou  is oncall for 2020-11-19 to 2020-11-26
15:16:12 <mobrien> ## .oncalltakeeu .oncalltakeus
15:16:31 <mobrien> .oncalltakeeu
15:16:34 <zodbot> mobrien: Error: You don't have the alias.add capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified.
15:16:39 <siddharthvipul> mobrien, before you ask, you can assign me the next one
15:16:45 <siddharthvipul> ^^
15:16:52 <mobrien> Always forget to register with zodbot
15:16:56 <mobrien> thanks siddharthvipul
15:17:40 <mobrien> #info siddharthvipul  is oncall for 2020-11-26 to 2020-12-03
15:18:01 <mobrien> #info Summary of last week: (from current oncall )
15:18:06 <mobrien> mkonecny, take it away
15:18:07 <nb> .capabilities mobrien
15:18:23 <mkonecny> I didn't received any ping
15:18:33 <nb> yep, mobrien has alias.add, so you might need to /msg zodbot identify yourpassword
15:18:49 <nb> you can also use hostmask list and hostmask add to set it to automatically recognize your hostname
15:18:54 <mobrien> nb yep, always forget about that thanks
15:18:57 <mkonecny> So it was quiet week
15:19:43 <mobrien> .oncalltakeeu
15:19:43 <zodbot> mobrien: Kneel before zod!
15:19:55 <mobrien> thanks for the update mkonecny
15:20:17 <mobrien> #topic Monitoring discussion [nirik]
15:20:17 <mobrien> #info https://nagios.fedoraproject.org/nagios
15:20:17 <mobrien> #info Go over existing out items and fix
15:20:37 <nb> mobrien I fixed your hostmasks I think, try the oncalltakeeu again
15:20:49 <mobrien> .oncalltakeeu
15:20:49 <zodbot> mobrien: Kneel before zod!
15:20:56 <mobrien> thanks nb
15:20:59 <mobrien> nb++
15:20:59 <zodbot> mobrien: Karma for nb changed to 8 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:21:05 <nb> it should automatically recognize you as long as your ip doesn't change
15:21:10 <pingou> nb++
15:21:10 <zodbot> pingou: Karma for nb changed to 9 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:21:13 <pingou> thanks :)
15:21:18 <mobrien> I don't think nirik is her for nagios
15:21:27 <pingou> +e
15:21:40 <mobrien> There is a few long running alerts we are aware of
15:22:11 <mobrien> A cert needs to be replaced for whatcanidoforfedora.org which will clear a lot of warnings
15:22:39 <mobrien> looks as though there is no ill effects of the mass update/reboot which is great
15:22:51 <mkonecny> mobrien: I noticed the amount of alerts
15:23:16 <mobrien> mkonecny, they are all for one cert, it just happens to be on all the proxies
15:23:43 <mkonecny> Yeah, I noticed this
15:24:10 <mobrien> sorry there is on for busgateway01 that might need attention
15:24:31 <pingou> seems to be a letsencrypt cert
15:25:13 <pingou> there seems to have been no koji build announced on the bus for a few hours
15:25:15 <pingou> which is odd
15:25:22 <mobrien> I'm not sure of the process to update those certs so I guess I'll create a ticket and try find out
15:25:53 * pingou looking if there is anything in ansible about it
15:26:57 * pingou may have found it
15:27:19 <pingou> fingers crossed
15:27:32 <mobrien> thanks pingou
15:28:06 <mobrien> lets move on with the meeting. Next is ....
15:28:27 <mobrien> Well it should be the learning topic but nirik isn't here
15:28:54 <mobrien> If anyone would like to present soemthing they are wlecome but otherwise I'll move on in 2 mins
15:29:31 <mkonecny> What was the topic of the last CPE Tech talk
15:30:56 <mkonecny> scoady talked about graphql
15:31:38 <mkonecny> And before lrossett was talking about mbbox
15:31:52 <mobrien> I don't think he is here so we'll move on
15:31:54 <mkonecny> I can talk about the mbbox if anyone is interested
15:32:02 <mobrien> sure mkonecny
15:32:08 <mobrien> any chance to learn
15:32:25 <mkonecny> Mbbox is hosted here https://github.com/fedora-infra/mbbox
15:33:07 <nirik> oh shoot. I thought we moved the meeting an hour later. ;( but I guess we never really decided that/announced it.
15:33:31 <mkonecny> It is Kubernetes operator for easy deployment of module building system
15:33:46 <mkonecny> Thus the name Mobule Building in a Box
15:33:49 <mobrien> I think it was just the NA standup nirik, I was going to suggest it for this one on open floor
15:34:51 <pingou> mobrien: I think we spoke about moving this one last week
15:35:08 <mkonecny> The operator let you configure and deploy koji-builder, koji-hub, kojira and MBS - both frontend and backend
15:35:09 <pingou> I guess we somehow missed applying it ^^
15:35:15 * nirik will save timing to the end, let mkonecny give his talk. ;)
15:36:01 <mkonecny> We decided to implement it in ansible
15:36:26 <mkonecny> For the Kubernetes operator there are few options: GO, Ansible or Helm
15:36:51 <mkonecny> We decided to use Ansible for few reasons
15:36:54 <mkonecny> We know it
15:37:30 <mkonecny> Koji is written in Python same as ansible, so the integration was much easier
15:38:42 <mkonecny> For the testing we are using molecule, which is trying to run the deployment in development environment and let you check if everything is deployed as expected
15:39:45 <mkonecny> The operator still needs some polishing, but we aim to have whole module building pipeline deployed with it
15:41:10 <nirik> pretty cool.
15:41:13 <mkonecny> The operator have multiple ansible roles, which corresponds to each component + 1 for deploying shared parts like persistent volume
15:42:30 <mkonecny> The most difficult part when we started to work on the roles was to have koji-hub running in container
15:43:17 <mkonecny> But every component is running in their own pod in the end, sharing few resources among themselves
15:43:59 <mkonecny> Shared resources are koji-database, which is used by almost every component except frontend for MBS
15:44:33 <mkonecny> And the other is persistent volume for koji, which is used by every koji component
15:45:28 <mkonecny> Manual deployment process is described in the documentation, so if anybody wants to try it, you can https://mbbox-operator.readthedocs.io/en/latest/
15:46:58 <mkonecny> There are few things that are still missing, namely kerberos authentication, adding it to operator hub, dist-git, automatic updates and automatic deployment
15:47:38 <mkonecny> And I think this is everything I can remember about it right now
15:48:12 <mkonecny> Any questions?
15:48:52 <mobrien> Sounds like a lot of work has gone into this. well done
15:49:08 <nirik> mkonecny: are builders external? or part of this?
15:49:45 <mkonecny> mobrien: We started with python deployment scripts, so we had at least some idea what we need to do
15:50:14 <nirik> oh, I see you say builder is in there, missed it.
15:51:06 <mkonecny> nirik: koji-builder is in there, but for what I hear it is only used when requesting the new repo
15:52:06 <mkonecny> The MBBox is now used in CentOS stream build pipeline and they probably have external builders doing the heavy lifting
15:53:07 <mkonecny> Any other question?
15:54:30 <smooge> not from me
15:54:36 <mobrien> Thanks for slotting in mkonecny it was very informative
15:54:41 <mobrien> mkonecny++
15:54:43 <smooge> thanks for the talk on short notice
15:54:50 <smooge> mkonecny++
15:54:51 <zodbot> smooge: Karma for zlopez changed to 3 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:55:17 <mobrien> ok I guess we'll move to open floor
15:55:18 <mobrien> #topic Open Floor
15:55:19 <mkonecny> Hm, cookies :-)
15:56:10 <mkonecny> Should we move the meeting a hour late?
15:57:36 <mobrien> Yes I think so. It was suggested to move it to EDT I think so it would change with the clocks
15:58:18 <mkonecny> mobrien: We already moved our morning meeting to Europe/Paris timezone as well
15:59:10 <smooge> so nirik was going to briong up moving the meeting but he is dealing with a build outage
15:59:46 <mobrien> Looks like its already changed to 16:00UTC in fedocal so I'll just change the board
15:59:56 <pingou> and the team's calendar
15:59:57 <smooge> personally I think that we will be back to putting everything back in UTC in about a year because people will wonder why it isn't
16:00:08 <pingou> ^^
16:00:23 <smooge> I think this is the 3rd time we have swapped back and forth
16:00:24 <nirik> someday we can dream... of dst dying
16:00:38 <mkonecny> I have one clock in my house set to UTC, because it's much better :-)
16:00:45 <mobrien> nirik, its likely to die in the EU next year
16:00:45 <smooge> I don't really care anymore
16:00:55 <mkonecny> mobrien: I hope so
16:00:56 <smooge> well likely to die to 20 different timezones
16:01:21 <mobrien> https://www.timeanddate.com/time/europe/eu-dst.html
16:01:29 * smooge goes to grump somewhere else
16:01:39 <mkonecny> smooge: I hope not, but it's possible, every EU nation could decide if they want summer time or not
16:01:40 <nb> nirik yeah, i wish there was DST all year
16:02:26 <mkonecny> I would be rather to stay in correct UTC+x timezone
16:02:43 <nirik> I don't care as long as it never changes.
16:02:44 <mobrien> The one issue with the EU directive is that each country decides which they will keep DST or not
16:02:48 <mkonecny> Or just get rid of timezones and just use UTC :-D
16:03:44 <mobrien> ok I guess we are out of time
16:03:51 <mobrien> thanks all for coming
16:03:56 <mobrien> #endmeeting