16:00:12 <siddharthvipul> #startmeeting Infrastructure (2021-07-22)
16:00:12 <zodbot> Meeting started Thu Jul 22 16:00:12 2021 UTC.
16:00:12 <zodbot> This meeting is logged and archived in a public location.
16:00:12 <zodbot> The chair is siddharthvipul. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:12 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:12 <zodbot> The meeting name has been set to 'infrastructure_(2021-07-22)'
16:00:14 <siddharthvipul> #meetingname infrastructure
16:00:14 <zodbot> The meeting name has been set to 'infrastructure'
16:00:16 <siddharthvipul> #chair nirik siddharthvipul mobrien zlopez pingou bodanel dtometzki jnsamyak computerkid
16:00:16 <zodbot> Current chairs: bodanel computerkid dtometzki jnsamyak mobrien nirik pingou siddharthvipul zlopez
16:00:18 <siddharthvipul> #info Agenda is at: https://board.net/p/fedora-infra
16:00:20 <siddharthvipul> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:00:22 <siddharthvipul> #topic greetings!
16:00:31 <mobrien> .hi
16:00:32 <zodbot> mobrien: mobrien 'Mark O'Brien' <markobri@redhat.com>
16:00:38 <siddharthvipul> Good morning (ugt) everyone
16:00:38 <jrichardson> .hi
16:00:38 <zodbot> jrichardson: jrichardson 'James Richardson' <jamricha@redhat.com>
16:00:40 <siddharthvipul> mobrien: hey hey
16:00:40 <eddiejennings> .hi
16:00:41 <zodbot> eddiejennings: eddiejennings 'Eddie Jennings, Jr.' <eddie@eddiejennings.net>
16:00:42 <siddharthvipul> jrichardson: \o
16:00:44 <jnsamyak> .hello jnsamyak
16:00:45 <zodbot> jnsamyak: jnsamyak 'Samyak Jain' <samyak.jn11@gmail.com>
16:00:50 <siddharthvipul> eddiejennings: jnsamyak \o/
16:00:52 <bcapper> .hi
16:00:53 <zodbot> bcapper: Sorry, but you don't exist
16:00:54 <jrichardson> siddharthvipul: o/
16:00:55 <siddharthvipul> how are we doing today
16:00:57 <ekidney_> hi!
16:00:59 <mkonecny> .hello zlopez
16:01:00 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@psmail.xyz>
16:01:06 <Southern_Gentlem> .hello jbwillia
16:01:06 <lenkaseg> .hello lenkaseg
16:01:06 <zodbot> Southern_Gentlem: jbwillia 'Ben Williams' <vaioof@gmail.com>
16:01:09 <siddharthvipul> bcapper and ekidney_ welcome :D
16:01:09 <eddiejennings> Busy in all directions
16:01:09 <zodbot> lenkaseg: lenkaseg 'Lenka Segura' <lenka@sepu.cz>
16:01:24 <siddharthvipul> lenkaseg: Southern_Gentlem, greetings :D
16:01:35 <petebuffon[m]> .hello petebuffon
16:01:36 <zodbot> petebuffon[m]: petebuffon 'Peter Buffon' <pabuffon@gmail.com>
16:01:39 <bcapper> thanks :)
16:01:48 <siddharthvipul> mkonecny: and petebuffon[m], so good to see so many people
16:02:10 <petebuffon[m]> hey all :-)
16:02:16 <siddharthvipul> let's wait for a couple of minutes for folks to do roll call / join late ;)
16:02:47 <siddharthvipul> .hello siddharthvipul1
16:02:48 <zodbot> siddharthvipul: siddharthvipul1 'Vipul Siddharth' <siddharthvipul1@gmail.com>
16:03:50 <siddharthvipul> #topic New folks introductions
16:03:52 <siddharthvipul> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:03:54 <siddharthvipul> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:04:16 <siddharthvipul> alright, do we have anyone new here? I am not sure if ekidney_ and bcapper are joining this meeting for the first time or they joined the last one (I missed it)
16:04:31 <bcapper> I was at the last
16:04:56 <ekidney_> this is my first time! I'm an intern :D
16:05:11 <siddharthvipul> oh perfect :) If you have any doubts around Infra, please don't hesitate to ping ^-^
16:05:23 <darknao> .hi
16:05:24 <zodbot> darknao: darknao 'Francois Andrieu' <naolwen@gmail.com>
16:05:40 <siddharthvipul> ekidney_: welcome :)
16:05:50 <siddharthvipul> for others, ekidney_ is an intern in CPE team
16:05:56 <siddharthvipul> darknao: \o
16:06:02 <jnsamyak> ekidney_: Welcome, and yay!
16:06:09 <siddharthvipul> hey hey jnsamyak
16:06:21 <lenkaseg> welcome ekidney_ !
16:06:29 <jnsamyak> \o/
16:06:39 <ekidney_> thanks everyone :)
16:06:44 <siddharthvipul> ekidney_: just like bcapper, please don't hesitate to ping me (or generally in fedora-admin) if you have any doubts
16:06:52 <jrichardson> hi ekidney_!
16:07:10 <siddharthvipul> great! let's move ahead in the agenda then!
16:07:26 <siddharthvipul> #topic Next chair
16:07:28 <siddharthvipul> #info magic eight ball says:
16:07:30 <siddharthvipul> #info chair 2021-07-29 -  petebuffon
16:07:32 <siddharthvipul> #info chair 2021-08-05 - jnsamyak
16:07:59 <siddharthvipul> I still think we don't need a volunteer for 2021-08-12
16:08:02 <jrichardson> thanks again for taking next week, damn holidays getting in the way
16:08:13 <jrichardson> ill take that then
16:08:20 <mobrien> 2 ahead is plenty
16:08:25 <siddharthvipul> and from next week, we can start having volunteers :D
16:08:31 <jrichardson> okay, np
16:08:31 <siddharthvipul> mobrien: yess
16:08:40 <petebuffon[m]> jrichardson: no worries
16:08:52 <siddharthvipul> also, dtometzki couldn't join today so I had to take over
16:09:05 <siddharthvipul> let's move then
16:09:45 <siddharthvipul> #topic announcements and information
16:09:47 <siddharthvipul> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:09:49 <siddharthvipul> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:09:51 <siddharthvipul> #info If your team wants support from the Fedora Program Management Team, file an isssue: https://pagure.io/fedora-pgm/pgm_team/issues?template=support_request
16:09:53 <siddharthvipul> #info nest with Fedora CFP is open for one day! https://communityblog.fedoraproject.org/announcing-dates-cfp-for-nest-with-fedora/ (Aug 5th-8th)
16:09:55 <siddharthvipul> #info mass update/reboot of Fedora Servers happened last week week (2021-07-05 - 2021-07-09)
16:09:57 <siddharthvipul> #info nirik/kevin is out this week.
16:09:59 <siddharthvipul> #info mass rebuild for f35 last week (Yesterday)
16:10:01 <siddharthvipul> #info Nest with Fedora registration is open: https://hopin.com/events/nest-with-fedora-2021
16:10:31 <mobrien> I think you can take out the mass reboot info
16:10:33 <siddharthvipul> so do we have anything else?
16:10:42 <siddharthvipul> mobrien: yep, just put it for yesterday
16:10:48 <siddharthvipul> that's removed now 0:)
16:10:58 <mobrien> +1
16:11:43 <siddharthvipul> I will wait for a minute or 2 to see if we have any more announcements
16:12:24 <siddharthvipul> mobrien: what are you watching this weekend :P
16:13:00 <mobrien> I haven't watched loki yet so might give that a go if I find time
16:13:13 <siddharthvipul> mobrien: A+ choice
16:13:25 <eddiejennings> What are these "movies?"  Weekends are for RHCE prep :P
16:13:44 <siddharthvipul> eddiejennings: ah, if you want to talk tech, we will have to wait for open floor lol
16:13:45 <siddharthvipul> XD
16:14:01 * eddiejennings laughs.
16:14:01 <siddharthvipul> eddiejennings: seriously though, all the best for prep
16:14:10 <siddharthvipul> #topic Oncall
16:14:12 <siddharthvipul> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:14:14 <siddharthvipul> #info siddharthvipul is on call for 2021-07-22 to 2021-07-29
16:14:29 <siddharthvipul> okay, so we are looking for a volunteer from 29-07-29 to 2021-08-05
16:14:37 <siddharthvipul> do we have anyone interested?
16:15:11 <mobrien> I can do it if no other volunteer
16:15:27 <siddharthvipul> mobrien: I was about to tap your shoulders anyway ;)
16:15:31 <siddharthvipul> thank you sincerely
16:15:41 <siddharthvipul> #info mobrien is on call for 29-07-29 to 2021-08-05
16:16:33 <siddharthvipul> do we have anyone for 2021-08-05 to 2021-08-12
16:16:48 <siddharthvipul> it's not urgent as we have next week covered.. but having 2 weeks sorted is ideal imo
16:17:02 <siddharthvipul> I will wait for 30 seconds before moving ahead :)
16:17:20 <darknao> siddharthvipul: you can sign me in
16:17:28 <darknao> for 08-05
16:17:48 <siddharthvipul> darknao: thank you :D it's yours
16:18:14 <siddharthvipul> #info darknao is on call for 2021-08-05 to 2021-08-12
16:18:20 <siddharthvipul> okay, so who is this week
16:18:22 <siddharthvipul> oh, me!
16:18:26 <siddharthvipul> .oncalltakeeu
16:18:26 <zodbot> siddharthvipul: Error: You don't have the alias.add capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified.
16:18:31 <siddharthvipul> sigh
16:18:53 <mobrien> zodbot register
16:18:53 <jnsamyak> Kneel before Zod!
16:18:54 <zodbot> mobrien: Error: That operation cannot be done in a channel.
16:18:58 <jnsamyak> XD
16:19:00 <siddharthvipul> .oncalltakeeu
16:19:00 <zodbot> siddharthvipul: Kneel before zod!
16:19:21 <siddharthvipul> jnsamyak: lol
16:19:24 <siddharthvipul> .fire jnsamyak
16:19:24 <zodbot> adamw fires jnsamyak
16:19:35 <siddharthvipul> you can't mock zodbot haha
16:19:44 <siddharthvipul> also, sorry Adam ;)
16:20:02 <jnsamyak> omg :3
16:20:04 <siddharthvipul> #info Summary of last week: (from current oncall)
16:20:31 <siddharthvipul> dtometzki: from logs, you were on call duty for last week
16:21:04 <siddharthvipul> How was the week? I am curious if mass reboot a couple of weeks ago and mass build a day before had any impact
16:22:07 <mobrien> I did see one the other day related to login, I don't think dtometzki is here to answer
16:22:20 <siddharthvipul> cool cool
16:22:44 <siddharthvipul> dtometzki: if you noticed anything major, feel free to send that on infra list when you some time! otherwise I am guessing all was well :D
16:22:53 <siddharthvipul> #topic Monitoring discussion
16:23:15 <mobrien> I can take this
16:23:17 <siddharthvipul> since nirik is out, mobrien, would you be able to shed some light on our monitoring stuff? what's triggering the alarm now :P
16:23:26 <siddharthvipul> mobrien: all yours
16:24:15 <mobrien> So we have some issues with proxies which as of yet we haven't found a root cause for. it was proxy14 all week but proxy31 seems to have joined the party all of a sudden
16:25:07 <mobrien> Aside from that we have a few disk space issues which will mean deleting of cruft most likely
16:25:26 <mobrien> otherwise nothing new by the looks of it
16:25:41 <siddharthvipul> mobrien: thank you, we okay to move?
16:25:52 <mkonecny> I had really slow update yesterday, is this related?
16:26:37 <mobrien> Zlopez: Fedora update?
16:26:46 <mkonecny> Yes, dnf had speed around 100kB/s
16:27:20 <mobrien> That would be separate issue, the download mirrors are not behind our proxies
16:27:48 <mkonecny> Ok, thanks, I was just curious
16:27:55 <mobrien> I don't think either of those proxies are EU either
16:28:00 <siddharthvipul> thank you mkonecny for bringing that up
16:28:27 <siddharthvipul> so, let's move to more exciting stuff than nagios ;)
16:28:37 <siddharthvipul> #topic Learning topic
16:28:45 <siddharthvipul> #info 2021-07-22 - petebuffon[m], Intro kubernetes (contd.)
16:28:52 <siddharthvipul> petebuffon[m]: how are you doing today?
16:28:52 <mobrien> what could possibly be more exciting than nagios?
16:29:08 <siddharthvipul> mobrien: some times even jenkins is more exciting than nagios :3
16:29:17 <petebuffon[m]> I'm doing well!  Let's see if I can keep this talk to around 10 minutes haha
16:29:21 <eddiejennings> It's on my list of things (Nagios) to eventually deploy at home for tinkering :P
16:29:50 <siddharthvipul> eddiejennings: we don't want to hate yourself.. we are working on Zabbix+prometheus for Fedora Infra monitoring.. we can talk that in open floor :)
16:30:04 <siddharthvipul> petebuffon[m]: all yours now
16:30:10 <petebuffon[m]> Alright last week we covered containers which naturally progresses into talking about Kubernetes!
16:30:28 <petebuffon[m]> Also side note, most of this is going to be broad strokes FYI
16:30:46 <petebuffon[m]> Kubernetes, or K8s, an open-source container-orchestration system. K8s was designed by Google and is now maintained by the Cloud Native Computing Foundation. K8s defines a set of building blocks  which collectively deploy, maintain, and scale applications.
16:31:12 <petebuffon[m]> K8s was designed with scale in mind and allows applications to be deployed across clusters of systems with high availability. Each week Google starts over several billion containers using Kubernetes.
16:31:30 <petebuffon[m]> K8s consists of a Control Plane, one or more systems responsible for managing the cluster.
16:31:41 <petebuffon[m]> The control plane coordinates all activities in the cluster, including scheduling applications, maintaining applications’ desired state, scaling applications, and rolling out new updates.
16:31:56 <petebuffon[m]> K8s also contains nodes, VMs or physical computers that serve as worker machines in a K8s cluster.
16:32:10 <petebuffon[m]> Each node has a Kubelet, an agent for managing nodes, and communicating with the control plane.
16:32:15 <petebuffon[m]> Each node uses a container runtime, responsible for running containers.
16:32:27 <petebuffon[m]> Kubernetes supports several runtimes including Docker, containerd, and CRI-O.
16:33:00 <petebuffon[m]> Various plugins, such as Flannel, are used provide an overlay network for Kubernetes.
16:33:07 <petebuffon[m]> Much like Podman, Kubernetes has units called Pods, which contain one or more containers.
16:33:17 <petebuffon[m]> Every Pod gets its own IP address.
16:33:28 <petebuffon[m]> Pods on a node can communicate with all pods on all nodes without NAT.
16:34:06 <petebuffon[m]> Deployments are declarative states of Pods.
16:34:14 <petebuffon[m]> Deployments can be scaled up or down, rolled back, and updates rolled out to one Pod at a time.
16:34:22 <petebuffon[m]> Services are an abstract way to expose an application running on a set of Pods as a network service.
16:34:32 <petebuffon[m]> Kubernetes assigns the service an IP address.
16:34:41 <petebuffon[m]> Kubernetes can also do load balancing, but more complex ingress controller plugins exist, including the Nginx ingress controller.
16:34:56 <petebuffon[m]> Volumes abstract storage for Kubernetes.
16:35:06 <petebuffon[m]> Many different types of volumes are supported, nfs, cephfs, glusterfs, and hsotPath just to name a few.
16:35:20 <petebuffon[m]> Essentially an application is deployed on K8s by telling the control plane to start the application containers. The control plane schedules the containers to run on the cluster’s nodes.
16:35:46 <petebuffon[m]> K8s can be installed on many of the most common Linux distributions by running the [Kubeadm](https://kubernetes.io/docs/reference/setup-tools/kubeadm/) tool.
16:36:01 <petebuffon[m]> There are also many K8s distributions including Openshift, Canonical Kubernetes, and Rancher.
16:36:31 <petebuffon[m]> One of the easiest way to get started with K8s is to install [Minikube](https://minikube.sigs.k8s.io/docs/start/), a local single node K8s.
16:36:46 <petebuffon[m]> Minikube can be installed either in a VM or container and is relatively light on resources, 2CPUs+, 2GB memory+, 20GB disk space+.
16:36:57 <petebuffon[m]> FYI the Podman driver is still experimental and I haven’t been able to get it to work.
16:37:15 * petebuffon[m] < https://libera.ems.host/_matrix/media/r0/download/libera.chat/7d26ff92ecfaba2ed2f58cb7fa2231040c23ee0f/message.txt >
16:37:59 <petebuffon[m]> Finally I just discovered there is a Fedora Kubernetes package, but I haven't messed around with it yet.
16:38:24 <petebuffon[m]> Links
16:38:24 <petebuffon[m]> https://kubernetes.io/docs/home/
16:38:24 <petebuffon[m]> https://fedoramagazine.org/kubernetes-on-fedora-iot-with-k3s/
16:38:24 <petebuffon[m]> https://fedoramagazine.org/introduction-kubernetes-fedora/
16:38:54 <petebuffon[m]> https://fedoramagazine.org/introduction-kubernetes-fedora/
16:39:15 <petebuffon[m]> I'm just now realizing talking about Kubernetes might make you sound like a crazy person.
16:39:23 <eddiejennings> Ha!
16:39:44 <petebuffon[m]> That was definitely a big wall of text! And my 1000 foot view of kubernetes
16:39:49 <jnsamyak> <petebuffon[m] "I'm just now realizing talking a"> Been there, done that!
16:40:00 <siddharthvipul> petebuffon[m]: heard someone say k8s is the cryptocurrency of devops :P
16:40:18 <siddharthvipul> I love k8s though (elegant architecture to how extensible the kubeapi is)
16:40:21 <petebuffon[m]> siddharthvipul: that sounds like a fair comparison!
16:40:26 <siddharthvipul> hahaha
16:40:44 <eddiejennings> physical host > hypervisor > VM > containers within VM.  How far down toward the physical host can K8s manage?
16:40:48 <petebuffon[m]> I'm still trying to convince myself to deploy a k8s cluster at home
16:41:13 <petebuffon[m]> eddiejennings: I think you're limited just by computing power?
16:41:28 <siddharthvipul> when I was learning more about kubernetes, this blog helped me understand basics pretty well: https://jvns.ca/blog/2017/06/04/learning-about-kubernetes/
16:41:52 <eddiejennings> I'm generally ignorant on containers, so my question might not be rooted in reality.
16:42:45 <petebuffon[m]> eddiejennings: So many layers of abstraction.
16:42:46 <siddharthvipul> eddiejennings: I believe k8s is mainly focused on scalibility and reliablility over performance. In the end it's containers orchestration and would depend on nodes and how all are connected together
16:44:22 <eddiejennings> Ah, so it's just touching (creating, destroying, configuring) containers.
16:44:37 <petebuffon[m]> The way I understand it k8s is a big lego set with different components
16:44:52 <petebuffon[m]> And yes a very complicated way of doing stuff with containers
16:44:54 <siddharthvipul> eddiejennings: kinda of.. along with all other component of networking, storage, monitoring etc etc
16:45:00 <eddiejennings> Clearly more research needs to be done by me :)
16:46:10 <siddharthvipul> anymore questions for petebuffon[m] ?
16:46:11 <lenkaseg> siddharthvipul: +1 for Julia Evans :)
16:46:39 <siddharthvipul> lenkaseg: she is awesome, and that blog is awesome
16:47:54 <petebuffon[m]> There is also a ton of stuff you can't normally or easily do with just containers
16:48:15 <petebuffon[m]> Things like firewalls and network policies
16:48:41 <petebuffon[m]> But at the end of the day it's all just Linux components
16:49:37 <petebuffon[m]> Hopefully y'all are less confused rather than more confused
16:49:49 <t0xic0der> petebuffon[m]: XD
16:50:12 <lenkaseg> petebuffon: eh heh :)
16:50:43 <siddharthvipul> petebuffon[m]: more confusion as per dunning-kruger graph is a good thing
16:50:49 <siddharthvipul> well, 50% of the time
16:51:14 <siddharthvipul> okay then, let's move to open floor for a last few minutes..
16:51:16 <petebuffon[m]> siddharthvipul: exactly!
16:51:57 <eddiejennings> I'm up on the 27th for training about scripting.
16:52:00 <siddharthvipul> #topic openfloor
16:52:16 <siddharthvipul> oh no, I totally missed that part
16:52:18 <siddharthvipul> #undo
16:52:18 <zodbot> Removing item from minutes: <MeetBot.items.Topic object at 0x7f4738917c90>
16:52:26 <siddharthvipul> #topic Upcoming learning topics
16:52:40 <siddharthvipul> eddiejennings: I see you have already entered in agenda
16:52:48 <siddharthvipul> #info 2021-07-29- eddiejennings, How to Get Into Scripting (bash / powershell)
16:52:54 <siddharthvipul> eddiejennings: thank you for that
16:53:14 <siddharthvipul> do we have any volunteer who wants to talk about $stuff_you_love on Aug 5th?
16:53:30 <eddiejennings> I won't be discussing exactly how to write scripts, but rather approaches to incorporating scripting into your administrator toolset.
16:53:46 <siddharthvipul> eddiejennings: oh great! looking forward to that :D
16:53:58 <petebuffon[m]> eddiejennings: that sounds great
16:55:29 <siddharthvipul> okay, now we can move to open floor :)
16:55:35 <siddharthvipul> #topic openfloor
16:56:56 <siddharthvipul> mkonecny: do you still want to talk about backlog refinement or that's a stale topic?
16:57:20 <mkonecny> Yes, I want to talk about the mail response
16:57:32 <siddharthvipul> we have 2 minutes :D
16:58:01 <mkonecny> There was one proposal to use this meeting once per two week for backlog refinement instead of learning topic
16:58:14 <mkonecny> What do you think?
16:58:28 <mobrien> I am in favour of that
16:59:24 <siddharthvipul> alternate? managing can be complicated to I am +1 to try that
16:59:46 <siddharthvipul> mkonecny: so far I see 100% agreement lol
17:00:02 <mkonecny> We are having problems with getting enough learning topics
17:00:21 <siddharthvipul> that's fair
17:00:37 <mkonecny> Ok, I will update the agenda
17:01:26 <siddharthvipul> thank you all for attending
17:01:28 <siddharthvipul> #endmeeting