16:00:03 #startmeeting Infrastructure (2021-06-17) 16:00:03 Meeting started Thu Jun 17 16:00:03 2021 UTC. 16:00:03 This meeting is logged and archived in a public location. 16:00:03 The chair is siddharthvipul. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:03 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:00:03 The meeting name has been set to 'infrastructure_(2021-06-17)' 16:00:06 #meetingname infrastructure 16:00:06 The meeting name has been set to 'infrastructure' 16:00:08 #chair nirik siddharthvipul mobrien zlopez pingou bodanel dtometzki jnsamyak 16:00:08 Current chairs: bodanel dtometzki jnsamyak mobrien nirik pingou siddharthvipul zlopez 16:00:10 #info Agenda is at: https://board.net/p/fedora-infra 16:00:12 #info About our team: https://docs.fedoraproject.org/en-US/cpe/ 16:00:14 #topic ahoy! 16:00:32 avast! 16:00:36 o/ 16:00:41 .hello jnsamyak 16:00:41 \o 16:00:41 jnsamyak: jnsamyak 'Samyak Jain' 16:00:44 Salut tout le monde 16:00:53 \o :D 16:01:02 .hello eddiejennings 16:01:06 eddiejennings: eddiejennings 'Eddie Jennings, Jr.' 16:01:10 .hello siddharthvipul1 16:01:11 siddharthvipul: siddharthvipul1 'Vipul Siddharth' 16:01:29 how is everyone today? 16:01:42 Doing well. 16:01:46 .hello jamricha 16:01:47 jamricha: Sorry, but you don't exist 16:01:54 :( 16:01:54 a bit sleepy, but otherwise fine. ;) 16:01:59 jamricha: :"( 16:02:09 nirik: I can't imagine being up when you already start working 16:02:16 siddharthvipul: "dancing in my room" - if someone has heard that song XD 16:02:27 jamricha: your FAS id is jrichardson iirc 16:02:39 so it would be `.hello jrichardson` 16:02:48 .hello sumantro 16:02:48 .hello jrichardson 16:02:48 sumantro: sumantro 'Sumantro Mukherjee' 16:02:51 jamricha: jrichardson 'James Richardson' 16:02:53 .hello sumantrom 16:02:54 sumantro: sumantrom 'Sumantro Mukherjee' 16:02:59 sumantro: heyy \o fancy seeing you here :D 16:02:59 need to change that 16:03:12 siddharthvipul, yes yes! 16:03:12 .hello jbwillia 16:03:13 Southern_Gentlem: jbwillia 'Ben Williams' 16:03:14 .hello lenkaseg 16:03:16 lenkaseg: lenkaseg 'Lenka Segura' 16:03:27 so many people.. amazing! 16:03:33 hey all 16:03:38 let's get to topics then ^-^ 16:03:48 wow. full crew today! 16:03:57 .hello lsegura 16:03:58 lenkaseg: Sorry, but you don't exist 16:04:03 nirik: I think it's because of me :P 16:04:06 .hello humaton 16:04:08 jednorozec: humaton 'Tomáš Hrčka' 16:04:16 siddharthvipul: XD 16:04:28 siddharthvipul: definitely! 16:04:34 .hi 16:04:35 darknao: darknao 'Francois Andrieu' 16:04:39 lenkaseg: `.hello ` :) 16:04:43 haha 16:05:07 #topic New folks introductions 16:05:09 #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves 16:05:34 so as I see a couple of fresh nicks here, would be great if you introduce yourself 16:05:50 so that we know each other :) 16:06:39 I am Sumantro and I am wokring as oart of Fedor a QE and help to help with Fedora Infra and join Fedora Apprentice programme 16:06:57 sumantro: perfect, we can sort that out. It's a great way to start looking into Infra. 16:07:11 I know jamricha also asked to be added in this group..let me add you two 16:07:23 meanwhile if someone else is here for their first meeting, do introduce 16:07:26 Hi All! James Richardson, new associate eng with the CPE team, specifically the infra team 16:07:29 I'd like to be added too please :-) 16:07:29 and sumantro: welcome :) 16:07:43 petebuffon[m]: got it, can you please tell me your FAS username? 16:07:52 Who all can participate in Fedora Apprentice group? I wanna too :D 16:07:58 siddharthvipul: petebuffon 16:08:11 jnsamyak: sure! 16:08:16 What does CPE stand for? 16:08:23 jnsamyak: your FAS? 16:08:28 Community Platform Engineering 16:08:33 Hello everybody! I'm Lenka Segura, new associate eng with the CPE team, specifically the infra team (yes, I copied it from James :) ) 16:08:34 * eddiejennings nods. 16:08:37 hi eddiejennings! 16:08:46 welcome everyone! 16:08:48 nice one lenkaseg 16:08:58 :D 16:09:37 yes, anyone can join the apprentice group... although we should probibly remove old members who aren't around anymore at some point again. 16:10:09 nirik: I can look around and create an infra ticket (with all usernames and ask them there if they still want that) 16:10:23 jnsamyak it is 16:10:51 petebuffon[m]: sumantro jnsamyak: added in fi-apprentice 16:10:53 siddharthvipul: at one point we said people should login at least once in the last month... but we never automated that. 16:11:08 lenkaseg: and jrichardson: you two are already a member :) 16:11:24 siddharthvipul: thanks! 16:11:36 thanks siddharthvipul 16:11:38 siddharthvipul: thank youu! 16:11:41 nirik: let's talk that in open floor.. I have a few questions :D 16:11:44 thanks vipul! 16:12:02 siddharthvipul: sure thing 16:12:03 thank YOU ALL for the interest :) feel free to ask any questions you have in #fedora-admin 16:12:16 siddharthvipul: Thanks! 16:12:21 and go through the apprentice wiki link to see what comes with it :) 16:12:33 let's move to next topic 16:12:42 #topic Next chair 16:12:44 #info magic eight ball says: 16:12:46 #info chair 2021-06-24 - jnsamyak 16:12:48 #info chair 2021-07-01 - Zlopez 16:12:50 #info chair 2021-07-08 - dtometzki 16:13:00 so we are 3 weeks in advance.. I say that's plenty for now. Should we revisit this next week? 16:13:06 +1 16:13:47 100% vote in support, let's move :D 16:14:10 #topic announcements and information 16:14:14 #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting 16:14:16 #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3 16:14:18 #info work is being done on getting COPR proper powerPC systems 16:14:20 #info The Red Hat Desktop team is hiring: https://blogs.gnome.org/uraeus/2021/05/20/new-opportunities-in-the-red-hat-desktop-team/ 16:14:22 #info Fedora Linux 32 reached end of life 16:14:24 #info If your team wants support from the Fedora Program Management Team, file an isssue: https://pagure.io/fedora-pgm/pgm_team/issues?template=support_request 16:14:26 #info nest with Fedora CFP is open! https://communityblog.fedoraproject.org/announcing-dates-cfp-for-nest-with-fedora/ (Aug 5th-8th) 16:14:39 Yay NEST! 16:14:41 oh wait 16:14:44 .hello2 16:14:45 computerkid: computerkid 'Grayson Penland' 16:14:49 #info Fedora's (first) annual contributors survey is active: https://fedoraproject.limequery.com/2021 - we have a shiny badge 16:15:07 computerkid: hey there 16:15:07 * nirik needs to put in a talk for nest. I think I might do a talk on pinephones... not sure. 16:15:29 HI siddharthvipul! 16:15:44 ooh, nice! if I had patience to deal with Indian customs, I would have ordered one :D 16:15:48 nirik: ^ 16:15:54 I should also do something for NEST. Riecatnor wants me to do live stream stuff, and/or fedora podcast stuff 16:16:00 the issue is, IDK what stuff is 16:16:02 do we have any more announcements? 16:16:21 computerkid: we can figure that out ;) there are enough $stuff to keep you busy haha 16:16:27 computerkid: +1 16:16:38 Shameless plug: 2 fedora podcast episodes out, new one in 5 days 16:17:01 #chair computerkid 16:17:01 Current chairs: bodanel computerkid dtometzki jnsamyak mobrien nirik pingou siddharthvipul zlopez 16:17:03 Sure siddharthvipul! I'd be happy to do this, but no one has opinions 🤷 16:17:09 computerkid: do you want to put that as an info? 16:17:13 better if there is a link 16:17:16 * computerkid is sits in the chair 16:17:29 the fedora podcast in general, or latest episode? 16:17:41 latest episode.. what it's about and link 16:17:48 just a short line 16:18:26 #info Fedora Podcast episode 2 is out, we talked about pipewire, and the survey https://podcast.fedoraproject.org/season-two-episode-2-the-host-wim-and-aleksandra/ 16:18:34 let's move to next topic 16:18:36 thank you computerkid 16:18:43 No problem 16:18:53 #topic Oncall 16:18:55 #info https://fedoraproject.org/wiki/Infrastructure/Oncall 16:18:57 #info dtometzki is on call for 2021-06-10 to 2021-06-17 16:18:59 #info darknao is on call for 2021-06-17 to 2021-06-24 16:19:01 #info jnsamyak is on call for 2021-06-24 to 2021-07-01 16:19:14 we are looking for oncall volunteers from 2021-06-01 to 2021-07-08 16:19:33 if I don't get a volunteer (unique from the list now), we will see again in next meeting :P 16:19:48 .oncalltakeeu 16:19:48 darknao: Error: You don't have the alias.add capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified. 16:19:54 🤷 I can do it 16:20:30 computerkid: perfect.. do read the wiki and reach out to me or generally in fedora-admins if you have any queries :) 16:20:37 .oncalltakeeu 16:20:37 darknao: Kneel before zod! 16:20:43 #info computerkid is on call for 2021-06-01 to 2021-07-08 16:20:46 #info computerkid is on call for 2021-06-01 to 2021-07-08 16:20:58 07-01? 16:21:08 siddharthvipul: on call for whole month :o 16:21:08 #undo 16:21:08 Removing item from minutes: INFO by siddharthvipul at 16:20:46 : computerkid is on call for 2021-06-01 to 2021-07-08 16:21:19 #info computerkid is on call for 2021-07-01 to 2021-07-08 16:21:37 no nirik , we were about to get a long term volunteer :"( 16:21:38 Wow, I'm not volunteering for a month sorry XD 16:21:39 haha 16:21:43 :) 16:21:49 computerkid: too bad we caught it in time 16:22:03 Just btw do i have to register myself using `.oncalltakeeu` for the upcoming week? 16:22:03 * computerkid is happy 16:22:29 jnsamyak: yes, we will help you with that :) just authenticate to zodbot 16:22:44 jnsamyak: yes, you need to register with zodbot... /msg zodbot misc help register' 16:23:03 Just kneel before the all powerful zod 16:23:28 .oncalltakeeu 16:23:28 jnsamyak: Error: You don't have the alias.add capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified. 16:23:42 jnsamyak: please don't take oncall duty now :) 16:23:48 it's next week if I see correctly 16:24:15 oh ok we will do it after the meet or in open floor XD (yes we have time) sorry! 16:24:18 jnsamyak: we can sort this out in open floor.. it's easy :) 16:24:28 no worries at all ^-^ 16:24:35 #info Summary of last week: (from current oncall ) 16:24:37 #topic Monitoring discussion [nirik] 16:24:39 #info https://nagios.fedoraproject.org/nagios 16:24:41 #info Go over existing out items and fix 16:24:43 nirik: now you do what you do :D 16:24:58 heh. 16:25:53 so, I managed to clear up about 5 management interfaces and 4 hosts the other day... they were down due to them forgetting their network config, but I managed to finally get someone to run some serial cables to the chassis and got them to reconfigure. 16:26:21 most of the alerts we have left right now are for low swap (which seems to be a rhel8 bug) 16:26:38 so, nothing to exciting this week... we can move along 16:27:30 got it :) 16:27:47 #topic Learning topic 16:27:49 #info 2021-06-17 - nirik - general introduction / tour of infrastructure' 16:27:56 #info 2021-06-17 - nirik - general introduction / tour of infrastructure' 16:28:01 ok. :) 16:28:21 I wonder if nirik was volunteered :P 16:28:24 so, there's a ton to go over, please do ask questions if I skip over something you all want to talk on. 16:28:31 +1 16:28:42 Where I am it's called "voluntold" 16:28:57 So, first, what is fedora infrastructure? 16:29:50 We are a community group who runs the infrastructure for the fedora project. This is primarially things related to creating Fedora Linux (the distro) but also many other related things for the community 16:30:39 So, we develop applications / software to build Fedora Linux, as well as deploy/update/manage/operations around those things. 16:31:29 We are very tied to the CPE team in Red Hat. CPE stands for Community Platform Engineering... this is the team that supports Fedora and CentOS within Red Hat. 16:32:01 But we are also Open and Community / volenteer based. 16:32:09 so, it's a mix of people 16:32:32 Red Hat also pays for lots of our infrastructure (machines, networks, datacenter space, etc) 16:32:42 one confusion I have seen around it CPE == Fedora-Infra which is not true. CPE members are just a subset of Fedora-Infra gorup 16:32:53 but also we get some donated from various places (hosting providers, Amazon, etc) 16:33:04 siddharthvipul: exactly true. :) 16:33:21 siddharthvipul: and CPE == the Red Hat employee presence in the Fedora-Infra group? 16:33:59 eddiejennings: CPE does more than just Fedora-Infra but I am comfortable saying that majority is around Fedora-Infrastructure and applications 16:34:10 In the beginning fedora infrastructure was 100% open source and everything was run by us. Over the years, this has relaxed some as the Fedora Council now permts non free things hosted by others in some cases... 16:34:44 but we strive to stick to open source wherever possible 16:35:04 nirik: sorry to break your flow, can you tell a couple of non-FOSS tools we use 16:35:28 Some services we don't run: bugzilla.redhat.com, taiga, discourse (ask/discuss.fedoraproject.org) 16:35:48 also we host development for some of our applications on github 16:36:28 * siddharthvipul thought taiga was FOSS 16:36:34 So, speaking of those. We have a number of applications we have developed for our infrastructure needs... 16:36:52 siddharthvipul: it is. nirik was listing services not run by infra, not non-foss services 16:36:54 github.com/fedora-infra and pagure.io/fedora-infra/ (and other places there) 16:37:03 ah, sorry! 16:37:07 right, sorry, bcotton_ is right there. 16:37:24 don't get used to it ;-) 16:37:31 :V :P 16:38:10 I'm very much on the operations side, so I would let others speak to application development. :) 16:38:28 for the ops side of things... we have ansible managing most everything we run. 16:38:44 our ansible git repo is in https://pagure.io/fedora-infra/ansible.git 16:39:00 it deploys things, installs all our vms, does all the heavy lifting 16:39:34 We have a staging env and a production env. Our staging is not 100% complete, but it has a lot of things so we can test before deploying to prod. 16:39:54 It has it's own seperate account system, and domain (stg.fedoraproject.org vs fedoraproject.org) 16:40:10 so if you want to go to say the staging account system, just add a stg in: 16:40:29 prod: https://accounts.fedoraproject.org stg: https://accounts.stg.fedoraproject.org 16:40:42 this allows us to make things very similar and just insert that .stg 16:41:27 We have 2 message busses (well, 4 I guess with staging): our old 'fedmsg' bus and a new 'fedora-messaging' bus 16:42:00 these are where our applications and scripts send messages when they do things... then we can have other applications act on those. It's very handy. 16:42:20 You can see the live feed at https://apps.fedoraproject.org/datagrepper/raw?rows_per_page=1&delta=127800 16:42:31 (and https://apps.stg.fedoraproject.org/datagrepper/raw?rows_per_page=1&delta=127800 for stg) 16:42:43 or #fedora-fedmsg and #fedora-fedmsg-stg on irc. ;) 16:43:05 /join #fedora-fedmsg 16:43:22 sorry* 16:43:29 Since our main focus is producing Fedora Linux (which is released about every 6 months) we have a series of freezes around that... 16:43:44 one before/around the Beta release and one before/around the Final release. 16:44:16 During a 'freeze' we stop all changes except those approved by release engineering or sysadmin-main folks... 16:44:29 this is to make sure we don't destablize anything and mess up the release. 16:44:59 We have things all over the world, but a quick summary: 16:45:40 iad2 -> ashburn, va, us. This is our primary datacenter, most of our machines are there. It's run by Red Hat IT folks (they handle networking and such) 16:45:52 * lenkaseg gets no results for #fedora-fedmsg-stg, am I the only one? (#fedora-fedmsg yes) 16:46:16 rdu-cc -> raleigh, nc, us -> this is a datacenter also run by Red Hat where some of our things are 16:46:53 lenkaseg: yeah, it's broken. ;( we moved irc networks recently and the staging server that should be connecting there can't due to a firewall rule. There's a ticket to get that rule fixed, but it's not happened yet. 16:47:28 aws -> we have a community account given us by amazon (hurray!) that we run some services/development instances in. 16:47:35 nirik: ok 16:47:55 we have a bunch of other servers where it's 1-3 machines in someones datacenter who have donated them to our use. 16:48:07 I guess the next biggest two are: 16:48:19 ibiblio -> also nc, us 16:48:21 and 16:48:36 osuosl -> or, us 16:49:22 Our account system which I mentioned above is a new application we wrote called noggin that sits on top of IPA. 16:49:40 IPA is now the source of truth, and it manages access to machines. 16:50:05 What is IPA? 16:50:51 eddiejennings: the user authentication system 16:50:55 eddiejennings: https://www.freeipa.org/page/Main_Page 16:50:59 eddiejennings: it's a set of applications for identity and policy 16:51:10 Thanks! 16:51:19 the free/opensource one is freeipa. The Red Hat product version is just called 'ipa' 16:51:41 So, in that account system we have a ton of groups and such. 16:51:53 Some important ones: 16:52:23 The fi-apprentice group is a group that has ssh access to a lot of machines, but is otherwise read only. Apprentices don't have sudo or the like. 16:52:36 The idea is to allow new interested people to login and look around 16:52:44 and see how things are setup, etc. 16:53:04 * jrichardson tries to not break anything important 16:53:14 Then we have a 'sysadmin' group. This group is the base group that allows you to commit to the ansible repo and is a requirement for all the other sysadmin groups. 16:53:39 We usually will add someone here once they have been around for a while and have submitted good PR's / fixes 16:53:55 then we have sysadmin-X groups for many of our applications. 16:54:22 So, say 'sysadmin-bodhi' will allow you to login to the bodhi application vm's and sudo there etc. 16:54:43 the idea is that we can add people to these groups to manage specific applications without giving them access to everything everywhere. 16:55:12 Also, these groups can run a script call 'rbac-playbook' on our ansible control host (batcave01) 16:55:26 this lets them run ansible against only their specific group of hosts. 16:55:54 Finally there's a 'syadmin-main' group. These are folks that have been around a long time and have basically access to everything, sudo everywhere, etc. 16:56:26 NOTE: you do not have to be a Red Hat employee to be in this group, just someone who has been around a long while, proved trusting and needs to do things. 16:57:05 A quick note on our permissions model: We don't want people to ask for permissions when they think they 'might need them' or as a contest... 16:57:35 instead you should try and solve problems/fix things/deploy things and when you get stuck because of permissions, _then_ ask for them and say why... 16:57:52 * nirik is running low on time. ;) 16:58:10 so I guess I will stop there, any questions? 16:58:16 thats a pile of stuff to digest. 16:58:36 none from me, thats really helpful though, thanks nirik 16:58:42 Nothing from myside. Its pretty clear this here 16:58:47 Thank you nirik! 16:58:47 thanks nirik! 16:58:48 permission model vs contest.. as s m o o g e said once.. the less permission I have, the less work I have to do! so yayy :P 16:58:53 This all should be documented somewhere (just copy pate :{) very verbose and nice info! 16:58:55 nirik: thank you 16:58:58 s/pate/paste 16:59:13 Thanks nirik 16:59:14 Great info! 16:59:15 nirik++ 16:59:15 jnsamyak: Karma for kevin changed to 31 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 16:59:18 does fedora-infra share infrastructure/apps with centos-infra ? or are they both very distinct team ? 16:59:18 yeah, a lot of it is in our getting started doc, but out of date. ;( 16:59:21 siddharthvipul: :D 16:59:27 darknao: great question! 16:59:29 do we discuss open floor topic or next week? 16:59:31 nirik++ 16:59:31 jrichardson: Karma for kevin changed to 32 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 16:59:42 Thanks nirik. Cannot wait to look around as apprentice (didn't have time yet) 16:59:48 we can discuss somewhere else (fedora-admin I guess) 16:59:53 let's close the meeting 16:59:56 thank you everyone for joining 17:00:01 it's always fun to chair this meeting 17:00:10 see you all next week :) 17:00:11 * jrichardson takes the initiative to update the getting-started docs 17:00:19 Thank you for chairing siddharthvipul ! See ya all o/ 17:00:28 #endmeetingh 17:00:30 ugh 17:00:32 darknao: they used to be seperate teams and completely seperate, but with CPE in Red Hat doing both now, we would like to share resources where we can... one example is that now centos us using the fedoraproject.org auth system (or vice versa) 17:00:32 #endmeeting