16:01:49 <petebuffon> #startmeeting Infrastructure (2021-10-28) 16:01:50 <zodbot> Meeting started Thu Oct 28 16:01:49 2021 UTC. 16:01:50 <zodbot> This meeting is logged and archived in a public location. 16:01:50 <zodbot> The chair is petebuffon. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions. 16:01:50 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:01:50 <zodbot> The meeting name has been set to 'infrastructure_(2021-10-28)' 16:01:54 <petebuffon> #meetingname infrastructure 16:01:54 <zodbot> The meeting name has been set to 'infrastructure' 16:01:59 <eddiejennings> .hello 16:01:59 <zodbot> eddiejennings: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1". 16:02:00 <petebuffon> #chair nirik siddharthvipul mobrien zlopez pingou bodanel dtometzki jnsamyak computerkid 16:02:00 <zodbot> Current chairs: bodanel computerkid dtometzki jnsamyak mobrien nirik petebuffon pingou siddharthvipul zlopez 16:02:01 <nirik> morning 16:02:05 <petebuffon> #info Agenda is at: https://board.net/p/fedora-infra 16:02:09 <petebuffon> #info About our team: https://docs.fedoraproject.org/en-US/cpe/ 16:02:09 <Leo[m]1234> .hi leo 16:02:10 <zodbot> Leo[m]1234: Sorry, but user 'Leo [m] 1234' does not exist 16:02:12 <petebuffon> #topic greetings! 16:02:15 <darknao> .hi 16:02:16 <zodbot> darknao: darknao 'Francois Andrieu' <darknao@drkn.ninja> 16:02:17 <LenkaSegura[m]> .hello lenkaseg 16:02:19 <zodbot> LenkaSegura[m]: lenkaseg 'Lenka Segura' <lenka@sepu.cz> 16:02:20 <petebuffon> .hello petebuffon 16:02:22 <zodbot> petebuffon: petebuffon 'Peter Buffon' <pabuffon@gmail.com> 16:02:23 <petebuffon> hey everyone :) 16:02:23 <eddiejennings> .hi 16:02:25 <zodbot> eddiejennings: eddiejennings 'Eddie Jennings' <eddie@eddiejennings.net> 16:02:28 <Leo[m]1234> .hello leo 16:02:29 <zodbot> Leo[m]1234: leo 'Leo Puvilland' <lpuvilla0001@mymail.lausd.net> 16:02:30 <AidenLangley[m]> /wave 16:03:14 <petebuffon> #topic New folks introductions 16:03:15 <petebuffon> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves 16:03:15 <petebuffon> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted 16:03:22 <austinpowered> .hi 16:03:22 <zodbot> austinpowered: austinpowered 'T.C. Williams' <fedoraproject@wootenwilliams.com> 16:03:24 <petebuffon> any newcomers today? 16:03:40 <AidenLangley[m]> I rambled enough in my email ^^ 16:03:47 <AidenLangley[m]> but yes, me, hey everybody 16:03:58 <Leo[m]1234> hello :D welcome 16:04:18 <petebuffon> welcome! if you want please feel free to introduce yourself. 16:04:35 <darknao> welcome AidenLangley[m] 16:04:57 <nirik> welcome AidenLangley[m] 16:05:03 <AidenLangley[m]> tyty, yeah briefly, I'm nedia or aiden, sometimes langs 🙂 it's 5am, I might sleep after this haha but I'm a night owl 16:05:22 <pmoura> welcome AidenLangley[m] 16:05:25 <eddiejennings> welcome! 16:05:27 <AidenLangley[m]> I'm just getting into fedora & open source and trying to see where I can be useful 16:05:32 <leothecat> alr am on irc now 16:05:39 <leothecat> but i'm Leo on matrix :) 16:05:41 <siddharthvipul> \o 16:05:52 <siddharthvipul> .hello siddharthvipul1 16:05:53 <zodbot> siddharthvipul: siddharthvipul1 'Vipul Siddharth' <siddharthvipul1@gmail.com> 16:06:08 <petebuffon> wow 5am is early 16:07:01 <petebuffon> well welcome welcome, if you have any questions feel free to ask on fedora-admin, we'd be happy to answer! 16:07:19 <AidenLangley[m]> I grew up in London but now in NZ, I think a part of me is still on the other side of the planet so I am always up far too late 16:07:42 <AidenLangley[m]> petebuffon: ty will do 16:07:50 <leothecat> hello! :D 16:08:01 <petebuffon> I spent some time in NZ and really liked it 16:08:22 * nirik has heard good things, but never been there. 16:08:23 <AidenLangley[m]> it's lovely 16:08:57 <petebuffon> alright let's move on to the next chair 16:09:02 <petebuffon> ### Determine who the next chair is 16:09:02 <petebuffon> #topic Next chair 16:09:03 <petebuffon> #info magic eight ball says: 16:09:03 <petebuffon> #info chair 2021-11-04 - lenkaseg 16:09:03 <petebuffon> ##info chair 2021-11-11 - ??? 16:09:03 <petebuffon> ##info chair 2021-11-18 - ??? 16:09:15 <petebuffon> any takers for the 11-11 or 11-18? 16:09:40 <eddiejennings> I can't do 11/11 because I might not make the meeting, but I can do 11/18 16:10:06 <petebuffon> sounds good eddiejennings 16:10:19 <petebuffon> ##info chair 2021-11-18 - eddiejennings 16:10:21 * nirik can do 11-11 if no one else wants it. 16:11:09 <petebuffon> and it's yours! 16:11:15 <petebuffon> ##info chair 2021-11-11 - nirik 16:11:34 <petebuffon> perfect let's move on 16:11:36 <petebuffon> #topic announcements and information 16:11:36 <petebuffon> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting 16:11:36 <petebuffon> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3 16:11:36 <petebuffon> #info If your team wants support from the Fedora Program Management Team, file an isssue: https://pagure.io/fedora-pgm/pgm_team/issues?template=support_request 16:11:38 <petebuffon> #info Fedora 35 Beta was released on 2021-09-28 16:11:40 <petebuffon> #info Fedora 35 Final freeze is in effect 16:11:49 <leothecat> yep 16:11:52 <petebuffon> any other news topics for today? 16:12:59 <nirik> #info f35 go/no-go in about 45min (after this meeting) 16:13:58 <petebuffon> perfect 16:14:19 <leothecat> cool 16:14:43 <petebuffon> alright oncall time then 16:14:44 <petebuffon> #topic Oncall 16:14:45 <petebuffon> #info https://fedoraproject.org/wiki/Infrastructure/Oncall 16:14:45 <petebuffon> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/ 16:14:45 <petebuffon> #info nirik on call from 2021-10-21 to 2021-10-28 16:14:46 <petebuffon> ## .oncalltakeeu .oncalltakeus 16:14:48 <petebuffon> #info darknao on call from 2021-10-28 to 2021-11-04 16:14:50 <petebuffon> #info eddiejennings on call from 2021-11-04 to 2021-11-11 16:14:52 <petebuffon> #info ??? on call from 2021-11-11 to 2021-11-18 16:15:07 <petebuffon> any takers for on call starting the week of 11-11? 16:15:33 <darknao> .oncalltakeeu 16:15:35 <zodbot> darknao: Kneel before zod! 16:15:41 <ahmedalmeleh> what is that call for? 16:16:10 <petebuffon> the person who is on call for the week accepts pings for people who need help. 16:16:19 <eddiejennings> If no one else wants, I can do a second week (11/11). I generally like ending my on-call when the week I chair. :D 16:16:37 <leothecat> wait lemme look at my calendar 16:16:50 <petebuffon> they then triage the help needed and help out if they can, make a new ticket, or if necessary alert other team members in an emergency 16:17:03 <leothecat> i can probably do it 16:17:04 <ahmedalmeleh> got it 16:17:08 <petebuffon> check out https://fedoraproject.org/wiki/Infrastructure/Oncall 16:17:42 <petebuffon> makes sense eddiejennings 16:17:50 <petebuffon> alright so i'll give it to leothecat then 16:17:51 <LenkaSegura[m]> I can take oncall too some week. 16:18:31 <petebuffon> #info leothecat on call from 2021-11-11 to 2021-11-18 16:18:34 <leothecat> yep 16:18:39 <eddiejennings> leothecat: Would you be willing to do 11/4's week, and I take 11/11? 16:19:02 <leothecat> uh probably? 16:19:23 <petebuffon> okay 16:19:38 <petebuffon> #info leothecat on call from 2021-11-04 to 2021-11-11 16:19:43 <eddiejennings> Thank you :D 16:19:54 <ahmedalmeleh> I want to join infrastructure's team 16:19:54 <petebuffon> #info eddiejennings on call from 2021-11-11 to 2021-11-18 16:20:20 <petebuffon> LenkaSegura want to take the week after, 11-25? 16:21:19 <eddiejennings> I hear you get time and a half when on-call on Thanksgiving ;) 16:21:20 <LenkaSegura[m]> ok! 16:21:39 <petebuffon> hah :) kk 16:21:57 <LenkaSegura[m]> in Spain we give hanks all year :) 16:22:07 <LenkaSegura[m]> s/hanks/thanks 16:22:14 <petebuffon> #info LenkaSegura on call from 2021-11-25 to 2021-12-02 16:22:28 <petebuffon> moving on 16:22:29 <petebuffon> #info Summary of last week: (from current oncall ) 16:22:29 <petebuffon> #topic Monitoring discussion [nirik] 16:22:29 <petebuffon> #info https://nagios.fedoraproject.org/nagios 16:22:29 <petebuffon> #info Go over existing out items and fix 16:23:01 <nirik> There were no oncall calls that I saw. I think mostly people are trying to get f35 out the door... :) 16:23:07 <nirik> for monitoring... 16:23:10 <leothecat> a service is flapping :P 16:23:38 <nirik> yeah, thats a bogus check we need to remove... 16:24:04 <nirik> otherwise it's pretty much the same as last week... and we are freeze, so we haven't really fixed anything 16:24:12 <nirik> we can move on unless there's questions... 16:24:44 <petebuffon> release purgatory 16:25:14 <nirik> we have a lot of things piled up for after freeze. Gonna be fun. 16:25:23 <leothecat> yep 16:25:37 <petebuffon> nice 16:25:51 <petebuffon> well I believe you're up again nirik, if you're ready for the learning topic 16:25:58 <nirik> ah yeah. 16:26:05 <petebuffon> #topic Learning topic 16:26:10 <petebuffon> #info 2021-10-28 - Introduction to AWS and Fedora Infrastructure [nirik] 16:26:33 <nirik> so, lots to cover here, so feel free to ask questions if I don't cover something you would like me to... 16:26:37 <ahmedalmeleh> are we using AWS? 16:26:48 <nirik> we are for some things. :) 16:26:51 <copperi[m]1> yes 16:26:57 <leothecat> oo 16:27:05 <leothecat> im interested in this 16:27:12 <AidenLangley[m]> me too 16:27:17 <ahmedalmeleh> me three 16:27:37 <nirik> So, amazon has very gracefully provided us with a community account. This lets us do things for our project and amazon takes care of the bill. Kudos to amazon for that. 16:27:54 <ahmedalmeleh> Good on them 16:28:04 <AidenLangley[m]> besoz can spare a penny or two 16:28:08 <nirik> We only have the one account thats setup this way and we need to share it for a lot of things that we don't want to interfere with each other. 16:28:40 <nirik> We have setup authentication on this account to use our auth system (via SAML2) and groups. 16:28:46 <ahmedalmeleh> Okay 16:29:12 <nirik> The aws-* groups in our account system map to permissions in aws. So, someone in that group logs in via our auth and gets logged in as that group. 16:29:34 <leothecat> oh cool so they pay 16:29:39 <nirik> We then take advantage of IAM policies. We start with "deny everything" and add only those permissions that group needs to do what they need to do. 16:30:02 <nirik> There's a number of groups involved: 16:30:22 <nirik> * copr - copr uses this for builders and their frontend/backend boxes. 16:30:49 <nirik> * centos - centos uses this for instances and also cloudfront (which I will get to in a few) 16:31:14 <nirik> *fedora-ci uses this for a eks cluster that does ci tests on fedora packages 16:31:32 <nirik> * fedora-infra runs maintainer-test instances in it. 16:32:06 <nirik> * fedora infra runs some proxy servers in it (in regions where we don't have any donated servers, like APAC, AU, etc) 16:32:25 <nirik> * we also use it for some one off development/test instances when someone is testing something. 16:32:41 <leothecat> is it used for quick spinup and then destroy machines? 16:32:44 <leothecat> like to test one thing 16:32:45 <nirik> There may be more that I am not thinking of. ;) 16:33:04 <nirik> leothecat: nope, only for persistent stuff currently. 16:33:11 <leothecat> oh okay 16:33:31 <nirik> well, copr manages it's own instances... it spins up builders to do builds then destroys them when the build is done. 16:33:42 <nirik> it's pretty cool. :) 16:34:19 <nirik> Another big use we have with amazon is cloudfront. cloudfront is their caching proxy / cdn product. 16:34:43 <leothecat> phew am back i disconnected 16:34:46 <nirik> We have a cloudfront setup for registry.fedoraproject.org (our container registry) 16:34:58 <nirik> and ostree repos 16:35:17 <nirik> and we also have one for internal to aws fedora/epel use 16:35:17 <ahmedalmeleh> ok 16:35:26 <nirik> centos also is using cloudfront for mirrors and such. 16:35:36 <leothecat> okay 16:35:47 <leothecat> and all of this except copr is all done manually? 16:35:50 <leothecat> *for now* 16:36:11 <nirik> cloudfront works by getting requests and then fetching data from an origin server, then caching that locally. it has endpoints in basically everyplace amazon has regions. 16:37:03 <nirik> well, each group controls their things... copr, centos, fedora-ci all manage their resources however they want... I think for example fedora-ci uses an amazon deployment thing for eks 16:37:12 <leothecat> o okay 16:37:19 <petebuffon> Is there a specific reason for using cloudfront for registry.fedoraproject.org? 16:37:21 <nirik> but in fedora-infra, yes, we currently manually deploy things then manage them via ansible 16:37:32 <nirik> petebuffon: it gets a LOOOOOOOOT of traffic. 16:37:40 <petebuffon> got it 16:37:59 <nirik> basically podman on every fedora box hits it first and asks for whatever image... 16:38:10 <nirik> I think it's first in the search path 16:38:25 <copperi[m]1> nice 16:38:26 <petebuffon> right, I may have done that a few times... 16:39:16 <nirik> and for ostree things, we don't mirror them in our normal mirror network, so ostree using folks were seeing it pretty slow from other parts of the world... 16:39:59 <nirik> And finally we also have a app called fedimg that uploads our fedora images to aws... 16:40:31 <nirik> It basically uploads everything thats composed and passed a simple 'does it boot' test, and it copies that to every region. 16:40:53 <nirik> but it also does that for final composes, etc... they are all there. 16:41:59 <nirik> So if you ever want to fire off a fedora instance, https://alt.fedoraproject.org/cloud/ has buttons to do that. ;) 16:42:27 <leothecat> oh wow it just launches aws 16:42:30 <nirik> I think thats about all... any questions? 16:42:50 <AidenLangley[m]> fedimg hehe nice name... Can picture the logs reading 'fedimg: fed image to aws' 16:43:11 <copperi[m]1> nice info nirik 16:44:16 <nirik> ok, thanks everyone. Hopefully that made things as clear as mud. :) 16:44:18 <austinpowered> nirik++ 16:44:24 <petebuffon> nirik++ 16:44:25 <leothecat> cool :) 16:44:27 <AidenLangley[m]> is this a new set up? 16:44:30 <eddiejennings> nirik++ 16:44:41 <eddiejennings> nirik keeps adding to my list of need-to-learn-and-become-familiar :P 16:44:41 <pmoura> nirik++ 16:44:45 <leothecat> by the way the aws topic on the list was replied again 16:45:28 <nirik> AidenLangley[m]: nope, been around for years. 16:46:01 <copperi> nirik++ 16:46:47 <nirik> petebuffon: back to you for open floor. ;) 16:46:51 <petebuffon> great! makes me want to dive into AWS 16:46:54 <petebuffon> #topic Open Floor 16:47:00 <petebuffon> got about 15 minutes left 16:48:23 <austinpowered> the learning topic for 2021-09-02 was ssh host keys signing and ansible interaction 16:48:52 <austinpowered> meetbot doesn't have any log files for that date 16:50:02 <AidenLangley[m]> so is somebody tasked w/ learning about the topic and giving the team a lesson? 16:50:07 <eddiejennings> I think meetbot and its logging has been broke for a while, no? The logging itself isn't broken but the display on the web is, right? 16:51:14 <petebuffon> Maybe something to do with the zodbot changeover to python3? 16:51:20 <nirik> it should be working. 16:51:34 <eddiejennings> Ah. I honestly haven't checked in a while. 16:51:46 <austinpowered> the dir at https://meetbot-raw.fedoraproject.org/teams/infrastructure/ skips from 08-19 to 09-09 16:51:49 <leothecat> so about the aws ansible integration 16:51:59 <leothecat> are we going to go ahead with it or wiat until freeze? 16:52:01 <nirik> or perhaps it was broken then... 16:52:34 <petebuffon> AidenLangley: the last portion of the meeting is either a learning topic or backlog refinement. It goes back and forth every week 16:52:48 <austinpowered> As I recall it was broken at the time. If the file available anywhere else? 16:53:00 <austinpowered> /If/Is/ 16:53:07 <eddiejennings> Is https://pagure.io/fedora-docs/docs-fp-o now the official place to create / update infra documentation? 16:53:44 <nirik> austinpowered: I am looking 16:54:05 <austinpowered> nirik: thanks 16:55:40 <leothecat> so we wait until freeze for aws->ansible? 16:56:07 <nirik> I think we can work on it... PR's would surely be fine before freeze is over. 16:56:18 <nirik> eddiejennings: yes 16:56:33 <eddiejennings> excellent 16:56:42 <ahmedalmeleh> ok 16:57:30 <nirik> austinpowered: https://meetbot-raw.fedoraproject.org/fedora-meeting-3/2021/fedora-meeting-3.2021-09-02-16.00.html 16:57:40 <nirik> it looks like it didn't get the right name somehow 16:58:33 <leothecat> okay we can kind of copy from https://pagure.io/fedora-infra/arc/blob/main/f/ansible 16:58:55 <nirik> copy? 16:59:31 <leothecat> or well base it on 16:59:48 <leothecat> because they already use aws in ansible 17:00:16 <nirik> the aws provisioning? no need to copy the repo... it should just be adjustments to the tasks/cloud_setup_basic or whatever it is. ;) 17:00:22 <petebuffon> thanks everyone, if you want to keep the conversations going please move over to #fedora-admin 17:00:27 <petebuffon> #endmeeting