16:00:02 <siddharthvipul> #startmeeting Infrastructure (2021-02-04)
16:00:02 <zodbot> Meeting started Thu Feb  4 16:00:02 2021 UTC.
16:00:02 <zodbot> This meeting is logged and archived in a public location.
16:00:02 <zodbot> The chair is siddharthvipul. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:02 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:02 <zodbot> The meeting name has been set to 'infrastructure_(2021-02-04)'
16:00:02 <siddharthvipul> #meetingname infrastructure
16:00:02 <zodbot> The meeting name has been set to 'infrastructure'
16:00:02 <siddharthvipul> #chair nirik pingou smooge cverna mizdebsk mkonecny abompard siddharthvipul mobrien
16:00:02 <siddharthvipul> #info Agenda is at: https://board.net/p/fedora-infra
16:00:02 <siddharthvipul> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:00:02 <zodbot> Current chairs: abompard cverna mizdebsk mkonecny mobrien nirik pingou siddharthvipul smooge
16:00:02 <siddharthvipul> #topic aloha
16:00:13 <siddharthvipul> Good morning everyone :)
16:00:18 <siddharthvipul> who is here today?
16:00:23 <siddharthvipul> .hello siddharthvipul1
16:00:24 <zodbot> siddharthvipul: siddharthvipul1 'Vipul Siddharth' <siddharthvipul1@gmail.com>
16:00:24 <nirik> morning
16:00:24 <dtometzki> hi
16:00:41 <ComputerKid> Good morning
16:00:50 <siddharthvipul> nirik, dtometzki, ComputerKid \o How are you all
16:01:05 <ComputerKid> Good!
16:01:11 <Zlopez[m]> .hello zlopez
16:01:12 <zodbot> Zlopez[m]: zlopez 'Michal Konečný' <michal.konecny@psmail.xyz>
16:01:15 <smooge> hello
16:01:18 <darknao> .hi
16:01:19 <zodbot> darknao: darknao 'Francois Andrieu' <naolwen@gmail.com>
16:01:24 <siddharthvipul> welcome Zlopez[m] and darknao :D
16:01:39 <dtometzki> fine thanks
16:01:47 <nirik> I can't complain... but sometimes I still do... lifes been good to me so far. :) (wonder if anyone will get the reference)
16:03:11 <siddharthvipul> I also hear you have a mansion? :P
16:03:17 * siddharthvipul just googled.. sorry haha
16:03:36 * siddharthvipul plays it now!
16:04:02 <siddharthvipul> okay.. so let's give chance to new folks here introduce (if any) :D
16:04:04 <siddharthvipul> #topic New folks introductions
16:04:04 <siddharthvipul> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:04:05 <siddharthvipul> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:04:17 <siddharthvipul> so, who has attended these meeting less that 2-3 times :)
16:05:08 <ComputerKid> Hi
16:05:10 <ComputerKid> I have
16:05:17 <siddharthvipul> oh welcome ComputerKid :)
16:05:20 <ComputerKid> This is my second meeting
16:05:28 <siddharthvipul> do you want to do a brief introduction? or we have already done that?
16:05:34 <siddharthvipul> ComputerKid++ for joining us :D
16:05:34 <zodbot> siddharthvipul: Karma for computerkid changed to 3 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:05:39 <ComputerKid> I think I get the gist
16:05:44 <ComputerKid> Aw, thanks!
16:06:01 <ComputerKid> Fedora community is great
16:06:08 <siddharthvipul> ComputerKid, do let us know if you need any help in gettting started.. or just want to know what we do :)
16:06:26 <siddharthvipul> ComputerKid, I will agree with you on that :)
16:06:32 <ComputerKid> Y'all keep all the fedora apps running right?
16:07:01 <siddharthvipul> we try to.. yes! we look after all of Fedora Infrastructure (most of?)
16:07:44 <siddharthvipul> right.. so seems we don't have anyone else. Let's move to next item on agenda
16:07:57 <siddharthvipul> #topic Next chair
16:07:57 <siddharthvipul> #info magic eight ball says:
16:07:57 <siddharthvipul> #info chair 2021-02-11 - zlopez
16:07:57 <siddharthvipul> #info chair 2021-02-18 - smooge
16:07:59 <ComputerKid> Lol. That's really cool! I run a decent homelab IMO, so I have a small appreciation of how much that would be
16:08:16 <siddharthvipul> ComputerKid, opos,sorry about that
16:08:19 <ComputerKid> Sorry
16:08:34 <siddharthvipul> that's great! we should talk about it. let's get back to it on open floor :D
16:08:42 <siddharthvipul> ComputerKid, not at all.. very welcomed :)
16:08:58 <Zlopez[m]> Don't let it too open, so we don't fall
16:09:09 <siddharthvipul> so we have chairs for next 2 meetings.. we are looking for a volunteer for Feb 25, 2021
16:09:20 <siddharthvipul> Zlopez[m], ah, what's life without some risk
16:09:51 <siddharthvipul> I would really like to see someone new running the meetings.. It's very easy and you can ping us if you are ever stuck
16:10:22 <dtometzki> i can try it
16:10:30 <siddharthvipul> dtometzki, oh awesome! sold :D
16:10:42 <dtometzki> i am "new" :-)
16:10:42 <siddharthvipul> #info chair 2021-02-25 - dtometzki
16:10:43 <ComputerKid> I would be happy to help
16:10:57 <ComputerKid> I am new to being in fedora community
16:11:07 <siddharthvipul> ComputerKid, let's do it the one after that? we can decide that in the next meeting?
16:11:23 <siddharthvipul> dtometzki, ComputerKid please feel free to contact me on how to run these
16:11:35 <siddharthvipul> basically we have everything in agenda and we follow that
16:11:39 <dtometzki> yes we will do it
16:11:44 <siddharthvipul> ComputerKid++ dtometzki++
16:11:44 <zodbot> siddharthvipul: Karma for dtometzki changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:11:55 <siddharthvipul> right, let's move to the next topic then :)
16:12:06 <siddharthvipul> #topic announcements and information
16:12:07 <siddharthvipul> #info CPE Infra&Releng EU-hours team has a Monday through Friday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:12:07 <siddharthvipul> #info CPE Infra&Releng NA-hours team has a Monday through Friday 30 minute meeting going through tickets at 1800 UTC in #fedora-admin
16:12:07 <siddharthvipul> #info Datacenter move is over, but some items still need to be done: see https://fedoraproject.org/wiki/Infrastructure/2020-post-datacenter-move-known-issues
16:12:34 <siddharthvipul> nirik, for the last info, are there still some stuff that needs attention after DC move?
16:12:53 <siddharthvipul> let me check the link
16:13:03 <siddharthvipul> oh I see
16:13:16 <nirik> well, ongoing. I guess we can drop it from the meeting.
16:13:33 <siddharthvipul> nirik, got it
16:13:48 <siddharthvipul> anyone has anything to add in announcements?
16:14:18 <nirik> #info master to rawhide/main changes have mostly finished on src.fedoraproject.org
16:14:37 <nirik> #info tomorrow is a Red Hat day of learning, so many employees will be out learning things. :)
16:14:53 <siddharthvipul> #info Fedora stand in FOSDEM. Please check https://stands.fosdem.org/stands/the_fedora_project/
16:15:13 <siddharthvipul> awesome
16:15:32 <siddharthvipul> I will wait for 30 seconds before moving to next topic
16:16:17 <siddharthvipul> #topic Oncall
16:16:17 <siddharthvipul> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:16:17 <siddharthvipul> #info smooge is oncall for 2021-01-28 to 2021-02-04
16:16:17 <siddharthvipul> #info nirik is oncall for 2021-02-04 to 2021-02-11
16:16:17 <siddharthvipul> #info dtometzki is oncall for 2021-02-11 to 2021-02-18
16:16:26 <pingou> that was more than 30 seconds!
16:16:28 <dtometzki> yes
16:16:48 <siddharthvipul> pingou, India is UTC+5:30.. time is different here
16:16:50 <nirik> everyone's a critic. ;)
16:17:16 <siddharthvipul> :P
16:17:41 <siddharthvipul> we are looking for an oncall volunteer from 2021-02-18 to 2021-02-25
16:18:03 <siddharthvipul> do we have any? if no one, I can take it
16:18:14 <siddharthvipul> but if you would like to.. please say it now :D
16:18:24 <ComputerKid> What do you mean by volunteer
16:18:45 <pingou> siddharthvipul++ good answer ;-)
16:19:06 <siddharthvipul> WhErE aRe My cOoKiES
16:19:12 <ComputerKid> Lol
16:19:18 <pingou> you ate them already!
16:19:18 <dtometzki> :-)
16:19:37 <siddharthvipul> ComputerKid, anyone can volunteer to be oncall for a week. It means if someone notices a problem, they will use oncall feature to interact with you
16:19:48 <siddharthvipul> instead of directly pinging anyone else
16:20:06 <siddharthvipul> ComputerKid, I would recommend you going through https://fedoraproject.org/wiki/Infrastructure/Oncall
16:20:38 <siddharthvipul> it's a nice experience.. if the week is good, you get to learn a few things about Fedora applications.. and if the week is really really good.. you don't realize you were oncall for the whole week
16:21:25 <dtometzki> I think the next week will be good
16:21:37 <siddharthvipul> haha.. I hope not :P
16:21:49 <siddharthvipul> right! let me assign it to myself
16:22:09 <ComputerKid> Cool. I would be up for it sometime, I don't know how available I could be. As a kid I have to sleep a little more than most people
16:22:19 <siddharthvipul> #info siddharthvipul is on call for 2021-02-18 to 2021-02-25
16:23:02 <siddharthvipul> ComputerKid, oh that's totally understandable.. we are just glad you are here and are interested in things O:)
16:23:12 <siddharthvipul> nirik, would you like to take on call duty?
16:23:36 <siddharthvipul> for the week I mean.. ## .oncalltakeus
16:23:46 <nirik> sure.
16:23:50 <nirik> .oncalltakeus
16:23:50 <zodbot> nirik: Kneel before zod!
16:24:00 <siddharthvipul> great
16:24:05 <siddharthvipul> #info Summary of last week: (from current oncall)
16:24:08 <siddharthvipul> smooge, hey
16:24:11 <siddharthvipul> you around?
16:24:21 <smooge> oh sorry
16:24:28 <smooge> too many windows.
16:24:59 <smooge> I had a couple of oncalls. One was for a bad permissions on NFS and another was on oci running out of disk space.
16:25:04 <smooge> otherwise it was quiet
16:26:08 <siddharthvipul> nice
16:26:17 <nirik> cool
16:26:21 <siddharthvipul> #topic Monitoring discussion [nirik]
16:26:22 <siddharthvipul> #info https://nagios.fedoraproject.org/nagios
16:26:31 <pingou> ComputerKid: there is no expectation to be online 24/7 when oncall, the message returned by zodbot says that if the oncall person does not respond a ticket is way to go
16:27:04 <nirik> so, I think we are pretty much the same as last week, but let me see.
16:27:14 <nirik> I did want to note one thing:
16:27:38 <nirik> There's some services we have to keep restarting all the time. We should look at making nagios do so for us.
16:27:59 <nirik> Those are: pdc-web01 httpd, pdc-web02 httpd and resultsdb01 httpd
16:28:50 <nirik> we still have 2 down aarch64 boxes... one with a bad drive, one needs power recycling. We haven't made much progress on those... smooge: perhaps you could bug people about pdus again?
16:29:28 <nirik> badges-backend has a big backlog. perhaps we could ask misc to look into it?
16:30:03 <nirik> the bodhi-sync-listener queue is big... we need to fix that from the main/rawhide branch changes:
16:30:06 <nirik> RABBITMQ_QUEUE CRITICAL - messages CRITICAL (167), messages_ready OK (167) messages_unacknowledged OK (0) consumers OK (0)
16:30:28 <nirik> otherwise it's all the old small things. ;)
16:30:35 <nirik> and we can move along.
16:30:49 <siddharthvipul> thank you nirik (as always)
16:31:08 <siddharthvipul> oh wow.. we are already here
16:31:09 <siddharthvipul> #topic Learning topic
16:31:23 <siddharthvipul> #info IPA [nirik]
16:31:28 <siddharthvipul> nirik, we are doing this, right?
16:31:28 <nirik> ah yes...
16:31:42 <siddharthvipul> :excited: :D
16:31:47 <ComputerKid> Same
16:31:48 <nirik> sure, I can give a overview of our setup... but probibly not a ton of detail. :)
16:31:51 <smooge> nirik the people with pdu access will not be able to fix it until they get to the dc
16:32:37 <nirik> so... we have been using ipa for about 4 years now with our current account system: fas.
16:33:22 <nirik> ipa provides a full suite of identity and auth management... it can do kerberos auth, dns, certificiate management, and all sorts of things. :)
16:33:48 <nirik> right now however we are only using it for kerberos authentication.
16:34:40 <nirik> The current system works by syncing data from our fas system on user login (to fas) to the ipa cluster, then later when a user uses kinit to get a kerberos ticket they talk to the ipa server, it has their password and they get a ticket, etc. :)
16:35:16 <nirik> Very soon however, our new replacement account system (noggin) is going to move into production. It's already there in staging.
16:35:49 <nirik> With the new setup, noggin is just a frontend to ipa. It uses ipa to store everything, it doesn't keep info itself.
16:35:49 <dtometzki> and for sso is it used too ?
16:36:33 <nirik> dtometzki: there's another part to our setup: ipsilon. ipsilon handles things like OIDC, SAML2, openid and such.
16:36:50 <nirik> ipsilon is going to stay the same with the new setup (at least for now)
16:36:54 <dtometzki> ahh
16:37:53 <darknao> is ipsilon related in some way to keycloak ?
16:38:09 <nirik> Under the current setup, ssh access to hosts is via a fasClient script. It gathers info from fas on users/groups and sets up those on the local host.
16:38:33 <nirik> Under the current setup, sudo is using pam_url and totpcgi
16:39:06 <nirik> ipsilon and keycloak handle the same usecases (mostly). ;) Just one is in python the other in java. And keycloak doesn't do openid
16:39:32 <nirik> Under the new setup, ssh and sudo will be via sssd and ipa. So much more standard.
16:39:54 <nirik> You can see the noggin interface in stg:
16:40:04 <nirik> https://admin.stg.fedoraproject.org/accounts/
16:40:37 <nirik> (all the prod accounts have been synced, but there's some issues with group membership). Probibly you will want to use 'forgot password' to reset your password at first
16:41:06 <nirik> Down the road we may move to keycloak from ipsilon, but no timetable for that yet...
16:42:08 <nirik> lets see... what else.
16:42:54 <nirik> Right now in prod we have 2 ipa servers in the cluster.
16:43:13 <nirik> Likely we will add a few more before we roll out the new setup (as they are gonna be under some more load)
16:43:40 <nirik> ipa is pretty cool in that you can pretty easily add more masters and they handle all the syncing and keeping them up to date.
16:44:23 <nirik> when we moved datacenters, we setup these 2 masters in the new dc and they synced with the old ones, then after moving we just turned off the old ones. :)
16:44:47 <dtometzki> fine i can login
16:45:11 <nirik> cool.
16:45:27 <nirik> So, any questions or things you would like me to expand on?
16:46:12 <nirik> we are still sorting out 2fa stuff... it's mostly working tho.
16:47:10 <nirik> sadly there's some more work to get kinit working with 2fa setups, but hopefully we can reduce that.
16:48:27 <nirik> ok, as always feel free to ask questions anytime in #fedora-admin or the like. ;)
16:48:37 <siddharthvipul> awesome
16:48:45 <siddharthvipul> nirik, thanks a lot for taking these
16:48:51 <siddharthvipul> very interesting and definitely learning a lot
16:49:10 <siddharthvipul> tomorrow for learning day, I am going to visit all the notes once more and might disturb you all next week :)
16:49:28 <siddharthvipul> if there are not questions, should we move to open floor?
16:50:24 <Zlopez[m]> I plan to play with my PinePhone and try to run Fedora on it :-)
16:50:39 <siddharthvipul> Zlopez[m], oh, very cool!
16:50:51 <nirik> Zlopez[m]: see #fedora-phone. ;)
16:51:14 <Zlopez[m]> nirik:  I wasn't aware there is a channel on freenode
16:51:20 <Zlopez[m]> nirik++
16:51:38 <siddharthvipul> you know what.. I am really excited for element now (helps with discovering channels)
16:51:45 <siddharthvipul> #topic Open Floor
16:51:58 <siddharthvipul> element == Fedora home matrix server
16:52:10 <siddharthvipul> It's too new and shiny for me.. but I will get used to it soon :P
16:52:25 <siddharthvipul> offers a ton of nice handy things in pros
16:52:35 <Zlopez[m]> siddharthvipul: Yes, I'm using it too
16:53:14 <dtometzki> i need any permission to do my oncall job next week ?
16:53:15 <siddharthvipul> Zlopez[m], I am not.. just using it for FOSDEM but might move later next week or month
16:53:59 <siddharthvipul> dtometzki, we can sort it out in next week call. but you will take the duty with  .oncalltakeeu or .oncalltakeus (depends on timezone)
16:54:13 <siddharthvipul> you have to be verified with zodbot
16:54:20 <siddharthvipul> now now, let me remember how to do that
16:54:28 <siddharthvipul> .whoami
16:54:28 <zodbot> siddharthvipul: siddharthvipul1
16:55:03 <ComputerKid> Thanks for the talk on ipa
16:55:05 <ComputerKid> Was cool
16:55:54 <ComputerKid> Is the meeting over?
16:56:01 <siddharthvipul> can anyone help me fetch the link or command to verify fas id with zodbot?
16:56:06 <siddharthvipul> ComputerKid, we have a few more minutes :)
16:56:37 <Zlopez[m]> user identify <username> <password>
16:56:42 <siddharthvipul> Zlopez[m], thank you
16:56:45 <siddharthvipul> dtometzki, ^
16:56:58 <Zlopez[m]> Just sent this to zodbot in private talk
16:57:04 <siddharthvipul> password being your freenode password (this is to sent to zodbot in private)
16:58:06 <siddharthvipul> okay folks.. if nothing else I will close the meeting in 60 seconds
16:58:18 <siddharthvipul> or 40-100 seconds in Indian time zone :P
16:58:31 <Zlopez[m]> thanks for chairing siddharthvipul
16:58:40 <siddharthvipul> Zlopez[m], my pleasure :)
16:58:41 <Zlopez[m]> siddhartvipul++
16:58:58 <Zlopez[m]> It looks like I gave all my cookies already :-D
16:58:59 <darknao> thanks siddharthvipul, and nirik
16:59:06 <siddharthvipul> Zlopez[m], I am sad
16:59:17 <Zlopez[m]> .thanks siddharthvipul
16:59:17 <zodbot> Zlopez [m] thinks siddharthvipul is awesome and is happy they are helping! (Please also type siddharthvipul++ since that is what gives them a cookie)
16:59:17 <darknao> siddharthvipul++
16:59:28 <siddharthvipul> haha
16:59:32 <siddharthvipul> #endmeeting