16:00:12 <lenkaseg__> #startmeeting Infrastructure (2023-06-01)
16:00:12 <zodbot> Meeting started Thu Jun  1 16:00:12 2023 UTC.
16:00:12 <zodbot> This meeting is logged and archived in a public location.
16:00:12 <zodbot> The chair is lenkaseg__. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:00:12 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:12 <zodbot> The meeting name has been set to 'infrastructure_(2023-06-01)'
16:00:23 <lenkaseg__> #meetingname infrastructure
16:00:23 <zodbot> The meeting name has been set to 'infrastructure'
16:00:30 <lenkaseg__> #chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg
16:00:30 <zodbot> Current chairs: bodanel dtometzki jnsamyak lenkaseg lenkaseg__ nb nirik zlopez
16:00:38 <lenkaseg__> #info Agenda is at: https://board.net/p/fedora-infra
16:00:47 <lenkaseg__> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:00:53 <lenkaseg__> #info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
16:00:56 <nirik> morning
16:00:59 <lenkaseg__> #topic greetings!
16:01:04 <SeddikAlaouiIsma> Hi
16:01:05 <lenkaseg__> Hello everybody!
16:01:06 <jednorozec> .hello humaton
16:01:07 <zodbot> jednorozec: humaton 'Tomáš Hrčka' <thrcka@redhat.com>
16:01:12 <leonkhan> good morning/afternoon everyone :-)
16:01:15 <lenkaseg> .hi
16:01:16 <zodbot> lenkaseg: lenkaseg 'Lenka Segura' <lenka@sepu.cz>
16:02:14 <mkonecny> .hello zlopez
16:02:15 <zodbot> mkonecny: zlopez 'Michal Konecny' <michal.konecny@pacse.eu>
16:02:26 <lenkaseg__> #topic New folks introductions
16:02:45 <lenkaseg__> Do we have new people here? Please say hello!
16:03:01 <lenkaseg__> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:03:12 <lenkaseg__> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:04:30 <lenkaseg> #topic announcements and information
16:04:40 <eddiejenningsjr> .hello eddiejennings
16:04:41 <zodbot> eddiejenningsjr: eddiejennings 'Eddie Jennings' <eddie@eddiejennings.net>
16:04:47 <lenkaseg> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0730 UTC in #centos-meeting
16:05:00 <lenkaseg> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:05:09 <lenkaseg> #info Fedora 38 Release party this Friday + Saturday! Register on Hopin: https://hopin.com/events/fedora-linux-38-release-party/registration
16:05:42 <nirik> #info wiki upgraded to f38/latest mediawiki
16:05:53 <nirik> #info koji outage today of database and s390x builders
16:06:19 <lenkaseg> Thanks nirik!
16:06:34 <lenkaseg> Do we have more info&announcement?
16:06:56 <lenkaseg> Drop a smiley if you plan to attend tomorrow's release party!
16:07:43 <nirik> I'm gonna try (but it starts early my time) ☺️
16:07:48 <lenkaseg> 🎉
16:08:06 <eddiejenningsjr> I will try.  Depends on work
16:09:24 <leonkhan> :)
16:09:29 <mkonecny> I will be there, but not sure how much. I plan to travel home at the same time
16:09:43 <lenkaseg> #topic Oncall
16:09:50 <lenkaseg> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:09:58 <lenkaseg> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
16:10:05 <lenkaseg> ## .oncalltakeeu .oncalltakeus
16:10:14 <lenkaseg> #info nirik is on call from 2023-05-26 to 2023-06-01
16:10:23 <lenkaseg> #info eddiejennings is on call from 2023-06-02 to 2023-06-08
16:10:30 <lenkaseg> #info dtometzki is on call from 2023-06-09 to 2023-06-15
16:11:01 <eddiejenningsjr> Ah, matrix!  Makes it easy to copy/paste what's needed for Zod to remember me :)
16:11:11 <eddiejenningsjr> .oncalltakeus
16:11:11 <zodbot> eddiejenningsjr: Kneel before zod!
16:11:16 <lenkaseg> Does somebody want to take oncall for the week 16-22?
16:12:05 <lenkaseg> let's leave it for next week then
16:12:27 <lenkaseg> Oops, I skipped the part of the next chair
16:12:43 <lenkaseg> #info chair 2023-06-08 - mkonecny
16:12:51 <lenkaseg> #topic Next chair
16:12:57 <lenkaseg> #info magic eight ball says:
16:13:02 <lenkaseg> #info chair 2023-06-01 - lenkaseg
16:13:07 <lenkaseg> #info chair 2023-06-08 - mkonecny
16:13:31 <lenkaseg> Does somebody want to chair 15th of June?
16:15:12 <lenkaseg> Ok, could be me :)
16:15:16 <eddiejenningsjr> I can chair on the 15th
16:15:30 <lenkaseg> Alright then! :)
16:16:10 <lenkaseg> #info chair 2023-06-15 - eddiejennings
16:16:10 <nirik> thanks Eddie Jennings, Jr.
16:16:12 <lenkaseg> and I can take the next one .... probably I'll be here
16:16:30 <lenkaseg> #info chair 2023-06-22 - lenkaseg
16:16:35 <eddiejenningsjr> :D
16:16:46 <lenkaseg> nirik, were there some oncalls in the past week?
16:17:16 <nirik> nope, it was quiet again... :)
16:17:45 <lenkaseg> 👍️
16:18:24 <lenkaseg> ah, that sould be mentioned under this info, heh:
16:18:25 <lenkaseg> #info Summary of last week: (from current oncall )
16:18:41 <lenkaseg> next topic!
16:18:42 <lenkaseg> #topic Monitoring discussion [nirik]
16:18:50 <lenkaseg> over to you nirik
16:19:15 <nirik> Yeah, so right now it looks a bit grim, but thats only because koji and the s390x builders are down for the outage. ;)
16:19:28 <nirik> but otherwise, aheath1992 did some work to clean up some old alerts.
16:19:41 <nirik> we have 2 still alerting that we haven't been able to figure out tho:
16:20:09 <nirik> Check fedmsg-hub consumers backlog This service has 1 comment associated with it	This service problem has been acknowledged	UNKNOWN 	06-01-2023 16:13:55 	0d 17h 56m 10s 	3/3 	UNKNOWN - ZMQ timeout. No message received in 20000 ms
16:20:13 <nirik> on notifs-backend01
16:20:30 <nirik> and another one there... its querying the right socket now, but timing out for some reason.
16:20:42 <mkonecny> This is the old FMN?
16:20:50 <nirik> yes
16:20:53 <nirik> https://nagios.fedoraproject.org/nagios/cgi-bin//status.cgi?host=all&type=detail&servicestatustypes=8&serviceprops=4
16:21:41 <nirik> otherwise we are looking better than we have in a while. :) We have seen some new fmn alerts still, but not too many.
16:21:52 <nirik> Thats all I had unless someone had questions
16:23:10 <lenkaseg> seems there are no questions
16:23:35 <lenkaseg> now, I think this week should be a learning topic, but I can't see any planned...
16:23:44 <lenkaseg> the Security roundtable?
16:23:55 <lenkaseg> or let's go to backlog refinement?
16:24:12 <nirik> either way is fine with me... whatever people vote for. ;)
16:25:46 <lenkaseg> How does the Security roundtable work? :)
16:26:18 <lenkaseg> we discuss some CVEs?
16:26:42 <eddiejenningsjr> Sure.
16:26:55 <nirik> not sure, it's the first one. ;) if people like I can give a short overview of how we do security stuff in infra... but I think the idea was that everyone shares their setup/thoughts or something?
16:26:58 <eddiejenningsjr> Could also discuss good practices.
16:27:13 <nirik> yeah.
16:28:35 <leonkhan> I was wondering how does the overall security of the infra team managed ?
16:29:02 <leonkhan> is there a layered approach ? like application , OS , Network etc ?
16:29:59 <leonkhan> does the team use any monitoring tools ?
16:30:02 <nirik> Yeah, it's a big subject. :) But we: apply any security update packages nightly (dnf-automatic) and from time to time apply all other updates.
16:30:13 <nirik> For very important CVEs we apply them right away manually.
16:30:58 <lenkaseg> I'm always confused what nightly means. Night in which timezone?
16:30:59 <nirik> We have been exploring the idea of running ACS in our openshift clusters (advanced cluster security) it basically scans images for issues and raises alerts about them.
16:32:32 <leonkhan> is there any team member dedicated who work on the security issues ?
16:32:56 <nirik> it's early in the day UTC...
16:33:01 <nirik> I don't recall the exact time.
16:35:16 <leonkhan> we don't have any SOC ( security operations center ) right ?
16:36:08 <nirik> not really no...
16:36:30 <nirik> I think the dnf-automatic stuff has some randomness to when it runs so it doesn't have everyone check at the same time.
16:37:10 <nirik> For laptops/client workstations we used to have a thing called CSI...
16:37:26 <nirik> https://infrastructure.fedoraproject.org/csi/security-policy/en-US/html-single/ has still an old copy of it. (warning, it's really old)
16:39:39 <nirik> some good concepts still in there tho
16:41:03 <eddiejenningsjr> I have an ansible-vault scenario that other may have encountered, which may be worth discussing (security related).
16:41:40 <nirik> sure, go for it. ;) We don't use ansible-vault, but I know many do...
16:42:05 <mkonecny> I also think we have some recommendations in Infra SOP for developers
16:42:07 <eddiejenningsjr> So you have an Ansible playbook that's using some variables encrypted by vault, since the variables contain credentials, and you don't want that in plain text in your git repository.  You'll be running this playbook as part of a cron job / systemd timer.  Are there better ways to handle the passing of the vault password than having a file on the system with the vault password, with permissions set to where only root can read it
16:42:07 <eddiejenningsjr> and using --vault-password-file in your task command?
16:42:31 <mkonecny> And here it is https://docs.fedoraproject.org/en-US/infra/developer_guide/security_policy/
16:44:01 <nirik> I don't see any other option for unattended/cron off hand there.
16:44:12 <eddiejenningsjr> I've seen some demos of AAP, and there's a credential managing thing in that, which I think speaks to this scenario.  I haven't used AWX, which I imagine has the same kind of thing.  But for good ole ansible-core folks, I can't think of another way.
16:44:43 <eddiejenningsjr> Maybe set an environment variable for the service account that runs the task that has the password, but that's still having it in plain text in a bashrc or something like that.
16:45:33 <eddiejenningsjr> Figured I'd bring it here to see what you folks thought (and if we did use ansible-vault with Fedora Infra, see how it's handled) :)
16:48:09 <nirik> yeah, makes sense... we don't use vault tho, so dunno. ;)
16:50:53 <lenkaseg> cool, we should have security roundtable more often :)
16:51:06 <leonkhan> :-)
16:51:15 <lenkaseg> If there's nothing more to the discussion, let's move either to see some tickets, or open floor since we have 10 last minuts
16:51:30 <nirik> Always good to talk on various topics. ;) security is pretty interesting...
16:52:15 <eddiejenningsjr> I'm going to have to depart early.  Being summoned by work.  Have a good day, all!
16:52:33 <lenkaseg> have a nice day Eddie Jennings, Jr.
16:52:36 <lenkaseg> #topic Open Floor
16:52:49 <mkonecny> Have a nice day Eddie Jennings, Jr.
16:53:08 <SeddikAlaouiIsma> @nirik Hope you can talk about fedora-ci
16:53:31 <mkonecny> We should ask somebody from the CI folks about that
16:53:48 <SeddikAlaouiIsma> Nice 😊
16:55:21 <lenkaseg> Yep, we didn't have anything about CI for some time...
16:55:29 <nirik> yeah, I don't know too much about the ci... but I would love to know more
16:55:54 <nirik> perhaps mvadkert could talk about it, or point us to someone who could
16:56:12 <SeddikAlaouiIsma> I have another question 😅
16:56:12 <SeddikAlaouiIsma> Flock event!! How we can register??
16:56:20 <SeddikAlaouiIsma> I don't see any link for that..
16:56:30 <mkonecny> I will add to my TODO list to ask on #fedora-ci channel
16:56:42 <nirik> I don't think registration is open yet... or call for papers/talks (but soon)
16:57:01 <mkonecny> Yeah, not open yet
16:57:09 <lenkaseg> https://flocktofedora.org/ will appear here
16:57:12 <nirik> perhaps there will be more info at the release party? not sure.
16:58:11 <SeddikAlaouiIsma> Maybe yes!!
16:59:34 <nirik> hope so. :)
17:00:06 <lenkaseg> #endmeeting