15:00:41 <smooge> #startmeeting Infrastructure (2020-04-30)
15:00:41 <zodbot> Meeting started Thu Apr 30 15:00:41 2020 UTC.
15:00:41 <zodbot> This meeting is logged and archived in a public location.
15:00:41 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:41 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:41 <zodbot> The meeting name has been set to 'infrastructure_(2020-04-30)'
15:00:41 <smooge> #meetingname infrastructure
15:00:41 <zodbot> The meeting name has been set to 'infrastructure'
15:00:41 <smooge> #chair nirik pingou smooge cverna mizdebsk mkonecny abompard
15:00:41 <smooge> #info Agenda is at: https://board.net/p/fedora-infra
15:00:42 <zodbot> Current chairs: abompard cverna mizdebsk mkonecny nirik pingou smooge
15:00:42 <smooge> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
15:00:43 <smooge> #topic aloha
15:00:55 <sincorchetes> .hello2
15:00:56 <zodbot> sincorchetes: sincorchetes 'Alvaro Castillo' <sincorchetes@gmail.com>
15:01:08 <nirik> morning.
15:01:09 <pingou> .hello2
15:01:11 <zodbot> pingou: pingou 'Pierre-YvesChibon' <pingou@pingoured.fr>
15:01:13 <austinpowered> .hello2 austinpowered
15:01:14 <zodbot> austinpowered: austinpowered 'None' <fedoraproject@wootenwilliams.com>
15:01:15 <iamkc> .hello2
15:01:17 <zodbot> iamkc: iamkc 'None' <iamkc@tuta.io>
15:01:21 <mobrien[m]> .hello2
15:01:22 <zodbot> mobrien[m]: Sorry, but you don't exist
15:01:40 <amrmzaki> .hello2
15:01:42 <zodbot> amrmzaki: Sorry, but you don't exist
15:01:42 <mobrien[m]> .hello mobrien
15:01:45 <zodbot> mobrien[m]: mobrien 'Mark O'Brien' <markobri@redhat.com>
15:02:28 <mkonecny> .hello zlopez
15:02:29 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@packetseekers.eu>
15:02:42 <smooge> #topic Next chair
15:02:42 <smooge> #info magic eight ball says:
15:02:42 <smooge> #info 2020-04-30 - smooge
15:02:42 <smooge> #info 2020-05-07 - cverna
15:02:43 <smooge> #info 2020-05-14 - siddharthvipul
15:02:44 <smooge> #info 2020-05-21 - ???
15:02:57 <mkonecny> smooge: I could take it
15:03:01 <smooge> any volunteers for the 21st or wait til next week to fill it?
15:03:10 <smooge> thanks mkonecny you run a great meeting
15:03:37 <mkonecny> thanks smooge, I will try to run another great meeting
15:03:51 <smooge> #topic announcements and information
15:03:51 <smooge> #info CPE Sustaining EU-hours team has standups on Tuesday and Thursday at 1400 UTC in #fedora-admin - please join
15:03:51 <smooge> #info CPE Sustaining NA-hours team has a Monday through Friday 30 minute meeting going through tickets at 1800 UTC in #fedora-admin
15:03:51 <smooge> #info Fedora Infrastructure will be moving in 2020-06 from its Phoenix Az datacenter to one near Herndon Va. A lot of planning will be involved on this. Please watch out for announcements on changes.
15:03:52 <smooge> #info Fedora Communishift move has started but will take longer than expected. Current estimate for bringing back into production is TBD
15:03:57 <smooge> #info F32 final freeze is in effect!
15:03:59 <smooge> #info Taskotron is EOL and turned off as of today
15:04:01 <smooge> #undo
15:04:01 <zodbot> Removing item from minutes: INFO by smooge at 15:03:59 : Taskotron is EOL and turned off as of today
15:04:03 <smooge> #undo
15:04:03 <zodbot> Removing item from minutes: INFO by smooge at 15:03:57 : F32 final freeze is in effect!
15:04:23 <nirik> #info Fedora 32 is out the door!
15:04:23 <smooge> #info Taskotron is EOL and turned off as of today
15:04:42 <smooge> smooge does not look at the notes as closely as he should
15:05:14 <smooge> #topic Oncall
15:05:14 <smooge> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
15:05:15 <smooge> #info cverna is oncall 2020-04-23-> 2020-04-30
15:05:15 <smooge> #info siddharthvipul is oncall 2020-04-30 -> 2020-05-07
15:05:15 <smooge> #info  ??? is oncall 2020-05-07 -> 2020-05-14
15:05:15 <nils> .hello nphilipp
15:05:16 <smooge> #info ??? is oncall 2020-05-14 -> 2020-05-21
15:05:17 <zodbot> nils: nphilipp 'Nils Philippsen' <nphilipp@redhat.com>
15:05:48 <smooge> cverna how was this week?
15:05:52 <smooge> #info Summary of last week: (from current oncall )
15:06:17 <cverna> A few pings, mailman is in a weird state
15:06:51 <cverna> And also some a few requests about ssh access and badges not working
15:07:10 <cverna> Other than that it was a good week :)
15:07:29 * nirik left the badges thing to misc
15:07:50 <smooge> #topic Monitoring discussion [nirik]
15:07:50 <smooge> #info https://nagios.fedoraproject.org/nagios
15:07:50 <smooge> #info Go over existing out items and fix
15:08:21 <nirik> I'd like to update/reboot those buildvmhosts with low swap soon...
15:08:27 <nirik> supposedly thats fixed in 8.2
15:08:44 <nirik> fas and planet messages still not coming thru
15:09:01 <nirik> notfs-backend01 is low on space.
15:09:11 <nirik> I guess we need to just grow it... ?
15:09:19 <pingou> notifs-backend01 is sending a lot of emails
15:09:24 <pingou> really a lot :(
15:09:26 * smooge looks forward to us moving to a new datacentre where non of this will happen again
15:09:29 <nirik> yeah, it does...
15:09:42 <nirik> smooge: suuuuuuuuuuuuuuuure it won't
15:09:53 <pingou> (and I mean: error emails, not genuine email notifications to people)
15:09:56 <pingou> smooge++
15:10:12 <nirik> pingou: oh? on what? is that likely related to the disk space issue?
15:10:47 <nirik> we also have torrent02 thats low on disk. When f30 goes EOL we can nuke all the f30 images... or we could increase space there
15:10:50 <pingou> nirik: bodhi broke its message format when it moved to fedora-messaging and fedmsg_meta_fedora_infra never got adjusted
15:11:23 <pingou> so every bodhi message ends up triggering an error (I guess a certain % of bodhi notifications sent by FMN have been dropped)
15:11:35 <mkonecny> pingou: Is fedmsg_meta_fedora_infra used for fedora-messaging messages?
15:11:36 <nirik> :(
15:11:56 <pingou> and the workers used to have a limit (at 100) of the number of time they send an error, and it looks like they either no longer have that limit or it's broken
15:11:58 <nirik> well, I am happy to see it's on our scoped backlog to re-do FMN.
15:12:12 <pingou> so I ended up recently with 2k+ emails from notifs-backend01
15:12:36 <pingou> mkonecny: nope
15:12:37 <nirik> pingou: hum, I wonder... do any of those go to mailing lists? (ie, could this be causing our mailman api issues)?
15:12:41 <pingou> mkonecny: for fmn uses it
15:13:05 <pingou> nirik: the error emails are sent to sysadmin-datanommer-members@fedoraproject.org
15:13:14 <nirik> ok.
15:13:24 <pingou> FMN used to sent the notifs on the mailing list that monitors all the commits
15:13:34 <pingou> but I've not heard that broke
15:13:49 <nirik> scm-commits... yeah, thats stil working fine
15:14:00 <cverna> I thought Bodhi was sending emails directly, does FMN needs to know about Bodhi ?
15:14:29 <pingou> FMN was meant to be the way notifications are received
15:14:38 * smooge really wonders if we turned off notif emails for a week if anyone would notice.. since most of the people I have contacted have been sending it to devnull
15:14:46 <pingou> so in theory people could filter out the notifs from bodhi and only get the ones they want fro FMN
15:15:00 <pingou> whether there is such case or not, is another question
15:15:09 <pingou> but bodhi notifs are part of the default rules set in FMN
15:15:13 <nirik> smooge: that would just cause the other 1/2 that really uses them to yell at us. ;)
15:15:26 <cverna> ha ok
15:15:35 <smooge> well I would prefer to know it and have them turn it back on
15:15:36 <nirik> bodhi used to do it's own email because it had to handle anon comments notifications.
15:15:52 <nirik> I think turning them off would be a pretty big security risk.
15:16:17 <pingou> we no longer allow annonimous comments though, do we?
15:16:23 <smooge> but I am mostly grousing on clearing bastion mailq of 40000 scm-commit-emails that were backed up
15:16:28 <nirik> we don't. So it could stop doing thaty now
15:16:44 <pingou> anyway, FMN needs a v4
15:16:45 <nirik> smooge: yeah, people who can't handle that volume should not subscribe to that
15:17:32 <cverna> pingou: no that was removed
15:17:46 <smooge> anyway.. agreed.. I will grouse another time
15:17:48 <nirik> anyhow, we should try and fix those low disk things... but I guess we could move on now from this topic.
15:18:11 <smooge> #topic Open Floor
15:18:23 <smooge> thats the next one in the doc
15:18:31 <nirik> smooge: want to give a datacenter update? (which we should also add to the doc)
15:18:40 <mkonecny> The door to the floor are now open
15:18:53 * nirik can, but my info is from last night, might be stuff this morning I don't know of.
15:18:57 <smooge> OK network changes we have been waiting on got approved and put through this morning at 5am
15:19:08 * pingou has 2 things for the open-floor
15:19:09 <nirik> hurray!
15:19:20 <smooge> I am working with networking on getting other ports open and have a firewall doc to go through
15:19:21 <nirik> 🥂🥂
15:19:54 <smooge> woke up at 3am worried about it and sat with the admins to mke sure it worked
15:20:26 <nirik> let me know when I can get in and start working on things.
15:20:32 <smooge> #info RDU-CC Hosts are preliminarily wired but a switch died in transit and needs repairs. Work will be done on that Friday
15:20:34 <nirik> smooge++
15:20:34 <zodbot> nirik: Karma for smooge changed to 4 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:20:51 <nirik> friday as tomorrow?
15:21:12 * sincorchetes has a 1 question for open-floor
15:21:18 <cverna> smooge++
15:21:18 <zodbot> cverna: Karma for smooge changed to 5 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:21:18 <smooge> as in tomorrow. I get up early, drive to dc, fix some wiring, label things in a spreadsheet, and move on
15:21:40 <mkonecny> smooge++
15:21:40 <zodbot> mkonecny: Karma for smooge changed to 6 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:21:50 <sincorchetes> smooge++
15:21:50 <zodbot> sincorchetes: Karma for smooge changed to 7 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:22:17 <nirik> smooge: thanks for the update smooge. I look forward to bringing those things all online.
15:22:32 <smooge> #info IAD2 initial networking routes inside of RH have been done. This means various chains of data can start to be replicated and we can begin standup of the hosts. Various steps still need to be case by case as cross networking firewalls have not been done
15:22:46 <smooge> and that is all for stuff.
15:23:04 <smooge> any questions or pingou called open floor first
15:23:12 <pingou> sincorchetes: go for it
15:23:22 <pingou> (if there are no questions for smooge )
15:23:41 * sincorchetes was wrong he thought this was "open questions" not pre-meting to choose what's topics will be talk
15:23:46 <mkonecny> not questions just words of thanks
15:24:32 <pingou> it's open floor, if you have a question go for it :)
15:24:55 <sincorchetes> oh ok! thanks, heh, I'm newbie here. If I wa/me asking about Fedora apprentice program in infra? for example, If I modified a Ansible playbok. How I could test it changes without broke productions servers?
15:25:44 <nirik> it's tricky. ;) We do have a staging setup thats just like (for some values of like) production, so we can try the playbook there...
15:26:34 <iamkc> hello, i'm new to the project, how can join apprentice group?
15:26:38 <nirik> also it's possible to run things with --check --diff and see what changes it would make... that can often help test complex changes
15:27:48 <cverna> Although only sysadmin-main can run in check mode afaik
15:28:33 <nirik> yeah, that might be. I thought we added something around that to rbac-playbook, but would have to look.
15:28:41 <siddharthvipul> sorry all, my power went off a while ago and it's not back it
15:28:46 <nirik> iamkc: see us in #fedora-admin after the meeting and we can add folks there.
15:28:51 <siddharthvipul> let me know when i should do oncall thing
15:28:54 <pingou> siddharthvipul: cycle faster!
15:29:09 <iamkc> Thanks
15:29:19 <smooge> siddharthvipul, you can start anytime
15:29:22 <siddharthvipul> pingou: do I look like someone who moves lol
15:29:29 <siddharthvipul> .takeoncalleu
15:29:53 <pingou> oncalltakeeu
15:30:04 <sincorchetes> or maybe create a test lab that would be recreate to test playbooks there without production data?
15:30:04 <siddharthvipul> of course it's not what I thought :)
15:30:07 <smooge> I don't know why it is that way
15:30:10 <siddharthvipul> .oncalltakeeu
15:30:10 <zodbot> siddharthvipul: Kneel before zod!
15:30:25 <smooge> ok pingou your turn
15:30:31 <nirik> sincorchetes: this should be possible too, but might need some adjustments on paths/etc.
15:30:58 <pingou> so together with smooge and nils and mobrien[m] we ensured batcave is setup to support shared tmux session
15:31:02 <pingou> it's documented in : https://pagure.io/Fedora-Infra/howtos/blob/master/f/share_tmux_session.md
15:31:25 <pingou> with this you can share a tmux session with anyone, allowing to easily shadow one another and collaborate on something
15:31:28 <nirik> kudos!
15:31:29 <amrmzaki> pingou++
15:31:41 <sincorchetes> pingou++
15:31:41 <zodbot> sincorchetes: Karma for pingou changed to 3 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:31:44 <pingou> well smooge++ nils++ mobrien++ as well :)
15:31:44 <zodbot> pingou: Karma for mobrien changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:31:47 <nirik> pingou++ nils++ mobrien[m]++
15:31:47 <zodbot> nirik: Karma for pingou changed to 4 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:31:50 <zodbot> nirik: Karma for nphilipp changed to 2 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:31:52 <mobrien[m]> pingou++
15:32:07 * nirik gives out all his fresh cookies
15:32:08 <nils> mobrien[m]++
15:32:24 <pingou> the second item is nirik and I have been looking at moving our ansible repo from the batcave into pagure.io
15:32:25 <smooge> pingou, second item?
15:32:31 <mkonecny> pingou++
15:32:31 <zodbot> mkonecny: Karma for pingou changed to 5 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:32:34 <smooge> ah smooge slow
15:32:43 <smooge> ok can we stop the cookies til after the meeting
15:32:55 <pingou> there was one issue with the fedora-messaging consumer which abompard tracked down this morning (took some effort but ended up being a bug in the doc)
15:33:06 <nirik> ah ha. cool.
15:33:09 <cverna> Cool there is also the tmate package which is really neat.
15:33:11 <smooge> yay!
15:33:29 <pingou> I'm running it locally on batcave for a little bit to see how it behaves
15:33:45 <pingou> but I think we should be able to announce a cut-over day and migrate
15:34:15 <pingou> I'll be looking at merging the ansible repo with the fedora-infrastructure one as praiskup described in the infra list
15:34:20 <nirik> excellent.
15:34:21 <pingou> it seems like the best of both world
15:34:24 <smooge> ok one item.. I would like to do this before we swap over to the next datacentre if possible
15:34:28 <nirik> let me know if I can do anything to assist further pingou
15:34:42 <pingou> nirik: wanna send the announce? :)
15:34:44 <nirik> it should be a pretty short outage I think...
15:34:49 <pingou> yup
15:34:52 <nirik> and yes, I'd like to do it soon.
15:35:04 <amrmzaki> what's about easy fix problems. i explored the issues and didn't figure out any of them :)
15:35:06 <smooge> next week? monday ro tuesday?
15:35:12 <pingou> I'm afk tomorrow, but early next week wfm
15:35:33 <nirik> I could possibly do it this weekend, but might be good to have pingou around...
15:35:43 <nirik> monday is kinda crazy, so how about tuesday
15:35:54 <pingou> nirik: in an evening that's potentially doable
15:36:02 <smooge> ok tuesday.. 1500 UTC or later?
15:36:27 * nirik needs to check calendar.
15:36:39 <nirik> lets figure it outside meeting. I can send an announce...
15:36:50 <smooge> #info Fedora Infra will be moving its ansible to pagure.io next week
15:36:55 <pingou> 15 UTC is good for me, after, from 1630 UTC till 1900 UTC I'd like to be afk
15:36:58 <smooge> #info time/date to be determined
15:37:12 <pingou> but I can be there from 1900 UTC
15:37:26 <smooge> ok anything else for this meeting. my coffee is done for today and i need to make a fresh pot
15:37:38 <pingou> eol for me
15:38:42 <smooge> #endmeeting