15:00:19 <smooge> #startmeeting Infrastructure (2019-07-18)
15:00:19 <zodbot> Meeting started Thu Jul 18 15:00:19 2019 UTC.
15:00:19 <zodbot> This meeting is logged and archived in a public location.
15:00:19 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:19 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:19 <zodbot> The meeting name has been set to 'infrastructure_(2019-07-18)'
15:00:19 <smooge> #meetingname infrastructure
15:00:19 <smooge> #topic aloha
15:00:19 <smooge> #chair nirik pingou puiterwijk relrod smooge tflink cverna mizdebsk mkonecny abompard bowlofeggs
15:00:20 <zodbot> The meeting name has been set to 'infrastructure'
15:00:20 <zodbot> Current chairs: abompard bowlofeggs cverna mizdebsk mkonecny nirik pingou puiterwijk relrod smooge tflink
15:00:26 <smooge> hello people
15:00:49 <relrod> morning
15:00:54 <nirik> morning
15:01:00 <mkonecny> .hello zlopez
15:01:01 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@packetseekers.eu>
15:02:46 <smooge> #topic New folks introductions
15:02:46 <smooge> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:02:46 <smooge> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
15:02:46 <smooge> #knstn introduction
15:02:52 <smooge> #info knstn introduction
15:03:10 <knstn> Hi, i'm here
15:03:12 <smooge> we have a new person knstn who has introduced themselves on the channels
15:03:18 <smooge> ehllo
15:03:22 <nirik> welcome knstn
15:04:02 * jlanda is around too, don't have too much free time but I'll be happy to help when I can =)
15:04:18 <knstn> Well, i want to clarify, that if my "skills" can't help in any way fedora-infra, i can withdraw from it
15:04:41 <nirik> everyone is welcome. There's stuff for everyone...
15:05:37 <mkonecny> There is always something to do :-)
15:05:42 <smooge> what skills do you have knstn and what are you interested in?
15:06:04 <knstn> I don't know programming. I study for sysadmin, and work remotely as support engineer for a SaaS company, troubleshooting embedded linux (raspbian).
15:06:20 <smooge> ah the iot side of the house
15:06:50 <knstn> I saw you have nagios,collectd, and haproxy. I could monitoring etc I have ccna, lpic-1, study for lpic-2 and then i'll switch to rhcsa/rhce
15:07:30 <smooge> cool
15:07:31 <nirik> yeah, improvement suggestions for any of those welcome for sure...
15:08:18 <smooge> one of the things a new person can do is just say 'I have some questions, can I schedule some time?' and work out a time with a sysadmin.. then go over the questions and improve the documentation on it
15:08:41 <smooge> it may not sound like a lot.. but it is actually incredibly useful
15:09:03 <smooge> because if you are wondering what htis does.. even some of the people here are probably doing hte same
15:09:16 * nirik nods. smooge +1
15:10:16 <knstn> i feel a bit lost, regarding a daily "workflow". what you look, where you update issues etc There are a ton of sites there for fedora.
15:10:45 <nils> .hello nphilipp
15:10:47 <zodbot> nils: nphilipp 'Nils Philippsen' <nphilipp@redhat.com>
15:11:21 <smooge> knstn, ah understood.
15:11:30 <nirik> yeah, there is. It can be confusing. :( For fedora infrastructure, https://pagure.io/fedora-infrastructure/issues is the main tracker of issues.
15:11:35 <nirik> (at least for now)
15:12:02 <smooge> that would be a good thing to write up as a document as it is opaque unless you have been in it for a while
15:12:34 <smooge> knstn, would you be around tomorrow to schedule some time to work on something?
15:12:56 <knstn> yeah, as i said i work from home, and i'm almost all day in irc
15:13:28 <smooge> ok . it is currently 15:00 UTC. I will be available after 18:00 UTC tomorrow
15:13:57 <knstn> that's fine, i'll be on that time
15:14:22 <smooge> ok cool. we can meet up in #fedora-admin and go over what questions and answers in an etherpad or similar document
15:14:41 <knstn> sure, no problem
15:14:46 <smooge> any other questions for today?
15:15:12 <knstn> nope, i'll just stay and watch the meeting
15:16:08 <smooge> #topic announcements and information
15:16:08 <smooge> #info bowlofeggs is on extended leave
15:16:08 <smooge> #info cverna will be going on extended leave
15:16:08 <smooge> #info abombard will be going on extended leave
15:16:08 <smooge> #info other vacations/leave/etc?
15:16:09 <smooge> #info Flock2Fedora 2019-08-08 -> 2019-08-11
15:16:10 <smooge> #info No site trip to PHX2
15:16:12 <smooge> #info Blog post on retiring apps https://communityblog.fedoraproject.org/application-service-categories-and-community-handoff/
15:16:26 <smooge> ok lots of people are on the summer breaks.
15:16:31 <nils> smooge, abompard_ _is_ on extended leave
15:16:47 <mkonecny> smooge: same for cverna
15:16:53 <nils> aye
15:17:02 <smooge> #info cverna is on extended leave
15:17:02 <smooge> #info abombard is on on extended leave
15:17:31 <smooge> updated the doc
15:17:52 <nirik> I did propose a infra/cpe workshop for flock...
15:18:00 <mboddu> nirik: +1
15:18:10 <nirik> everyone who's there and interested should show up to it. ;)
15:18:45 <mkonecny> nirik: It got accepted?
15:19:12 <nirik> not sure yet... there was talk about moving it to 1/2 day instead of full day so people could attend more other things...
15:19:22 <nirik> I hope it gets selected. I guess we will see.
15:19:42 <mkonecny> Mine was already accepted, this is why I'm asking
15:20:44 <nirik> they were doing several 'rounds' of them.
15:20:55 <nirik> I do also have a short talk on communishift...
15:21:00 <nirik> (that was accepted already)
15:21:15 <mkonecny> I have one about release-monitoring.org
15:21:26 <smooge> i have a talk about how not to attend FLOCK
15:21:27 <nirik> https://pagure.io/flock/issue/190 is the hackfest one if anyone wants to add comments or suggest things.
15:21:27 <drznpy> hi all, am I still on time for introductions ? :)
15:21:39 <mkonecny> smooge: Interesting :-)
15:21:42 <nirik> smooge: you can do it remotely! :)
15:22:11 <smooge> it probably should be "how not to regularly attend FLOCK or FUDCON"
15:22:35 <smooge> drznpy, we did that earlier but if you can wait until the openfloor we can do it then
15:22:46 <smooge> ok one thing I forgot
15:22:53 <drznpy> np, I’ll introduce myself during the next meeting :)
15:22:54 <smooge> next week is mass rebuild?
15:23:15 <smooge> or mass branching?
15:23:22 <nirik> mass rebuild
15:23:45 <nirik> 2019-07-24 	Mass Rebuild
15:23:50 <smooge> #info Fedora 31/rawhide mass rebuild. Builders will be busy. Packagers tempers will be short
15:23:55 <nirik> 2019-08-13 is branching.
15:24:01 <smooge> #undo
15:24:01 <zodbot> Removing item from minutes: INFO by smooge at 15:23:50 : Fedora 31/rawhide mass rebuild. Builders will be busy. Packagers tempers will be short
15:24:07 <nirik> right after flock. ;(
15:24:12 <smooge> #info 2019-07-24 Fedora 31/rawhide mass rebuild. Builders will be busy. Packagers tempers will be short
15:24:27 <smooge> #info 2019-08-13 is mass branching..
15:24:37 <smooge> #topic Oncall
15:24:37 <smooge> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
15:24:37 <smooge> #info relrod is on call from 2019-07-11 -> 2019-07-18
15:24:37 <smooge> #info ?????? is on call from 2019-07-18 -> 2019-07-25
15:24:37 <smooge> #info ?????? is on call from 2019-07-25 -> 2019-08-01
15:24:38 <smooge> #info smooge is on call from 2019-08-01 -> 2019-08-13 #flock
15:24:40 <smooge> #info ?????? is on call from 2019-08-13 -> 2019-08-15
15:24:42 <smooge> #info Summary of last week: (from relrod )
15:25:06 <nirik> I can take the next week if no one else wants it
15:25:11 <smooge> i need someone to take oncall from relrod and I need someone for next week. I will be doing it for the flock time but will be out afterwords
15:25:18 <nirik> or the week after
15:25:39 <smooge> nirik, week afterwords as next week you will be on rebuild issues and other stuff
15:25:43 <nirik> but yeah, after that I will be flockwards
15:26:14 <smooge> i can put up the nobody is on call.
15:26:40 <nirik> well, I can take this week...
15:26:53 <nirik> next week we see if we can find someone or just nobody
15:27:07 <relrod> I can probably do next week again
15:27:28 <smooge> can you do the 25 -> 01 relrod?
15:27:40 <relrod> smooge: not too much to report this week. A few pings but mostly me telling people "I don't have access to that system, please file a ticket and we'll get to it"
15:28:14 <smooge> yeah most of the others were things while you were off
15:28:37 <smooge> that being proxy11 being continually slow and proxy05 having a migraine
15:28:43 <nirik> so, me, relrod, smooge and figure out after that?
15:28:45 <smooge> then a bunch of copr related stuff
15:28:50 <smooge> got it
15:29:12 <jlanda> and the brno ipv6 thing with pagure's new location
15:29:15 <relrod> smooge: the copr stuff they pinged me about at 3am this morning, but didn't have a solidified question so I ended up going to sleep ;)
15:29:33 <smooge> relrod, other copr stuff
15:29:46 <smooge> like ipv6 problems in brno, and bad builders
15:29:48 <nirik> BTW, looking at flock and f31 schedules... I think our next mass update/reboot cycle window is the week of aug 19th.
15:29:50 * mboddu commented on the mass rebuild tickets to get a status update, FYI
15:30:33 <smooge> nirik, yeah.. at this point I am just doing update/reboots on systems when I run into them and they are already mostly down
15:30:59 <smooge> #topic Monitoring discussion
15:30:59 <smooge> #info https://nagios.fedoraproject.org/nagios
15:31:00 <smooge> #info Go over existing out items and fix
15:31:10 * mkonecny really liked smooge commit this week: "s390.koji was caught behind enemy lines. disavow all knowledge and purge all team members from records."
15:31:11 <smooge> nirik, the ceremonial coffee cup is yours
15:31:11 <nirik> sure. and we could do a 'no outage causing' update... but we have things every week until the week of the 19th that we don't want to mess with
15:31:14 <relrod> Oh, I should've put this in announcements probably. I'm on jury duty from aug 19-30, not sure which days they'll call me in.
15:32:01 <nirik> so, there's a alert about hotness messages...
15:32:15 <nirik> mkonecny: was going to ping you about that... is hotness no longer emitting messages?
15:32:28 <mkonecny> nirik: there is issue on bugzilla side
15:32:43 <nirik> odcs-backend01/02 disk space. we need to get the attention of someone on the factory2 team. Not sure who is the 'owner' there.
15:32:47 <smooge> nirik, and commands are .oncall_take_us
15:32:48 <nirik> ah.. ok.
15:32:48 <mkonecny> https://bugzilla.redhat.com/show_bug.cgi?id=1679385
15:32:54 <nirik> .oncall_take_us
15:32:54 <zodbot> nirik: Kneel before zod!
15:34:08 <nirik> buildvmhost-s390x-01 has a raid alert, I just need to probibly disable that, as the disk there is... different.
15:34:24 <smooge> the disk there is magical
15:34:49 * mkonecny prepares wizard hat
15:34:51 <smooge> it is a virtual disk inside of a virtual machine inside of a virtual container
15:35:34 <smooge> nirik, do you want me to take tracking dwon odcs people?
15:35:53 <smooge> or have I done enough 'helping' this week :P
15:36:09 <nirik> https://paste.fedoraproject.org/paste/SKDl04gTW0RAvJefYEJuoA
15:36:24 <nirik> please do if you like...
15:36:33 <nirik> we were also going to move that volume to a nfs one vs gluster.
15:36:41 <nirik> but haven't heard anything back from them.
15:36:41 <smooge> nirik, that looks like a scat song
15:37:17 <nirik> I was thinking odcs wasn't much used, but turns out I am pretty sure it's used in the flatpack building pipeline now.
15:37:31 <smooge> yeah
15:38:00 <smooge> yet another tool which we think we could turn off but find it has bailing wire connection to something else :)
15:38:08 <smooge> although that might be more duct-tape
15:38:35 <nirik> the new thing is ramen...
15:38:57 <smooge> sticky
15:39:03 <nirik> https://knowyourmeme.com/memes/fixing-things-with-ramen
15:39:28 <nirik> anyhow, thats all the alerts I think
15:39:41 <smooge> ok next up
15:39:47 <smooge> #topic Tickets discussion
15:39:47 <smooge> #info https://pagure.io/fedora-infrastructure/report/Meetings%20ticket
15:40:22 <smooge> taking that ticket out of htis
15:40:28 <smooge> any other tickets which need review?
15:40:53 <nirik> we continue to have a nice high flow of tickets in
15:41:40 <smooge> I think I am going to make monday ticket monday because tues->friday are assigned other things these days
15:41:55 <smooge> man I am not sure I can parse what I just wrote
15:41:59 <nirik> I've been using the later part of the day...
15:42:22 <nirik> so morning and midday I work on long term/big things... then I do a few tickets to unwind
15:42:28 <jlanda> ty for fixing the ssl cert thing on stg.pagure.org nirik++
15:42:29 <zodbot> jlanda: Karma for kevin changed to 23 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:42:39 <nirik> jlanda: no problem...
15:42:41 <mkonecny> nirik: But it's better than pinging
15:42:53 <smooge> mkonecny++
15:42:53 <zodbot> smooge: Karma for zlopez changed to 3 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:42:59 <nirik> mkonecny: oh yeah. Much better.
15:43:00 <smooge> ah come-on
15:43:26 <jlanda> now i have another different problem there, but I'll need to take a deeper look on it, stg pagure is trying to load pagure.io on the iframe where I had that ssl error
15:43:30 <nirik> the big thing I still would love to have is a view on what exactly everyone is working on right at the time I look...
15:43:33 <jlanda> I'll open a ticket when I have some more info :D
15:43:44 <smooge> jlanda, please do.. we need to finish this meeting up before fpc
15:43:54 <nirik> ie, smooge is heads down in epel8 work and I should leave him alone, etc.
15:44:06 <mkonecny> nirik: Yes, some overview could be nice
15:44:09 <smooge> yeah.. do you want me to put in tickets for that
15:44:31 <nirik> lets have a meeting to leverage the synergies of ticket flows
15:44:51 * smooge screams
15:45:01 <smooge> he has become one of them..
15:45:04 <tflink> O.o
15:45:40 <mkonecny> nirik: What about meeting about the meeting to leverage the synergies of ticket flows
15:46:14 * smooge is crawling in the closet and closing the door
15:46:19 <nirik> I'm gonna need you all to come in on... saturday... </officespace>
15:46:53 <smooge> ok back to the meeting
15:46:58 <smooge> #topic Open Floor
15:47:22 <smooge> drznpy, if you are still available.. welcome
15:49:28 <smooge> ok well next time
15:50:04 <smooge> alright I am going to kick you all out now so fpc can start.
15:50:10 <smooge> thank you all for coming and see you next week
15:50:12 <smooge> #endmeeting