15:00:23 <cverna> #startmeeting Infrastructure (2019-10-03)
15:00:23 <zodbot> Meeting started Thu Oct  3 15:00:23 2019 UTC.
15:00:23 <zodbot> This meeting is logged and archived in a public location.
15:00:23 <zodbot> The chair is cverna. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:23 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:23 <zodbot> The meeting name has been set to 'infrastructure_(2019-10-03)'
15:00:38 <cverna> #meetingname infrastructure
15:00:38 <zodbot> The meeting name has been set to 'infrastructure'
15:00:38 <cverna> #topic aloha
15:00:38 <cverna> #chair nirik pingou relrod smooge tflink cverna mizdebsk mkonecny abompard bowlofeggs
15:00:38 <zodbot> Current chairs: abompard bowlofeggs cverna mizdebsk mkonecny nirik pingou relrod smooge tflink
15:01:00 <nirik> morning
15:01:01 <tflink> morning
15:01:07 <mkonecny> .hello zlopez
15:01:08 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@packetseekers.eu>
15:01:18 <cverna> hello o/
15:01:43 <scoady> hello
15:02:01 <joalon> Hi
15:02:03 <austinpowered> .hello
15:02:03 <zodbot> austinpowered: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1".
15:02:11 * marcdeop is present
15:02:42 <cverna> #topic Next chair
15:02:43 <cverna> #info magic eight ball says:
15:02:52 <cverna> anyone willing to chair next meeting ?
15:02:54 <austinpowered> .hello austinpowered
15:02:55 <zodbot> austinpowered: austinpowered 'None' <fedoraproject@wootenwilliams.com>
15:04:34 <cverna> any volunteer to run next meeting ? this is fun :P
15:04:41 <bowlofeggs> .hello2
15:04:42 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <rbarlow@redhat.com>
15:04:48 <mkonecny> cverna: I could try it
15:04:49 <relrod> .hello2
15:04:51 <zodbot> relrod: Sorry, but you don't exist
15:04:54 <relrod> .hello codeblock
15:04:55 <zodbot> relrod: codeblock 'Rick Elrod' <codeblock@elrod.me>
15:04:59 <cverna> mkonecny++
15:05:07 <nirik> mkonecny: aweseome. thanks!
15:05:10 <mkonecny> Do I need any special permissions?
15:05:53 <cverna> no just need to follow the template in the agenda at https://board.net/p/fedora-infra
15:06:04 <cverna> and do a lot of copy pasting :P
15:06:16 <mkonecny> Ok, thanks :-)
15:06:22 <nirik> we could clone the fesco one on the wiki...
15:06:32 <nirik> it automafically fills in stuff for the current meeting
15:06:41 <cverna> #info mkonecny to run next meeting (2019-10-10)
15:07:08 <cverna> ah that might be cool
15:07:32 <cverna> ok moving to next topic
15:07:33 <nirik> https://fedoraproject.org/wiki/FESCo_meeting_process
15:07:41 <cverna> #topic New folks introductions
15:07:41 <cverna> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:07:41 <cverna> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
15:07:44 <nirik> if someone wants to try and clone it.
15:07:59 <dustymabe> .hello2
15:08:00 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
15:08:17 <joalon> Hi! Sorry, I lurked a bit last meeting but didn't introduce myself. Nice to meet you all!
15:08:34 <cverna> joalon: welcome and nice to meet you :-)
15:08:46 <nirik> welcome joalon!
15:08:56 <joalon> Thanks :)
15:09:17 <scoady> Hi everyone, this is my first meeting. Looking forward to getting started on Fedora! This is my first IRC meeting ever too
15:09:28 <cverna> joalon: are you already in the apprentice group ? if not is this something you would like to be part of ?
15:09:43 <cverna> scoady: whoohoo welcome :-)
15:10:15 <joalon> @cverna I'm not a part yet, would absolutely be interested
15:10:17 <nirik> hey scoady. Welcome to the joy and fun of irc meetings. ;)
15:10:55 <mkonecny> scoady: Nice to see you in bodhi channel :-)
15:11:12 <cverna> joalon: cool hang on in #fedora-admin after the meeting and someone can add you to the group then
15:11:38 <joalon> cverna Thanks, will do
15:11:52 <cverna> ok moving to next topic
15:12:11 <cverna> #topic announcements and information
15:12:11 <cverna> #info We are looking for people to maintain Fedocal - mkonecny
15:12:11 <cverna> #info Anitya 0.17.2 was deployed on production (release-monitoring.org) - mkonecny
15:12:11 <cverna> #info f31 final freeze starts next tuesday (2019-10-08)
15:12:11 <cverna> #info kevin/nirik on PTO from 2019-10-23 to 2019-10-27
15:12:12 <cverna> #info robosignatory running with fedora-messaging deployed in stg.
15:12:27 <cverna> anything else to add ?
15:13:14 <cverna> ok let's move on then
15:13:15 <relrod> #info mostly have fasClient working on f31, just need to test some more, but so far it is looking good
15:13:26 <nirik> oh nice.
15:13:32 <cverna> nice
15:13:55 <nirik> BTW, thats the last thing using python2 on our builders currently. :)
15:14:34 <cverna> relrod could you assign the ticket to yourself so we know someone is working on that --> https://pagure.io/fedora-infrastructure/issue/8210
15:14:42 <relrod> sure
15:14:58 <nirik> once it's working how are we maintaing it?
15:15:00 <cverna> :-)
15:15:22 <nirik> I mean we need a rpm in our infra repos, but should we just leave it at that? or put it in git somewhere? upstream branch?
15:15:32 <relrod> nirik: yeah that I'm not sure about...if we want to update the package or just sync it out with ansible and hope we move to something else before that bites us
15:16:26 <nirik> yeah, we could patch it in ansible... seems messy, but... ok
15:16:34 <cverna> does not cost much to create a git repo for it in pagure or github ? or it is not worth it ?
15:16:48 <relrod> nirik: Doesn't matter much to me. Right now it's just in /root on f31-test ;)
15:16:57 <nirik> well, just another thing to remember whats there and where it is...
15:17:13 * cverna hope relrod has a local copy :P
15:18:15 <relrod> I do
15:18:24 <relrod> But yeah open to thoughts there. I was just concerned about getting the code to work
15:18:30 <relrod> I *don't* know if the changes are backward compat
15:18:45 <relrod> so we might need a distinction between "this version is for f31+" and "this version is for...everything not that"
15:19:15 <relrod> I *think* they maybe should be, but I haven't tested
15:20:05 <cverna> moving on and we can come back to that in the open floor if time allows
15:20:07 <relrod> that's all from me on it though - if someone wants to help with the packaging part or whatnot, reach out ;)
15:20:35 <cverna> #topic Oncall
15:20:35 <cverna> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
15:20:35 <cverna> #info cverna is on call 2019-10-03 -> 2019-10-10
15:20:35 <cverna> #info pingou is on call 2019-10-10->2019-10-17
15:20:35 <cverna> #info smooge is on call 2019-10-17->2019-10-24
15:20:36 <cverna> #info Summary of last week: (from cverna )
15:21:33 <cverna> relatively quiet week on my side, some issues with FAS, and today some issues with kojira that I was not really able to help fixing ;-)
15:21:42 <cverna> other than that nothing major
15:22:41 <cverna> pingou is away until next week so I can stay oncall until monday
15:22:44 <relrod> yeah the FAS issue is annoying. Nothing in proxy logs about it that I could find, but I think it's on the app side, but haven't dug beyond that
15:23:16 <nirik> yeah, I have tried a bunch of random things to no avail on it.
15:23:22 <nirik> it's definitely on the app side.
15:23:49 <nirik> I tried a new rollout/pods, restarted db, vacuumed db
15:24:11 <smooge> if only we had a project to replace fas :)
15:24:22 <cverna> :-)
15:24:33 <smooge> which I expect it heard and has decided to rush things
15:24:43 <relrod> smooge: ssshhhh, don't tell people about my plan to subtly break FAS in odd ways to get us to move to something else quicker!
15:25:02 <nirik> I think the next step is to try and get more logs out of it...
15:25:16 <smooge> relrod, I have worked with software long enough to know that when you decide to make a change, ti will just do it for you
15:25:59 <smooge> relrod, since you got the client working with python3.. you want to look at moving the rest there too :)
15:26:07 <relrod> ha. hahahah. no.
15:26:18 <smooge> and with that.. I stop sidelining cverna's meeting
15:26:20 <cverna> ok being aware of time here, I ll move on to the next topic :-)
15:26:29 <cverna> haha thanks smooge
15:26:34 <cverna> #topic Monitoring discussion
15:26:34 <cverna> #info https://nagios.fedoraproject.org/nagios
15:26:34 <cverna> #info Go over existing out items and fix
15:27:07 <cverna> osbs is still on me, still did not have time to look at it (maybe one day)
15:27:24 <nirik> all the rest is known I think.
15:27:37 <nirik> we have a drive replaced in qa09, just need to reboot it sometime
15:27:48 <nirik> need fedmsg/fedora-messaging for fas
15:28:35 <relrod> yeah I keep pushing that lower in my list because I know my openshift skills and how long it's going to take.
15:29:17 <cverna> if it is not really needed tho maybe it is fine to leave it broken
15:29:20 <nirik> well, I think we can look back when we did this for other apps and just copypasta it.
15:29:34 <cverna> what are the FAS messages used for ?
15:29:34 <nirik> well, it causes the job that updates ssh keys to not work
15:29:46 <cverna> ah ok
15:29:47 <nirik> so people are always bugging us about their keys not updated and we have to run it manually
15:30:06 <mizdebsk> we can run the job periodically
15:30:06 <cverna> could we make it a cron job ?
15:30:12 <nirik> we do run it daily
15:30:17 <mizdebsk> fas client is ran daily on all systems
15:30:20 <nirik> but that is way not fast enough for people
15:30:33 <mizdebsk> but there is a playbook to run it on bastion, people02 and pkgs02
15:30:34 <nirik> and faster causes stress on fas...
15:30:36 <mizdebsk> that can be ran more often
15:30:42 <nirik> if fas is working even
15:31:34 <nirik> I guess running it faster on the big ones would help... but we should get fas not throwing 500's first
15:31:59 <cverna> there is no way to win in this game :)
15:32:00 <relrod> I can try my hand at debugging that some this afternoon
15:32:32 <nirik> I'm not sure if fas in stg is hitting the same thing, we should check
15:32:55 <cverna> relrod: let's try to do a thing at the time (focus on finishing fasclient ) :)
15:32:59 <mizdebsk> i think that broken fas fedmsg affects badges too
15:33:26 <relrod> cverna: well this makes working on fasclient annoying too because to test it I keep having to rerun it a bunch of times until it randomly works
15:33:33 <nirik> it is not it seems.
15:33:37 <relrod> (due to fas 500ing on it)
15:34:01 <nirik> it also most importantly breaks signing/rawhide gating.
15:34:03 <cverna> ok that's a valid point :)
15:34:23 <relrod> I'll see if I can find a trace in the logs somewhere
15:34:45 <nirik> robosignatory doesn't retry... so when it asks sigul to sign something and sigul looks up it's user to make sure it can do so, and gets a 500, it just errors and sits in the tag
15:34:59 <smooge> well a trace which isn't the multiple ones which fas dumps regularly
15:35:01 <nirik> relrod: I can perhaps tag team with you after the meeting on it?
15:35:07 <relrod> sure
15:35:19 <nirik> or we can take over admin and anyone can join in the fun.
15:36:03 * nirik fires off a new build just in case and also to get build logs so I can see what it's doing better
15:36:10 <nirik> shall we move on?
15:36:13 <cverna> ok moving to next topic
15:36:18 <cverna> yes
15:36:20 <cverna> #topic Tickets discussion
15:36:20 <cverna> #info https://pagure.io/fedora-infrastructure/report/Meetings%20ticket
15:36:27 <cverna> no tickets tagged
15:36:51 <nirik> we might want to replace this with backlog, or just skip if no tickets...
15:37:02 <cverna> yes backlog is next
15:37:11 <cverna> so let's go to backlog
15:37:41 <cverna> #topic backlog discussion
15:37:41 <cverna> #info go over our backlog and discuss and determine priority
15:37:41 <cverna> #link https://pagure.io/fedora-infrastructure/issue/8178
15:37:41 <cverna> #link https://pagure.io/fedora-infrastructure/issue/8157
15:37:41 <cverna> #link https://pagure.io/fedora-infrastructure/issue/8065
15:37:57 <cverna> I linked the 3 tickets we have selected last week
15:37:59 <nirik> so, how did we do on the things we pointed out last week?
15:38:12 <cverna> I don't think we have made much progress :(
15:38:31 <nirik> so, on 8178 (aarch64). I managed to sort out one of them... but then we needed more switch ports/interfaces connected.
15:38:45 <cverna> I am looking for someone to work with me on https://pagure.io/fedora-infrastructure/issue/8157
15:38:56 <cverna> raise your hand if interested
15:39:25 <nirik> so, waiting on that getting figured out. I was hoping on working on it some more later today/tomorrow... need to pick a time to work on it.
15:39:36 <mkonecny> I thought this one will be part of the PR workflow
15:39:50 <cverna> mkonecny: 8157 ?
15:40:19 <mkonecny> Yes, run ansible-lint on every PR against ansible repo
15:40:31 <nirik> On 8065, smooge and mboddu_ and I met up the other day and went over everything in irc. (fedora-admin). Then, mboddu_ was going to write up a infra-docs/sop for it... also he archived the f28 stuff to make sure he understood everything.
15:40:47 <cverna> nice ^^
15:40:59 <nirik> so, we can't really have PR's until we have repospanner... but yeah, we could run stuff then...
15:41:19 <nirik> also, we looked at ansible-lint and I thought decided ansible-report was much better for our needs?
15:41:25 <smooge> yes
15:41:33 <nirik> but perhaps I am misremembering. (ansible report calls ansible-lint I think)
15:41:35 <cverna> nirik: ansible-report seems dead
15:41:44 <cverna> it was archived in github
15:41:47 <nirik> oh thats too bad.
15:42:04 <cverna> maybe I did not looked at the good one tho
15:42:34 <cverna> https://github.com/manuBocquet/ansible-report
15:42:37 <nirik> now that bowlofeggs fixed some speed issues we could look at ansible/repospanner again (or test it I suppose)
15:42:41 <cverna> hum seems alive
15:42:46 <smooge> sorry to derail but why do we have to have repospanner? Why can't we just have it in pagure.io and have the batcave be a pull copy of that?
15:43:01 <smooge> I expect it is something obvious I am forgetting
15:43:30 <mkonecny> smooge: +1
15:43:32 <bowlofeggs> nirik: i only made it 2x as fast, which is good but imo still orders of magnitude away from what it should be in an ideal world
15:43:38 <nirik> oooh... ansible-review?
15:44:03 <nirik> smooge: well, what if pagure is down and you need to push something to ansible to fix it/
15:44:04 <nirik> ?
15:44:21 <bowlofeggs> nirik: i'm currently testing out an idea that patrick had to make it yet faster than this, but it's a more invasive change
15:44:21 <cverna> ha ansible-review :-)
15:44:22 <nirik> or pagure is down and you need to fix something really important...
15:44:35 <smooge> nirik, if repospanner is down we are going to be in the same boat
15:45:10 <bowlofeggs> smooge: if a majority of repospanner nodes are up and reachable, you can push to the cluster still
15:45:23 <bowlofeggs> smooge: but, i'm also personally not sure if that's the standard we need
15:45:28 <nirik> sure, there's still some risk...
15:45:49 <bowlofeggs> repospanner is def more HA than pagure
15:45:57 <nirik> I guess we could just say bag it... but we need a way to pull to batcave or every commit...
15:46:04 <cverna> I think that does not stop us from adding ansible-review as a pre-commit hook
15:46:29 <bowlofeggs> nirik, smooge: what if we just put a clone of our repo on gitlab.com?
15:46:38 <cverna> since even with PRs we will not enforce that all changes should go through a PR ? or do we ?
15:46:42 <bowlofeggs> nirik, smooge: and used a commit hook to keep that one up to date?
15:46:45 <nirik> bowlofeggs: -1 :)
15:47:05 <nirik> cverna: no, I wouldn't think we would...
15:47:06 <bowlofeggs> nirik: what issue do you see?
15:47:19 <nirik> bowlofeggs: yet ANOTHER place for people to look for our stuff.
15:47:33 <bowlofeggs> nirik: well this would just be as a backup
15:47:37 <nirik> lets put it in a basement in a filing cabnet labeled beward of leopard. ;)
15:47:45 <bowlofeggs> i.e., for the "pagure is down and our infra needs the repo!"
15:48:01 <smooge> one item we have with putting it on gitlab.com is that we can't use personal accounts. We will need to set up a corporate contract account because we are doing this for Fedora as work from Red Hat.
15:48:06 <nirik> ah, well... if we have a up to date copy on batcave do we need that?
15:48:07 <cverna> there is a clone of repo in github already
15:48:12 <bowlofeggs> smooge: fair
15:48:20 <smooge> the same with github
15:48:20 <cverna> I can't remember which team maintain that
15:48:24 <nirik> cverna: yeah, it already causes confusion tho I think.
15:48:26 <bowlofeggs> nirik: well what if batcave went down too?
15:48:52 <mizdebsk> bowlofeggs, we have batcave13 in a different datacenter
15:48:53 <bowlofeggs> well anyways, we could do the batcave thing sure
15:48:55 <nirik> bowlofeggs: thats the same problem we have today. ;) but I have a local copy on my laptop... ;) we have backups on our netapp backups drive in phx2 and in rdu2
15:49:00 <bowlofeggs> it's better than not doing it ☺
15:49:08 <bowlofeggs> nirik: yeah
15:49:28 <bowlofeggs> nirik: and i have a local copy in my local repospanner cluster that only takes forever to pull ☺
15:49:47 <nirik> proposal: someone come up with a plan on list and we figure out what sounds best. ;)
15:49:55 <bowlofeggs> +1
15:49:56 <smooge> yeah sorry.
15:50:04 <cverna> +1
15:50:13 <cverna> don't who someone is tho :P
15:50:18 <nirik> I'd be ok with pagure having the repo as long as we can make sure batcave updates after every commit (since thats where ansible runs from)
15:50:33 <bowlofeggs> i'm not opposed to using repospanner btw, i just want to be honest that its speed is still going to be unpleasant
15:50:39 <nirik> pagure has some mirror thing, but I don't know how it works
15:51:01 <cverna> ok we have 5 min left
15:51:08 <smooge> we are going to need that somehow with repospanner also. Just because we have pushed it into repospanner it does not mean that the batcave version would reflect that
15:51:39 <cverna> #topic Fedora CoreOS related tickets
15:51:39 <cverna> #info for 2019-10-03 meeting
15:51:39 <cverna> #info https://github.com/coreos/fedora-coreos-tracker/blob/master/Fedora-Requests.md#existing-requests-for-fedora-infra
15:51:45 <nirik> I guess I can look into it... since I feel strongly about it.
15:51:56 <cverna> dustymabe: did you have anything to raise ?
15:52:13 <mkonecny> I'm getting 404 on that link
15:52:27 <cverna> yes I think we need to update it
15:53:16 <cverna> ok a quick Open Floor
15:53:18 <cverna> #topic Open Floor
15:53:31 <cverna> the floor is Open
15:53:31 <bowlofeggs> nirik: i'd be happy to chat with you about it some more if you want
15:53:46 <nirik> bowlofeggs: sure... after the fas debugging?
15:53:55 <mkonecny> cverna: Let's dance
15:53:58 <bowlofeggs> sure, i'm available ost of the day
15:54:16 * cverna takes mkonecny on the dancefloor
15:54:22 <nirik> quick:
15:54:36 <nirik> everyone look at the nearest window and describe what you see? :)
15:54:51 <smooge> a closed shade
15:54:55 <cverna> grass that I should have cut 2 weeks ago :)
15:55:03 <bowlofeggs> closed shade here too
15:55:09 <bowlofeggs> and outside is my dead grass
15:55:13 <mkonecny> Apple trees :-)
15:55:15 <joalon> large sugar cane field :)
15:55:16 <bowlofeggs> also, it's hot as junk here
15:55:22 <bowlofeggs> even though it's october
15:55:29 <smooge> 37 Celsius today
15:55:31 <bowlofeggs> this is not normal at all
15:55:47 <bowlofeggs> yep
15:55:50 <bowlofeggs> ooof
15:55:51 <cverna> oh that's hot
15:55:57 <smooge> Phoenix Arizona is oging to be cooler than here
15:55:59 <nirik> here there's thick fog and shadowy trees.
15:56:08 <bowlofeggs> (high of 36°C/97°F predicted here today)
15:56:22 <bowlofeggs> i live kinda close to smooge
15:56:24 <smooge> bowlofeggs, they moved it up 2 degrees
15:56:27 <nirik> and now sunlight shining thru. fun. ;)
15:56:27 <bowlofeggs> but prob an hour or so
15:56:39 <bowlofeggs> smooge: well we might get slightly different forecasts
15:57:03 <dustymabe> cverna: sorry stepped away
15:57:07 <smooge> ok time for me to go look at my task list and see what I can put off another day
15:57:14 <smooge> meeting time is over
15:57:23 <cverna> dustymabe: you missed your chance :P
15:57:31 <dustymabe> oh boo :(
15:57:40 <bowlofeggs> last year i switched my heat on on oct 23. hard to believe that given that that's only 20 days from now and it's so hot outside
15:57:47 <smooge> thanks everyone. thank you cverna.
15:57:59 <mkonecny> thanks cverna
15:58:05 <cverna> thanks all for joining
15:58:17 <cverna> #endmeeting