15:03:56 <dustymabe> #startmeeting  Infrastructure Meeting
15:03:56 <zodbot> Meeting started Thu Oct 31 15:03:56 2019 UTC.
15:03:56 <zodbot> This meeting is logged and archived in a public location.
15:03:56 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:03:56 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:03:56 <zodbot> The meeting name has been set to 'infrastructure_meeting'
15:04:51 <nirik> morning everyone
15:04:55 <pingou> morning flks
15:04:57 <bowlofeggs> sup
15:04:57 <pingou> +o
15:05:05 <dustymabe> morning
15:05:19 <austinpowered> good morning
15:05:22 <scoady> hey everyone
15:05:26 <dustymabe> #info agenda is at https://board.net/p/fedora-infra
15:05:49 <relrod> here
15:05:50 <dustymabe> sorry I didn't realize I needed to send out the agenda yesterday
15:06:06 <dustymabe> #meetingname infrastructure
15:06:06 <zodbot> The meeting name has been set to 'infrastructure'
15:06:12 <dustymabe> #topic aloha
15:06:19 <mkonecny> .hello zlopez
15:06:20 <dustymabe> #chair nirik pingou relrod smooge tflink cverna mizdebsk mkonecny abompard bowlofeggs
15:06:20 <zodbot> Current chairs: abompard bowlofeggs cverna dustymabe mizdebsk mkonecny nirik pingou relrod smooge tflink
15:06:20 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@packetseekers.eu>
15:06:42 <dustymabe> .hello dustymabe
15:06:43 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
15:06:58 <dustymabe> #topic Next chair
15:06:59 <nirik> no worries, thanks for running things today dustymabe!
15:07:06 <dustymabe> #info magic eight ball says:
15:07:16 <dustymabe> #undo
15:07:16 <zodbot> Removing item from minutes: INFO by dustymabe at 15:07:06 : magic eight ball says:
15:07:19 <dustymabe> #info magic eight ball says:     2019-11-07: mkonecny
15:07:24 <mizdebsk> hello
15:07:32 <dustymabe> do we want to add anyone else to run a future meeting?
15:07:55 <pingou> I can take the 21st, but I'd rather leave it to someone else on the 14th
15:09:06 <smooge> I can do the 14th
15:09:09 <nirik> I guess I could do the 14th... but happy for someone else to.
15:09:12 <dustymabe> mizdebsk: bowlofeggs austinpowered ?
15:09:40 <dustymabe> smooge++
15:09:45 <dustymabe> #topic New folks introductions
15:09:52 <dustymabe> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:09:54 <dustymabe> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
15:11:29 <dustymabe> i'm guessing no new folds
15:11:31 <dustymabe> folks*
15:11:41 <dustymabe> will move on in 30 seconds
15:12:08 <dustymabe> #topic announcements and information
15:12:15 <dustymabe> #info f31 final freeze now over
15:12:17 <dustymabe> #info aarch64 hardware having issues
15:12:19 <dustymabe> #info koji backups are done on secondary host. Slowdown in koji still occurring.
15:12:29 <dustymabe> yay for freeze being over
15:12:46 <nirik> actually I don't think koji is having slowdowns anymore...
15:12:53 <nirik> and I think I fixed the backups yesterday. :)
15:12:57 <dustymabe> #undo
15:12:57 <zodbot> Removing item from minutes: INFO by dustymabe at 15:12:19 : koji backups are done on secondary host. Slowdown in koji still occurring.
15:12:58 <pingou> #info ci-resultsdb-listener schedule early next week
15:13:29 <nirik> but I guess I will need to see how tonights backup runs.
15:14:35 <dustymabe> anything else before next topic?
15:15:18 <dustymabe> #topic Oncall
15:15:25 <dustymabe> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
15:15:27 <relrod> oh hi there
15:15:27 <dustymabe> #info smooge is on call 2019-10-17->2019-10-24
15:15:28 * cverna waives
15:15:29 <dustymabe> #info relrod is on call 2019-10-24->2019-10-31
15:15:31 <dustymabe> #info nirik is on call 2019-10-31->2019-11-07
15:16:01 <pingou> I'll take it from the 14th on
15:16:02 <nirik> zodbot: takoncallus
15:16:09 <nirik> zodbot: takeoncallus
15:16:14 <pingou> which leaves    one week in b/w
15:16:35 <smooge> nirik, I think there is a _us
15:16:39 <relrod> So, all things considered, it was a slow week even with the release. I don't think I got any .oncall pings. Just lots of websites questions, but that's separate. ;)
15:16:48 <nirik> zodbot: oncalltakeus
15:16:48 <zodbot> nirik: Kneel before zod!
15:17:09 <nirik> yeah, I fear I have been backsliding a bit... and need to try harder.
15:17:24 <nirik> I have been answering people that ping me directly. Need to stop that...
15:17:59 <dustymabe> pingou: I updated the board to have you for 14th+
15:18:20 <pingou> thanks
15:18:27 <pingou> who takes next week then?
15:18:53 <relrod> nirik: Wonder if it's worth adding another alias to zodbot, .notoncall or something that says "Hi, you've directly pinged someone who isn't the on-call. Please direct your question through the on-call person (see .oncall) or file an Infra ticket."
15:19:02 <relrod> and when they ping you just type that alias
15:19:14 <nirik> I have an away message like that. should use it more.
15:19:43 <dustymabe> any volunteers for next week on call?
15:20:06 <smooge> nope
15:20:10 <cverna> I can take it
15:20:28 <cverna> sorry I was distracted
15:20:43 <dustymabe> perfect
15:20:45 <dustymabe> thanks
15:20:50 <dustymabe> #info Summary of last week: (from smooge )
15:21:01 <dustymabe> #undo
15:21:01 <zodbot> Removing item from minutes: INFO by dustymabe at 15:20:50 : Summary of last week: (from smooge )
15:21:15 <dustymabe> I think relrod already gave us an update
15:21:32 * marcdeop is super late :'(
15:21:55 <dustymabe> #topic Monitoring discussion
15:22:02 <dustymabe> #info https://nagios.fedoraproject.org/nagios
15:22:03 <relrod> marcdeop: For your punishment you must fix all the broken things ;)
15:22:04 <dustymabe> #info Go over existing out items and fix
15:22:20 * nirik looks
15:22:29 <nirik> the 3 down servers are all known
15:22:36 <pingou> relrod++
15:23:03 <nirik> the datanommer ones... bugzilla is known broken, we have an internal ticket to fix it, but they don't seem to be making much progress.
15:23:10 <nirik> the fas one needs us to setup a fas listener
15:23:18 <nirik> sorry, emitter
15:23:39 <nirik> and the qa09 one is because that machine is hosed up and should be re-installed. ;)
15:24:01 * marcdeop will buy a beer to any infra teem member if he/she drops by Berlin
15:24:38 <nirik> marcdeop: carefull, we may have a flock near there sometime. ;)
15:24:40 <dustymabe> nirik: does that address all open issues ?
15:25:17 <pingou> marcdeop: I'll let you know when I passssSS BY :)
15:25:21 * pingou blames keyboard
15:26:03 <nirik> dustymabe: yep. move along, unless anyone wants to discuss those more
15:26:41 <dustymabe> #topic Tickets discussion
15:26:56 <dustymabe> #info https://pagure.io/fedora-infrastructure/report/Meetings%20ticket
15:26:58 <dustymabe> #info no tickets marked for the meeting
15:27:00 <dustymabe> anything to bring up?
15:27:16 <nirik> well, we should talk about the backlog...
15:27:24 <nirik> unless we have a seperate place for that (I can't recall)
15:27:24 <dustymabe> yeah, that's next
15:27:28 <nirik> ok, cool.
15:27:34 <dustymabe> well next topic
15:27:39 <dustymabe> there's alos this in the current topic
15:27:48 <dustymabe> #info Fedora CoreOS Team requests
15:27:54 <cverna> :P
15:28:20 <dustymabe> for this one I'd like to discuss https://pagure.io/fedora-infrastructure/issue/8218
15:28:38 <dustymabe> any public docs for adding users to communishift ?
15:28:38 <smooge> likes how I keep getting confused on whether i am looking at releng tickets or fedora-infrsatructure tickets because they seem to have the same number of open ones all the time
15:29:19 <nirik> I know bowlofeggs was looking at making docs, but I don't know where that is.
15:29:24 <cverna> I think bcotton started something when working on the fedocal migration
15:29:36 <dustymabe> maybe we can get him to make an update in the ticket
15:29:45 <cverna> ie he took some notes that could be later be used to create docs
15:30:14 <mizdebsk> dustymabe, https://fedora-infra-docs.readthedocs.io/en/latest/sysadmin-guide/sops/communishift.html#onboarding-new-users
15:30:42 <bowlofeggs> nirik: i've put that project aside - it got bumped down in priority by many other things :(
15:30:51 <nirik> alright, fair enough.
15:31:16 <dustymabe> mizdebsk: i'd like to be able to add a group of people to a named project in communishift - do we know how add multiple users to a project
15:32:15 * cverna has no clue
15:32:41 * nirik doesnt know off hand either.
15:32:56 <cverna> but we have a ansible role that does that, we could just look at which oc command it refers too
15:33:04 <cverna> ie the appowners in ansible
15:33:21 <dustymabe> cverna: we have an appowners file for communishift?
15:33:29 <nirik> no
15:33:41 <cverna> can't we have one ?
15:34:03 <nirik> I'm not sure how it would work.
15:34:15 <nirik> since we don't contol new apps.
15:34:45 <dustymabe> yeah if we've been encouraging people to run apps there (i.e. infra doesn't own the apps) but they can't add more than one person to a project, then that's not good if that person gets hit by a bus or goes on vacation
15:34:47 <nirik> it may be that those projects that want to add people create a permission for that?
15:34:56 <cverna> Ideally it would be nice if the person that created the project could run a few oc command to add people to the project
15:35:04 <dustymabe> cverna++
15:35:10 * nirik completely agrees.
15:35:38 <dustymabe> so how do we capture that?
15:35:38 <nirik> we likely need to investigate/ask other openshift folks how to do this...
15:36:41 <dustymabe> ok I'll try to update the ticket
15:36:43 <nirik> someone has to investigate. I can try, but it's not going to be high on my list
15:37:29 <cverna> maybe we can flag that as a possible apprentice task
15:37:34 <cverna> and ask for help on the list
15:37:42 <pingou> +1 that would be nice
15:37:50 <dustymabe> nirik: can I bring up a releng ticket (that has infra implications)?
15:37:58 <dustymabe> or should i wait on that?
15:38:13 <nirik> sure...
15:38:25 <nirik> I'm not sure this is apprentice level, but sure. :)
15:38:27 <dustymabe> releng ticket: https://pagure.io/releng/issue/8811#comment-608625
15:38:39 <dustymabe> cverna: thanks!
15:39:18 * nirik nods at the last update there.
15:39:55 <nirik> I think we could sync things to the new volume then swap them with minimal downtime
15:40:10 <dustymabe> nirik: yep, i'm worried less about bullet #2
15:40:18 <dustymabe> bullet #1 is the open question
15:40:30 <dustymabe> so maybe we can look at that sometime in the next few weeks ?
15:40:40 <nirik> sure.
15:41:02 <dustymabe> will coordinate with you offline (or someone else if it's more appropriate)
15:41:11 <dustymabe> #topic backlog discussion
15:41:17 <dustymabe> #info go over our backlog and discuss and determine priority
15:41:53 <dustymabe> is that this? https://pagure.io/fedora-infrastructure/issues?status=Open&tags=backlog&close_status= ?
15:41:56 <nirik> so the 2 there we have had for a while...
15:42:13 <nirik> dustymabe: yes, but the two listed there are the ones we are trying to finish
15:42:17 <dustymabe> nirik: ok
15:42:21 <dustymabe> i'll info them
15:42:29 <dustymabe> #link https://pagure.io/fedora-infrastructure/issue/8178
15:42:31 <dustymabe> #info topic: provision new aarch64 builders
15:42:33 <dustymabe> #link https://pagure.io/fedora-infrastructure/issue/8157
15:42:35 <dustymabe> #info topic: ansible: enable ansible-report as a hook
15:42:54 <nirik> ok, on aarch64 builders we are making some progress now... after hitting lots of issues. ;)
15:43:12 <nirik> Hopefully that will be done in the next week now.
15:43:32 <nirik> On the ansible-report hook, we need to decide what we want to do there...
15:43:53 <cverna> I think the nightly report would be a good first step
15:44:10 <nirik> how bad is it currently? I thought we fixed all the errors?
15:44:14 <austinpowered> has anyone ever run ansible-report against the current repo?
15:44:33 <cverna> we don't have errors, just a lot of warnings
15:44:58 <nirik> austinpowered: yep. ;)
15:45:02 <cverna> mostly about bad practices, like using the command module
15:46:08 <nirik> ok. perhaps an initial version of that to the list and people can fix things?
15:47:01 <nirik> I was pondering if we shouldn't just move to pagure as a normal repo, modify ansible-playbook/rbac-playbook to just pull before running and move to repospanner/gitlab/the moon later?
15:47:44 <pingou> we could add a cron job to pull from pagure into the batcave on a regular basis
15:48:03 <cverna> could pagure automatically sync the batcave on each commit ?
15:48:11 <pingou> so if pagure goes down we have an ansible repo that is only missing some commits but not all
15:48:13 <cverna> like a post-receive hook
15:48:15 <pingou> it can
15:48:49 <nirik> I don't like cron.
15:48:59 <nirik> someone commits something they want to run/apply it...
15:49:35 <cverna> yes
15:49:58 <pingou> nirik: what about the post-receive hook?
15:50:05 <pingou> from pagure to the batcave
15:50:12 <nirik> I guess we could take this back to list to try and come up with a short term solution. I really want pr's dammit. ;)
15:50:17 <bowlofeggs> why not put it on gitlab to reduce the chance of a second migration later? or wait until we settle on a git forge?
15:50:23 <nirik> bowlofeggs: -100
15:50:49 <nirik> well, we could wait... I do not want our ansible repo to be a scout for all the other things.
15:51:10 <bowlofeggs> i think waiting would be advisable
15:51:17 <bowlofeggs> the ticket in question is not really that important to get done
15:51:26 <nirik> we could. I just want pr's and such... sigh
15:51:28 <austinpowered> why not pagure? is it going away?
15:51:32 <cverna> we could put in on github since we are already on github, but then people might not be happy
15:51:43 <bowlofeggs> and doing integration work twice to solve a ticket that is honestly kinda low value i don't think is a good use of time
15:51:45 <pingou> github doesn't have push mirroring iirc
15:51:47 <nirik> yeah, this ticket isn't, but other things are. ;)
15:52:03 <cverna> ha
15:52:09 <nirik> austinpowered: unknown.
15:52:23 <bowlofeggs> putting it on github could work if rbac-playbook ran a git pull
15:52:26 <nirik> there's some talk about moving away from it, but nothing decided or known yet
15:52:39 <bowlofeggs> then there's no cron but also nirik's thing about how you want to run the thing that got merged is also solved
15:52:51 <nirik> I am not very in favor of github
15:53:21 <pingou> nirik: what about mirroring from pagure to batcave (w/o cron)?
15:53:24 <nirik> but I guess I can get over it if everyone else thinks it's what to do
15:53:58 <nirik> that would be fine for me, how does the mirroring work tho?
15:54:02 <austinpowered> I'm a fan of open-source - plenty of open git forge choices
15:54:10 <cverna> I am very much in the lines of nirik, that we should try to have everything in one place
15:54:19 <pingou> nirik: pagure creates a pair of ssh keys, give you the public one
15:54:26 <pingou> and will push to the mirror on git push to the project
15:54:37 <nirik> in this case since all our tickets are in pagure, it makes sense that our ansible repo that we solve the tickets in is also there...
15:55:17 <nirik> if we later move to gitlab or whatever, we should move both parts at the same time, after we have all the issues sorted out with less important repos.
15:56:02 <bowlofeggs> what sorts of issues do you expect?
15:56:06 <cverna> Yes, and I think we should start to have the discussion about the possible later move sooner rather than later
15:56:08 <nirik> pingou: ok, I guess I'd be ok with that for now
15:56:10 <bowlofeggs> i don't think the repo is particularly unusual or special
15:56:20 * dustymabe notes we are running out of time
15:56:30 <dustymabe> any closing statements before a brief open floor ?
15:57:00 * relrod ducks out to get ready for a 1:1 meeting.
15:57:01 <nirik> bowlofeggs: I have never used gitlab in my life. I am sure there will be a learning curve, problems with settings or permissions, or whatever... I'd prefer to learn those things on a repo that we aren't always using.
15:57:16 <nirik> we can continue in -admin or the like for anyone that wants to.
15:58:18 <bowlofeggs> when i started using gitlab from being a github user, i didn't find it difficult to learn
15:58:34 <bowlofeggs> it does a lot of the same things, and the settings menus are pretty self explanatory
15:58:38 <bowlofeggs> they have good docs too
15:59:05 <dustymabe> can we move the discussion elsewhere ?
15:59:15 <cverna> yes
15:59:22 <dustymabe> #topic open floor
15:59:23 <cverna> thanks dustymabe for chairing the meeting :)
15:59:33 <cverna> that was my openfloor
15:59:34 <dustymabe> anyone with a quick open floor topic ?
15:59:38 <dustymabe> cverna: :)
15:59:42 <pingou> I'd like to discuss this announce: https://hackmd.io/qIIcJYR_QWWPUHoRNc5dkg?both more specifically the last paragraph, are we fine with having folks open infra ticket for general issues that they don't know where to report?
16:00:06 <cverna> +1
16:00:11 <pingou> once this moves to production, I think we can use the fedora-ci/general issue tracker then
16:01:06 <nirik> I guess... who should be pinged on such tickets? you?
16:01:19 <pingou> yeah that'll be fine
16:02:41 <dustymabe> ok ending meeting in 30 seconds
16:03:05 <pingou> ttthanks
16:03:41 <dustymabe> #endmeeting