15:03:56 #startmeeting Infrastructure Meeting 15:03:56 Meeting started Thu Oct 31 15:03:56 2019 UTC. 15:03:56 This meeting is logged and archived in a public location. 15:03:56 The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:03:56 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:03:56 The meeting name has been set to 'infrastructure_meeting' 15:04:51 morning everyone 15:04:55 morning flks 15:04:57 sup 15:04:57 +o 15:05:05 morning 15:05:19 good morning 15:05:22 hey everyone 15:05:26 #info agenda is at https://board.net/p/fedora-infra 15:05:49 here 15:05:50 sorry I didn't realize I needed to send out the agenda yesterday 15:06:06 #meetingname infrastructure 15:06:06 The meeting name has been set to 'infrastructure' 15:06:12 #topic aloha 15:06:19 .hello zlopez 15:06:20 #chair nirik pingou relrod smooge tflink cverna mizdebsk mkonecny abompard bowlofeggs 15:06:20 Current chairs: abompard bowlofeggs cverna dustymabe mizdebsk mkonecny nirik pingou relrod smooge tflink 15:06:20 mkonecny: zlopez 'Michal Konečný' 15:06:42 .hello dustymabe 15:06:43 dustymabe: dustymabe 'Dusty Mabe' 15:06:58 #topic Next chair 15:06:59 no worries, thanks for running things today dustymabe! 15:07:06 #info magic eight ball says: 15:07:16 #undo 15:07:16 Removing item from minutes: INFO by dustymabe at 15:07:06 : magic eight ball says: 15:07:19 #info magic eight ball says: 2019-11-07: mkonecny 15:07:24 hello 15:07:32 do we want to add anyone else to run a future meeting? 15:07:55 I can take the 21st, but I'd rather leave it to someone else on the 14th 15:09:06 I can do the 14th 15:09:09 I guess I could do the 14th... but happy for someone else to. 15:09:12 mizdebsk: bowlofeggs austinpowered ? 15:09:40 smooge++ 15:09:45 #topic New folks introductions 15:09:52 #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves 15:09:54 #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted 15:11:29 i'm guessing no new folds 15:11:31 folks* 15:11:41 will move on in 30 seconds 15:12:08 #topic announcements and information 15:12:15 #info f31 final freeze now over 15:12:17 #info aarch64 hardware having issues 15:12:19 #info koji backups are done on secondary host. Slowdown in koji still occurring. 15:12:29 yay for freeze being over 15:12:46 actually I don't think koji is having slowdowns anymore... 15:12:53 and I think I fixed the backups yesterday. :) 15:12:57 #undo 15:12:57 Removing item from minutes: INFO by dustymabe at 15:12:19 : koji backups are done on secondary host. Slowdown in koji still occurring. 15:12:58 #info ci-resultsdb-listener schedule early next week 15:13:29 but I guess I will need to see how tonights backup runs. 15:14:35 anything else before next topic? 15:15:18 #topic Oncall 15:15:25 #info https://fedoraproject.org/wiki/Infrastructure/Oncall 15:15:27 oh hi there 15:15:27 #info smooge is on call 2019-10-17->2019-10-24 15:15:28 * cverna waives 15:15:29 #info relrod is on call 2019-10-24->2019-10-31 15:15:31 #info nirik is on call 2019-10-31->2019-11-07 15:16:01 I'll take it from the 14th on 15:16:02 zodbot: takoncallus 15:16:09 zodbot: takeoncallus 15:16:14 which leaves one week in b/w 15:16:35 nirik, I think there is a _us 15:16:39 So, all things considered, it was a slow week even with the release. I don't think I got any .oncall pings. Just lots of websites questions, but that's separate. ;) 15:16:48 zodbot: oncalltakeus 15:16:48 nirik: Kneel before zod! 15:17:09 yeah, I fear I have been backsliding a bit... and need to try harder. 15:17:24 I have been answering people that ping me directly. Need to stop that... 15:17:59 pingou: I updated the board to have you for 14th+ 15:18:20 thanks 15:18:27 who takes next week then? 15:18:53 nirik: Wonder if it's worth adding another alias to zodbot, .notoncall or something that says "Hi, you've directly pinged someone who isn't the on-call. Please direct your question through the on-call person (see .oncall) or file an Infra ticket." 15:19:02 and when they ping you just type that alias 15:19:14 I have an away message like that. should use it more. 15:19:43 any volunteers for next week on call? 15:20:06 nope 15:20:10 I can take it 15:20:28 sorry I was distracted 15:20:43 perfect 15:20:45 thanks 15:20:50 #info Summary of last week: (from smooge ) 15:21:01 #undo 15:21:01 Removing item from minutes: INFO by dustymabe at 15:20:50 : Summary of last week: (from smooge ) 15:21:15 I think relrod already gave us an update 15:21:32 * marcdeop is super late :'( 15:21:55 #topic Monitoring discussion 15:22:02 #info https://nagios.fedoraproject.org/nagios 15:22:03 marcdeop: For your punishment you must fix all the broken things ;) 15:22:04 #info Go over existing out items and fix 15:22:20 * nirik looks 15:22:29 the 3 down servers are all known 15:22:36 relrod++ 15:23:03 the datanommer ones... bugzilla is known broken, we have an internal ticket to fix it, but they don't seem to be making much progress. 15:23:10 the fas one needs us to setup a fas listener 15:23:18 sorry, emitter 15:23:39 and the qa09 one is because that machine is hosed up and should be re-installed. ;) 15:24:01 * marcdeop will buy a beer to any infra teem member if he/she drops by Berlin 15:24:38 marcdeop: carefull, we may have a flock near there sometime. ;) 15:24:40 nirik: does that address all open issues ? 15:25:17 marcdeop: I'll let you know when I passssSS BY :) 15:25:21 * pingou blames keyboard 15:26:03 dustymabe: yep. move along, unless anyone wants to discuss those more 15:26:41 #topic Tickets discussion 15:26:56 #info https://pagure.io/fedora-infrastructure/report/Meetings%20ticket 15:26:58 #info no tickets marked for the meeting 15:27:00 anything to bring up? 15:27:16 well, we should talk about the backlog... 15:27:24 unless we have a seperate place for that (I can't recall) 15:27:24 yeah, that's next 15:27:28 ok, cool. 15:27:34 well next topic 15:27:39 there's alos this in the current topic 15:27:48 #info Fedora CoreOS Team requests 15:27:54 :P 15:28:20 for this one I'd like to discuss https://pagure.io/fedora-infrastructure/issue/8218 15:28:38 any public docs for adding users to communishift ? 15:28:38 likes how I keep getting confused on whether i am looking at releng tickets or fedora-infrsatructure tickets because they seem to have the same number of open ones all the time 15:29:19 I know bowlofeggs was looking at making docs, but I don't know where that is. 15:29:24 I think bcotton started something when working on the fedocal migration 15:29:36 maybe we can get him to make an update in the ticket 15:29:45 ie he took some notes that could be later be used to create docs 15:30:14 dustymabe, https://fedora-infra-docs.readthedocs.io/en/latest/sysadmin-guide/sops/communishift.html#onboarding-new-users 15:30:42 nirik: i've put that project aside - it got bumped down in priority by many other things :( 15:30:51 alright, fair enough. 15:31:16 mizdebsk: i'd like to be able to add a group of people to a named project in communishift - do we know how add multiple users to a project 15:32:15 * cverna has no clue 15:32:41 * nirik doesnt know off hand either. 15:32:56 but we have a ansible role that does that, we could just look at which oc command it refers too 15:33:04 ie the appowners in ansible 15:33:21 cverna: we have an appowners file for communishift? 15:33:29 no 15:33:41 can't we have one ? 15:34:03 I'm not sure how it would work. 15:34:15 since we don't contol new apps. 15:34:45 yeah if we've been encouraging people to run apps there (i.e. infra doesn't own the apps) but they can't add more than one person to a project, then that's not good if that person gets hit by a bus or goes on vacation 15:34:47 it may be that those projects that want to add people create a permission for that? 15:34:56 Ideally it would be nice if the person that created the project could run a few oc command to add people to the project 15:35:04 cverna++ 15:35:10 * nirik completely agrees. 15:35:38 so how do we capture that? 15:35:38 we likely need to investigate/ask other openshift folks how to do this... 15:36:41 ok I'll try to update the ticket 15:36:43 someone has to investigate. I can try, but it's not going to be high on my list 15:37:29 maybe we can flag that as a possible apprentice task 15:37:34 and ask for help on the list 15:37:42 +1 that would be nice 15:37:50 nirik: can I bring up a releng ticket (that has infra implications)? 15:37:58 or should i wait on that? 15:38:13 sure... 15:38:25 I'm not sure this is apprentice level, but sure. :) 15:38:27 releng ticket: https://pagure.io/releng/issue/8811#comment-608625 15:38:39 cverna: thanks! 15:39:18 * nirik nods at the last update there. 15:39:55 I think we could sync things to the new volume then swap them with minimal downtime 15:40:10 nirik: yep, i'm worried less about bullet #2 15:40:18 bullet #1 is the open question 15:40:30 so maybe we can look at that sometime in the next few weeks ? 15:40:40 sure. 15:41:02 will coordinate with you offline (or someone else if it's more appropriate) 15:41:11 #topic backlog discussion 15:41:17 #info go over our backlog and discuss and determine priority 15:41:53 is that this? https://pagure.io/fedora-infrastructure/issues?status=Open&tags=backlog&close_status= ? 15:41:56 so the 2 there we have had for a while... 15:42:13 dustymabe: yes, but the two listed there are the ones we are trying to finish 15:42:17 nirik: ok 15:42:21 i'll info them 15:42:29 #link https://pagure.io/fedora-infrastructure/issue/8178 15:42:31 #info topic: provision new aarch64 builders 15:42:33 #link https://pagure.io/fedora-infrastructure/issue/8157 15:42:35 #info topic: ansible: enable ansible-report as a hook 15:42:54 ok, on aarch64 builders we are making some progress now... after hitting lots of issues. ;) 15:43:12 Hopefully that will be done in the next week now. 15:43:32 On the ansible-report hook, we need to decide what we want to do there... 15:43:53 I think the nightly report would be a good first step 15:44:10 how bad is it currently? I thought we fixed all the errors? 15:44:14 has anyone ever run ansible-report against the current repo? 15:44:33 we don't have errors, just a lot of warnings 15:44:58 austinpowered: yep. ;) 15:45:02 mostly about bad practices, like using the command module 15:46:08 ok. perhaps an initial version of that to the list and people can fix things? 15:47:01 I was pondering if we shouldn't just move to pagure as a normal repo, modify ansible-playbook/rbac-playbook to just pull before running and move to repospanner/gitlab/the moon later? 15:47:44 we could add a cron job to pull from pagure into the batcave on a regular basis 15:48:03 could pagure automatically sync the batcave on each commit ? 15:48:11 so if pagure goes down we have an ansible repo that is only missing some commits but not all 15:48:13 like a post-receive hook 15:48:15 it can 15:48:49 I don't like cron. 15:48:59 someone commits something they want to run/apply it... 15:49:35 yes 15:49:58 nirik: what about the post-receive hook? 15:50:05 from pagure to the batcave 15:50:12 I guess we could take this back to list to try and come up with a short term solution. I really want pr's dammit. ;) 15:50:17 why not put it on gitlab to reduce the chance of a second migration later? or wait until we settle on a git forge? 15:50:23 bowlofeggs: -100 15:50:49 well, we could wait... I do not want our ansible repo to be a scout for all the other things. 15:51:10 i think waiting would be advisable 15:51:17 the ticket in question is not really that important to get done 15:51:26 we could. I just want pr's and such... sigh 15:51:28 why not pagure? is it going away? 15:51:32 we could put in on github since we are already on github, but then people might not be happy 15:51:43 and doing integration work twice to solve a ticket that is honestly kinda low value i don't think is a good use of time 15:51:45 github doesn't have push mirroring iirc 15:51:47 yeah, this ticket isn't, but other things are. ;) 15:52:03 ha 15:52:09 austinpowered: unknown. 15:52:23 putting it on github could work if rbac-playbook ran a git pull 15:52:26 there's some talk about moving away from it, but nothing decided or known yet 15:52:39 then there's no cron but also nirik's thing about how you want to run the thing that got merged is also solved 15:52:51 I am not very in favor of github 15:53:21 nirik: what about mirroring from pagure to batcave (w/o cron)? 15:53:24 but I guess I can get over it if everyone else thinks it's what to do 15:53:58 that would be fine for me, how does the mirroring work tho? 15:54:02 I'm a fan of open-source - plenty of open git forge choices 15:54:10 I am very much in the lines of nirik, that we should try to have everything in one place 15:54:19 nirik: pagure creates a pair of ssh keys, give you the public one 15:54:26 and will push to the mirror on git push to the project 15:54:37 in this case since all our tickets are in pagure, it makes sense that our ansible repo that we solve the tickets in is also there... 15:55:17 if we later move to gitlab or whatever, we should move both parts at the same time, after we have all the issues sorted out with less important repos. 15:56:02 what sorts of issues do you expect? 15:56:06 Yes, and I think we should start to have the discussion about the possible later move sooner rather than later 15:56:08 pingou: ok, I guess I'd be ok with that for now 15:56:10 i don't think the repo is particularly unusual or special 15:56:20 * dustymabe notes we are running out of time 15:56:30 any closing statements before a brief open floor ? 15:57:00 * relrod ducks out to get ready for a 1:1 meeting. 15:57:01 bowlofeggs: I have never used gitlab in my life. I am sure there will be a learning curve, problems with settings or permissions, or whatever... I'd prefer to learn those things on a repo that we aren't always using. 15:57:16 we can continue in -admin or the like for anyone that wants to. 15:58:18 when i started using gitlab from being a github user, i didn't find it difficult to learn 15:58:34 it does a lot of the same things, and the settings menus are pretty self explanatory 15:58:38 they have good docs too 15:59:05 can we move the discussion elsewhere ? 15:59:15 yes 15:59:22 #topic open floor 15:59:23 thanks dustymabe for chairing the meeting :) 15:59:33 that was my openfloor 15:59:34 anyone with a quick open floor topic ? 15:59:38 cverna: :) 15:59:42 I'd like to discuss this announce: https://hackmd.io/qIIcJYR_QWWPUHoRNc5dkg?both more specifically the last paragraph, are we fine with having folks open infra ticket for general issues that they don't know where to report? 16:00:06 +1 16:00:11 once this moves to production, I think we can use the fedora-ci/general issue tracker then 16:01:06 I guess... who should be pinged on such tickets? you? 16:01:19 yeah that'll be fine 16:02:41 ok ending meeting in 30 seconds 16:03:05 ttthanks 16:03:41 #endmeeting