18:00:01 <relrod> #startmeeting Infrastructure (2017-08-17)
18:00:01 <zodbot> Meeting started Thu Aug 17 18:00:01 2017 UTC.  The chair is relrod. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:01 <zodbot> The meeting name has been set to 'infrastructure_(2017-08-17)'
18:00:01 <relrod> #meetingname infrastructure
18:00:01 <zodbot> The meeting name has been set to 'infrastructure'
18:00:01 <relrod> #topic aloha
18:00:01 <relrod> #chair smooge relrod nirik abadger1999 dgilmore threebean pingou puiterwijk pbrobinson
18:00:01 <zodbot> Current chairs: abadger1999 dgilmore nirik pbrobinson pingou puiterwijk relrod smooge threebean
18:00:12 <relrod> Good afternoon everyone!
18:00:20 <marc84> hi relrod
18:00:22 <netcronin> hi relrod
18:00:22 <pingou> Good evening :)
18:00:25 <jcline> Hello folks
18:00:36 <tflink> hello
18:01:29 * threebean waves
18:01:46 <relrod> #topic New folks introductions
18:02:03 <nirik> morning everyone
18:02:07 <relrod> Anyone looking to join the team or get involved who wants to give a short introduction about themselves and what they are looking to help out with?
18:02:56 <relrod> nirik: yeah I guess it is still morning for you ;)
18:03:05 <nirik> it's always morning on irc. ;)
18:03:24 <puiterwijk> morning
18:03:24 <pingou> noon nirik
18:03:45 <nirik> http://www.total-knowledge.com/~ilya/mips/ugt.html
18:03:48 <nirik> ^
18:03:53 <puiterwijk> Yep, that
18:04:03 <puiterwijk> nirik: was just about to send the link myself as well :)
18:04:53 <relrod> Okay, I guess we will move into the info/announcements section if nobody wants to give an introduction
18:05:14 <relrod> #topic announcements and information
18:05:14 <relrod> #info PHX2 Colo Trip, Dec 4th - 9th
18:05:14 <relrod> #info FLOCK at Cape Cod  Aug 29th - Sep 1st
18:05:15 <relrod> #info beta freeze coming up 2017-09-05 - everyone
18:05:46 <puiterwijk> #info Ipsilon 2.1 has just been tagged and will be in stg today - patrick
18:05:47 <relrod> Any other items that people should be aware of over the last week or upcoming week?
18:05:56 <puiterwijk> Oh!
18:06:01 <pingou> #info FMN 1.4.1 in stg with CI notifications
18:06:06 <pingou> (thanks to jcline )
18:06:12 <puiterwijk> #info SSH known_hosts file has been thrown out and replaced with SSH certificates
18:06:19 <relrod> woo!
18:06:36 <puiterwijk> Look forward to more SSH changes!
18:06:37 <nirik> puiterwijk: perhaps you could do a blog post or other longer thing about how you set that all up...
18:06:44 <nirik> I bet lots of people would be interested.
18:06:47 <nirik> or a magazine post
18:06:52 <puiterwijk> nirik: yeah, was going to do that once I got the other side of things working too
18:07:02 <puiterwijk> (which is part of Ipsilon 2.1)
18:07:18 <nirik> ok, fair enough.
18:08:06 <relrod> #info greenwave is almost set up in openshift stg
18:08:14 <threebean> \ó/
18:08:50 <relrod> which I guess leads me to a #topic and putting nirik on the spot a bit:
18:08:55 <relrod> #topic prod openshift
18:09:04 <relrod> are there any blockers still here, or can we start getting this up?
18:09:38 <nirik> I think we should stand it up soon.
18:09:47 <nirik> I just haven't had any time to yet... I was hoping to last week
18:10:16 <relrod> Okay. I'd be happy to try it or to help with it.
18:10:19 <nirik> and then it would be ready for apps as they are ready for it.
18:10:39 <nirik> we have the ssl cert... just a matter of making the hosts
18:11:10 <relrod> nirik: it's all ansibled from the stg one, right? So should just be defining some host_vars and running playbooks?
18:11:17 <nirik> yep.
18:11:23 <relrod> cool
18:11:28 <nirik> oh... wait...
18:11:35 * nirik checks something.
18:13:10 <nirik> trying to tell if our stg version got updated or not
18:13:25 <jcline> #info bodhi-2.10 beta in stg, release on friday assuming no issues, prod early next week
18:13:25 <nirik> the new version came out recently... we may want to upgrade it before doing prod
18:13:43 * relrod nods
18:13:44 <puiterwijk> nirik: it did not. It's still the version from before the hackfest
18:13:55 <nirik> ok, we should try and upgrade it.
18:13:57 <puiterwijk> Note that as Adam said, the new one bumps the requirements significantly
18:14:05 <puiterwijk> (to 16GB mem/node)
18:14:08 <nirik> yeah, 16gb memory for all nodes.
18:14:12 <nirik> anoying
18:14:22 <relrod> :(
18:14:24 <puiterwijk> And then they still have the 110pods/node hard limit :/
18:15:07 <puiterwijk> (or I think they might've bumped it, but it's still way below the memory requirements if you're doing anything python)
18:15:25 <nirik> we may need to look at adding some memory... but I think we can hopefully handle it
18:16:42 <relrod> puiterwijk: is upgrading just a matter of updating the packages or is there more to it?
18:17:00 <puiterwijk> relrod: there's more to it. There's an upgrade playbook in the ansible-openshift repo
18:17:07 <relrod> okay
18:18:03 <nirik> https://docs.openshift.com/container-platform/3.6/install_config/upgrading/automated_upgrades.html
18:18:50 <relrod> Alright, any other topics that people want to discuss before we move to apprentice office hours?
18:19:30 <relrod> #topic Apprentice Open office hours
18:20:06 <relrod> Any apprentices have questions, looking for things to work on, want to check in and say hi?
18:20:26 <netcronin> Hi all. I'm working on issue 5931 and would love help on how to safely sed IP changes.
18:20:54 <netcronin> I've also been idling around if anyone needs help with other issues. -- mainly been dealing with Zodbot breaking.
18:21:21 <relrod> #link https://pagure.io/fedora-infrastructure/issue/5931
18:21:29 <netcronin> relrod, thanks.
18:22:34 <nirik> thanks for being around netcronin. ;)
18:22:43 <netcronin> Absolutely. Happy to help.
18:22:44 <relrod> netcronin: I think smooge was heading that ticket, but he's out today
18:22:44 <nirik> can you use $ (end of line) in your sed?
18:23:35 <netcronin> Yes. I'm seeing the issue where I'll do sed -i 's/10.5.126.3/10.5.128.33' and it will wreck any 10.5.128.3xx IPs.
18:23:56 <netcronin> So I'll have entries after the sed that look like 10.5.126.333
18:24:18 <Southern_Gentlem> why would you have any greater than 255
18:24:46 <netcronin> It's because the sed replace has no boundary after the 3
18:24:50 <x3mboy> .hello2
18:24:51 <zodbot> x3mboy: x3mboy 'Eduard Lucena' <eduardlucena@gmail.com>
18:25:28 <netcronin> Southern_Gentlem, Oh I see what you're saying. I meant it more like if I have .3x not .3xx
18:25:29 <nirik> toss a $ in there if thats the end of the ip...
18:25:40 <netcronin> nirik, Okay, I'll give that a shot, thanks!
18:25:51 <nirik> 's/10.5.126.3$/10.5.128.33/'
18:26:03 <netcronin> Cool, appreciate it.
18:26:38 <nirik> if you can get a patch for changing all those we could apply it, run it and then reboot them all and hopefully have em changed. ;)
18:27:04 <netcronin> Yeah I'm working through it now on batcave, I have a branch open, once it's done I'll get a patch uploaded.
18:27:23 <relrod> if it's the end of the line $ should work. Otherwise [:space:] or whatever posix regex uses.
18:27:31 * relrod doesn't claim to know posix regex very well ;)
18:28:43 <relrod> #topic Open foor
18:28:46 <relrod> #topic Open floor
18:29:23 <relrod> Anyone have anything else they'd like to bring up, other than the fact that I can't type? :)
18:30:13 <vbird> hey sorry i was in the train
18:30:14 * nirik tries to recall what he wanted to bring up
18:30:28 <relrod> welcome vbird
18:30:32 <relrod> and x3mboy
18:30:40 <vbird> as I said poor connection and now on my smartphone :)
18:30:44 <x3mboy> relrod
18:30:46 <vbird> hi all! :)
18:30:48 <x3mboy> relrod, o/
18:31:09 <nirik> welcome. ;)
18:31:14 <nirik> oh, one thing to mention...
18:31:45 <nirik> we just got a new machine that we are going to migrate the koji database to. It's got SSD's in it and hopefully will help koji performance a fair bit.
18:32:11 <nirik> we may need an outage for doing that, or I might be able to live migrate it.
18:32:54 <vbird> in my $DAYJOB I'm a sysadmin (freelance), spent most of my time working with / thanks to foss, now I'm looking for a way to contribute
18:33:19 <nirik> Oh and some elections are still going on, please look at candidates and vote! https://admin.fedoraproject.org/voting/
18:33:22 <vbird> the rest was in my post on the list :)
18:34:11 <relrod> vbird: awesome, I assume you're looking to do sysadmin more than app development then?
18:34:53 <vbird> relrod: more, yes (it's what I know the best) :)
18:35:01 * relrod nods
18:35:50 <vbird> but I'm a multitask boy too, so no pb working several skills ;)
18:36:28 <relrod> vbird: Alright. Well hang around in #fedora-admin and #fedora-noc and #fedora-apps, and we can get you sponsored into the apprentice group so you can start looking around the infra.
18:37:11 <relrod> Alright, anyone have anything else?
18:37:33 <puiterwijk> Oh, one thing
18:37:56 <puiterwijk> I think we need to find a list of all people that are able to run playbooks, and just forcefully subscribe them to the infra list
18:38:04 <vbird> relrod: great! thanks a lot! (I already started browsing the SOPs and I'll take some more time next weekend on it)
18:38:20 <x3mboy> I'm a recurrent one, super-slow apprentice (in the way that it's my sixth or seventh meeting and still just looking around) I'm kind of into deep in marketing in the project, but I'm a sysadmin too
18:38:21 <x3mboy> :D
18:38:21 <vbird> puiterwijk: :))
18:38:22 <puiterwijk> Since there's been at least one time where people that are able to either tag into the infra tags or run playbooks in our infra weren't aware of changes because they're not subscribed to the infra list
18:39:16 <nirik> can we just invite them? if they fail then it's their fault.
18:39:57 <puiterwijk> nirik: I'd personally say that if they don't want on the infra list, they get removed from access. Since otherwise we're going to have to explain changes lots of times
18:40:24 <puiterwijk> Aka access to run playbooks (not cloud)
18:40:31 <puiterwijk> or infra tags -> infra list
18:41:20 <puiterwijk> but sure, let's start with inviting them, and then get more strict next week. (or discuss becoming more strict on the infra list!)
18:41:40 <relrod> +1
18:42:27 <relrod> Alright, will wait another minute or so and then close out
18:42:33 <nirik> sure.
18:42:58 <vbird> puiterwijk: making #fedora-noc monitoring compulsory wouldn't be enough?
18:43:29 <puiterwijk> vbird: my point was actually about changes that we announced to the process, and people with access not being aware because they're not on the mailing list
18:43:55 <puiterwijk> Like, we recently added staging tags for the infra tags, and at least one person with access to the tags was not aware of the change because they weren't on the infra list
18:44:20 <puiterwijk> In my opinion, if you have the permissions to make changes to our prod systems, you also have the responsibility of staying in the loop on process changes
18:44:21 <vbird> people from the group not on the list?
18:44:50 <puiterwijk> From other groups that got access to executing the playbook for their app
18:45:12 <puiterwijk> Anyway, that was my remark, and we can discuss further on #fedora-admin :)
18:45:26 * vbird (humbly) approves :)
18:46:18 <relrod> Cool, thanks for coming all!
18:46:25 <relrod> #endmeeting