18:00:24 <smooge> #startmeeting Infrastructure (2017-07-27)
18:00:24 <zodbot> Meeting started Thu Jul 27 18:00:24 2017 UTC.  The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:24 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:24 <zodbot> The meeting name has been set to 'infrastructure_(2017-07-27)'
18:00:24 <smooge> #meetingname infrastructure
18:00:24 <zodbot> The meeting name has been set to 'infrastructure'
18:00:24 <smooge> #topic aloha
18:00:24 <smooge> #chair smooge relrod nirik abadger1999 dgilmore threebean pingou puiterwijk pbrobinson
18:00:24 <zodbot> Current chairs: abadger1999 dgilmore nirik pbrobinson pingou puiterwijk relrod smooge threebean
18:00:27 <clime> hey
18:00:27 <smooge> Hello all
18:00:35 <marc84> hi everyone
18:01:06 <relrod> here
18:01:12 * doteast here
18:01:20 * pingou around but not really here
18:02:05 <nirik> morning
18:02:39 <smooge> #topic New folks introductions
18:02:47 <smooge> Hi do we have any new folks here?
18:02:51 * cverna waves
18:04:04 <smooge> ok looks like not
18:04:15 <smooge> #topic announcements and information
18:04:15 <smooge> #info PHX2 Colo Trip coming up, Aug 14-18
18:04:15 <smooge> #info Major outage planned for Aug14->18
18:04:15 <smooge> #info FLOCK at Cape Code  Aug29->Sep01
18:04:16 <smooge> #info Fedora F27 Rebuild
18:04:28 <smooge> Any other announcements from people?
18:04:51 <clime> it's Cape Cod.
18:04:51 * nirik has nothing off hand.
18:05:15 <smooge> clime, its in gobby you can fix it :)
18:05:27 <clime> ok
18:05:28 <nirik> ah, but while we are there it will cape code. ;) so much coding to do
18:05:49 <nirik> (yes, this is a joke, thats a typo)
18:06:20 <clime> fixed
18:06:33 <clime> :)
18:06:45 <smooge> thanks :). I was leaving it that way until someone did so because I found my mistype funny
18:07:04 <bowlofeggs> .hello bowlofeggs
18:07:04 <smooge> nirik, how is the rebuild going?
18:07:04 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <randy@electronsweatshop.com>
18:07:11 <bowlofeggs> .hello jcline
18:07:12 <zodbot> bowlofeggs: jcline 'Jeremy Cline' <jeremy@jcline.org>
18:07:18 <smooge> I haven't seen much on it this round
18:07:57 <nirik> it's in the 'r's commiting
18:08:29 <nirik> probibly finish that later today... then just needs to finish up all the builds.
18:08:35 <smooge> I saw a couple of weird errors on lists like arm32 running out of memory for no reason and some ppc items.. do we need much of a re-rebuild?
18:08:37 <nirik> probibly take another day or so
18:08:50 <smooge> ok cool
18:09:01 <nirik> the pcc one we might rebuild failed things again for
18:09:06 <nirik> (once it's fixed)
18:09:13 <smooge> there was a pagure pkgs thing also this week. But I don't know the details on it
18:09:32 <pingou> it's waiting on mass-rebuild to be done
18:10:01 <pingou> but I just run into what is I think a bug in pkgdb, which is going to be required to be fixed for the migration to work properly
18:10:07 * pingou will investigate
18:10:28 <nirik> https://bugzilla.redhat.com/show_bug.cgi?id=1475636 is the ppc binutils bug
18:11:38 <smooge> ok thanks
18:11:54 <smooge> ok next up
18:11:57 <smooge> #topic (2017-07-27) Service Level Expectations (SLE)
18:11:57 <smooge> #info What are SLE's?
18:11:57 <smooge> #info Why do we need them?
18:11:57 <smooge> #info Who sets them?
18:11:57 <smooge> #info How are they followed?
18:11:58 <smooge> #info Where do they affect things?
18:12:00 <smooge> #info When do we put them in place?
18:12:02 <smooge> #info https://pagure.io/fedora-infrastructure/issue/6140
18:12:04 <smooge> #info https://fedoraproject.org/wiki/Infrastructure/ServiceLevelExpectations
18:12:06 <smooge> #info https://confluence.cornell.edu/display/itsmp/Service+Level+Expectations
18:12:18 <nirik> yeah, not too much discussion on list on this...
18:12:22 <smooge> Hi nirik I put in questions just to frame a possible discussion
18:12:25 <nirik> I guess I will try and work on it some more.
18:12:36 <nirik> I can answer some of those questions. :)
18:13:03 <nirik> SLE is a service level expectation... it's letting users/consumers of a service know what kind of service to expect.
18:13:55 <nirik> ie, if the service was down at 3am, would people wake to fix it? would it be fixed the next morning? if on a weekend or holiday would it be fixed the next business day?
18:14:24 <nirik> we can also use these with other projects we work with...
18:14:53 <nirik> ie, say we use centos ci and something is broken on our side of that, what expectations do they have for us to fix things when, etc.
18:15:14 <nirik> I set out some broad outline on the wiki page using domains...
18:15:25 <nirik> but there's more to fill in, which I can work on this coming week
18:15:41 <nirik> we will need/want to redo our status app to reflect this
18:16:03 <nirik> and possibly outage pages from apps, etc.
18:16:27 <nirik> any further thoughts on this?
18:17:50 <smooge> so we could key items to nagios alerts
18:18:03 <nirik> that also.
18:18:28 <nirik> have all continue to go to irc... but pages only for some things at some time periods
18:18:29 <smooge> with 24x7x52 vs 8x5x50 vs meh you got the bits didnt ya
18:19:21 <smooge> yeah agreed
18:19:41 <nirik> so there would definitely be some work to adjust to this, but in the end I hope it would be nicer for our users to know and us to not have to treat everything as urgent all the time
18:21:35 <nirik> we might also at the same time work on the consistent app naming setup...
18:21:36 <smooge> agreed there too
18:21:48 <smooge> now lets not go overboard
18:22:04 <nirik> https://pagure.io/fedora-infrastructure/issue/5644 is that ticket
18:22:42 <doteast> sorry, SLE no SLA right?
18:22:49 <smooge> correct
18:23:08 <smooge> an agreement requires two entities
18:23:15 <nirik> and money and signing things
18:23:21 <doteast> I see ;)
18:23:34 <nirik> There's no such relationship between us and our community...
18:23:45 * doteast kind like the E more than the A
18:24:14 <doteast> even in outside of community context
18:24:27 <nirik> if we fail to meet expectations, then thats something we would revist when it happened... and depending on why adjust the expectation or add more resources or something
18:24:44 <doteast> sweet
18:24:54 <smooge> Knowing some in our community.. we will always fail their expectations.
18:24:57 <smooge> and that is ok
18:25:19 <doteast> ah, you can win them all, even in business settings I guess
18:25:33 <nirik> sure... but these are our expectations.
18:25:58 <nirik> if we say app X will be addressed in 4 hours, but we don't... we need to adjust that or make sure we notify people harder or whatever.
18:26:20 <smooge> ah that goes to #info Who sets them?
18:26:21 * doteast nods
18:26:24 <nirik> probibly when apps are in their RFE process we can ask what expectations should be...
18:27:00 <mizdebsk> you mean RFR?
18:27:04 <nirik> "this is a fedoracommunity app we are just playing around with" -> no monitoring, whoever runs it responsible, no SLE at all.
18:27:11 <nirik> yeah, RFR, sorry. ;)
18:27:20 <doteast> request for resources
18:27:49 <nirik> "this is mirror lists for end users" -> monitored, will be acked in 15min, will be fixed asap, all hands on deck until it's working again.
18:28:47 <clime> .thisdoesntwork would be a good domain for apps in RFR ;)
18:28:59 <bowlofeggs> can we get a 1 minute SLE on puiterwijk? ☺
18:29:04 <smooge> no
18:29:07 <bowlofeggs> haha
18:29:29 <smooge> he requires same reverse SLE
18:29:32 <bowlofeggs> jcline pointed out that first we would need monitoring on puiterwijk
18:29:46 <smooge> and then monitoring on us
18:29:54 <nirik> since he's not here we can assign him that. ;)
18:30:04 <nirik> who monitors the monitors?
18:30:05 <bowlofeggs> but who monitors the monitoring?
18:30:07 <bowlofeggs> hahaha
18:30:07 <smooge> so much more on this?
18:30:25 <smooge> or time to move onto clime's topic?
18:30:28 <bowlofeggs> i'm a +1 to definiing clear SLEs
18:30:45 * nirik has nothing else on this unless theres questions.
18:30:57 <bowlofeggs> cool far fetched idea: the page that shows when an app is down could also state that app's SLE
18:31:01 <nirik> I'll try and add to it and get another round of review
18:31:09 <bowlofeggs> (i think the proxy does that?)
18:31:14 <nirik> bowlofeggs: yeah, we would want to adjus our status page for that
18:31:25 <nirik> and yeah, there's a outage html page for down apps
18:31:27 <bowlofeggs> i think that might be slightly different
18:31:34 <smooge> #topic possible future Fedora Infrastructure support for COPR - clime
18:31:37 <bowlofeggs> yeah i mean the page you see when you go to bodhi
18:31:37 <nirik> yeah, sorry, but both should be adjusted.
18:32:15 <clime> right, so we will have COPR solely from Fedora packages soon
18:32:21 <clime> like this week
18:32:25 <clime> I hope
18:32:43 <bowlofeggs> clime: what does it mean for them to be solely from fedora packages?
18:32:56 <bowlofeggs> like, the install of copr itself?
18:33:01 <bowlofeggs> or it hosts only fedora packages?
18:33:02 <clime> bowlofeggs: currently we deploy COPR from @copr/copr
18:33:05 <bowlofeggs> ah i see
18:33:07 <bowlofeggs> cool
18:33:15 <clime> which is good for hotfixes
18:33:26 <nirik> cool.
18:33:37 <bowlofeggs> we have an infra repo that we can use for hotfixes like that too
18:33:43 <clime> but one of the conditions for getting support of FI was to install everything from Fedora
18:33:52 <clime> oh that would be cool
18:34:05 <clime> I would really like to have an option to hotfix things quickly :)
18:34:22 <clime> but anyway I am happy that we are ready
18:34:44 <nirik> the reason we have the koji tag is that the koji builds are immutable once done, etc.
18:35:17 <clime> right, so I would like to ask what next progress should be
18:35:31 <clime> but I guess I will need to discuss it with puiterwijk
18:35:39 <clime> once he is here
18:35:43 <nirik> so, looking back at the RFR ticket... did we ever finish discussing/deciding what parts we want of copr to be in cloud and what parts outside?
18:35:58 <clime> actually I would like to move to OpenShift
18:36:08 <nirik> hopefully our cloud will be improved a lot soon...
18:36:19 <clime> not sure if it is related or not but It would be cool
18:36:33 <clime> so dev instance first
18:36:34 <nirik> clime: so, builds are done in build containers (like images are done in openshift)?
18:36:49 <clime> yup
18:36:57 <clime> nirik: that would be cool
18:37:17 <nirik> I think that could be interesting... but not sure how much work it might be. it changes a lot of what copr does...
18:37:22 <clime> well I guess there will be some work included
18:38:23 <nirik> but that is just the builder part right? the frontend/backend/keyserver/distgit would all be pretty much the same?
18:38:38 <clime> nirik: yes, exactly
18:38:49 <clime> nirik: unless they would also need to run in pods
18:39:16 <nirik> well, we currently have no good story for persistent storage in openshift... and some of those need a lot of that
18:39:17 <clime> but that wouldn't normally affects things...
18:39:35 <clime> I see.
18:39:55 <clime> maybe I could cooperate on investigating this
18:40:04 <doteast> wasn't there an idea to go for glusterfs for that....
18:40:09 <nirik> all the options are kinda poor for us...
18:40:20 * doteast is hazy on those matters
18:40:22 <clime> but I no pretty much none so far
18:40:26 <clime> *know
18:40:28 <nirik> yeah, we could use glusterfs...
18:40:42 <nirik> but we have had problems with performance in the past....
18:41:11 <nirik> and folks who run openshift in production don't use it, so hard to say how effective it will be
18:41:49 <bowlofeggs> i'm excited about openshift coming to an infra near me
18:41:50 <nirik> we could use nfs, but it requires us to manually make volumes to be used
18:42:05 <nirik> anyhow...
18:42:09 <bowlofeggs> nfs can also have problems with some apps, depending on what they do
18:42:43 <nirik> so, in the next month or so we should get new cloud hardware and install RHOSP 10+ with HA.
18:43:05 <nirik> we can move copr as it is now to that and get it much better supported hopefully.
18:43:21 <clime> that would be cool
18:43:23 <nirik> I think a way to use openshift for builders at least (or the entire thing) would be pretty cool tho down the road
18:43:49 <clime> Can I talk about this to somebody?
18:44:00 <clime> relrod maybe
18:44:01 <nirik> which part? :)
18:44:09 <clime> builders in openshift
18:44:36 <clime> and the first one as well
18:44:37 <nirik> I think we are all feeling our way on openshift right now, perhaps the list would be good or just admin irc channel?
18:44:52 <smooge> I would say the list would be good to capture it
18:45:05 <clime> alright
18:45:07 <relrod> Yeah I'm not the best person to talk to about it just yet. Still learning the ropes.
18:46:04 <clime> That's everything I have got.
18:46:10 <mizdebsk> what about nearer future? once copr is installable from fedora rpms, can at least the frontend be made "supported" service (so that other apps can depend on it)?
18:46:17 <nirik> https://docs.openshift.com/container-platform/3.4/creating_images/custom.html#creating-images-custom may be of note
18:46:29 <nirik> mizdebsk: once it moves to the new openstack yes.
18:46:40 <nirik> the current one is not HA, has issues
18:47:07 <clime> it is working pretty well in the end
18:47:11 <mizdebsk> i meant having frontend on kvm virthosts, with backend/builders still in cloud
18:47:12 <clime> but yeah
18:47:55 <clime> could be better for sure
18:48:02 <nirik> we could do that. (see above where I mention we haven't decided that yet)
18:48:31 <nirik> I think frontend would be easy to move out if we wanted.
18:48:51 <nirik> backend makes more sense to stay in cloud IMHO.
18:49:02 <smooge> OK we are coming up to the top of the hour and I would like to do the apprentice questions before we end
18:49:09 <nirik> (it has storage there, it also doesn't need public ips to talk to builders)
18:49:22 <nirik> so, anything else please bring up on list. ;)
18:49:29 <clime> okay
18:49:33 <clime> thank you.
18:49:58 <smooge> I think a plan to finish making this 'supported' will be part of the list thread
18:50:04 <smooge> ok next up
18:50:15 <smooge> #topic Apprentice Open office hours
18:50:24 <smooge> Hi any apprentices or new people with questions?
18:51:04 <bgray> no questions, just a note - please let me/us know if there is more to help out with.
18:51:12 <ole88> I'm new to both the project and the OS recently
18:51:19 <smooge> hello ole88
18:53:58 <smooge> as a new person, are you interested in sysadmin or coding?
18:54:15 <smooge> bgray, yes.. we do need to put some more or clear up the easyfix
18:54:48 <bgray> smooge: thanks!
18:55:00 <smooge> bgray, have you looked at our docs?
18:55:21 <ole88> I am a sysadmin at work and we are moving toward RHEL, but run Windows
18:55:31 <smooge> those are probably the place we need the most work on these days even if it is a "hey this doc is from 1984.. does this still work?)
18:55:44 <smooge> cool ole88 that can be a big leap
18:55:47 <ole88> I am also learning python in my spare time
18:55:49 <bgray> smooge: i have. they are very good!
18:56:07 <ole88> I have prior experince (years ago) with c, c++ and cobol
18:56:11 <nirik> Theres a issue about updating our mirrormanager docs... that could be a nice one for someone new.
18:56:16 <smooge> bgray, well I am going to put in an easy fix ticket for them to get get updated/reviewed
18:56:16 <bgray> smooge: ah, so outdated in places?
18:56:36 <smooge> bgray, if the data at the top is over a year old they are probably outdated
18:56:39 <ole88> I also use PowerShell and dotnetcore on my Fedora laptop
18:56:45 <bgray> smooge: :)
18:57:04 <smooge> ole88, that is a level of work I have not been able to get working yet.
18:57:48 <ole88> I can help out there
18:57:51 <smooge> cool
18:57:58 <smooge> well we are at the top of the hour
18:58:07 <smooge> #topic Open Floor
18:58:19 <smooge> I would like to thank eveyrone for coming to the meeting today
18:58:26 <marc84> thanks smooge
18:58:36 <nirik> thanks everyone
18:58:36 <relrod> thanks for running it, smooge!
18:58:39 <clime> thank you, smooge
18:58:48 <smooge> we will have another meeting in a week and we will also have a mailing list for people to talk on
18:58:51 <smooge> see you later
18:58:54 <smooge> #endmeeting