18:00:04 <nirik> #startmeeting Infrastructure (2015-12-03)
18:00:04 <zodbot> Meeting started Thu Dec  3 18:00:04 2015 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:04 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:04 <zodbot> The meeting name has been set to 'infrastructure_(2015-12-03)'
18:00:04 <nirik> #meetingname infrastructure
18:00:04 <nirik> #topic aloha
18:00:04 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk pbrobinson
18:00:04 <zodbot> The meeting name has been set to 'infrastructure'
18:00:04 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pbrobinson pingou puiterwijk relrod smooge threebean
18:00:04 <nirik> #topic New folks introductions / Apprentice feedback
18:00:14 * relrod here
18:00:30 <puiterwijk> hi
18:00:35 * tflink here
18:00:35 <nirik> morning everyone. Any new folks like to introduce themselves? Or apprentices with questions or comments? If so, chime right in. ;)
18:00:35 <nb> hi
18:00:36 <odin2016> hi
18:00:38 <ladisone> hi
18:01:11 <threebean> hi :)
18:01:47 <ladisone> Hi I am Lada and I would like to help with the administration infrastructure
18:02:17 <nirik> welcome ladisone. Are you more interested in sysadmin type tasks or application development? or both?
18:03:31 <ladisone> nirk: sysadmin task, but a little develop
18:04:18 <nirik> ok. cool. See us all in #fedora-admin after the meeting and we can get you started. ;)
18:04:28 <nirik> any other new folks? or apprentices with questions?
18:05:06 <odin2016> not i, not today. :)
18:05:39 <ladisone> nirik: ok
18:05:44 * nirik sighs at wayland and it's focus issues.
18:05:51 <nirik> ok, on to status and info dump:
18:05:56 <nirik> #topic announcements and information
18:05:56 <nirik> #info [release] pagure: 0.1.34 and 0.1.35 - pingou
18:05:56 <nirik> #link http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/BMOP6VP3VFLI56DIASTKX6JF3W6ORXNO/
18:05:56 <nirik> #info Request for resources: Bugyou - sayan
18:05:57 <nirik> #link http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/GQVDJTL6RLMEDE75YQKKMQESWLUTNMLU/
18:05:59 <nirik> #info Some new versions of sigul have landed with some improvements - kevin
18:06:01 <nirik> #info sigul stg is created, but needs to have the sigul part and keys setup yet - kevin
18:06:03 <nirik> #info Some more ansible 2.0 testing, master playbook finishes ok now and in reasonable time - kevin
18:06:05 <nirik> #info New EPEL ppc builders are up and on the right vlans - kevin/peter
18:06:07 <nirik> #info [release] bodhi: 2.1.4 - lmacken
18:06:09 <nirik> #link https://lists.fedoraproject.org/archives/list/infrastructure%40lists.fedoraproject.org/thread/QAO6HRU6GAJKQK7Y4HB42AYVT4J26Z3T/
18:06:12 <nirik> #info taskotron production upgraded to f22, now emitting fedmsgs - tflink
18:06:14 <nirik> #info openqa stg and prod successfully deployed - all firewall issues resolved - tflink
18:06:16 <nirik> anything anyone woud like to add or expand on there?
18:06:51 <nirik> #topic Technical Debt Fighting Week - threebean
18:06:59 <nirik> threebean: you wanted to discuss this one?
18:08:13 * pingou late
18:09:08 * danofsatx is hiding in the back
18:10:04 <nirik> we can come back around if threebean is not available. ;)
18:10:50 <pingou> wanna me to try to fill in?
18:11:01 <puiterwijk> pingou: I'd say sure.
18:11:18 <pingou> so technical debts are all these points where we said
18:11:30 <pingou> I'm leaving it this way, it's not optimized/good but it's works
18:11:35 <pingou> I'll get back to it later
18:11:39 <pingou> and we never do :)
18:11:45 <nirik> right.
18:11:48 <pingou> so proposal here
18:12:10 <pingou> is to spend a week or so fighting these debts in all our apps
18:12:14 <nirik> I think we were talking about the first week in jan when everone is back... ? or later?
18:12:27 <pingou> that is the current planning yes
18:12:57 <pingou> there are a few ways we can approach it
18:13:33 * nirik suspects he could work on technical debt in ansible playbooks. Theres a lot in there if we look I am sure.
18:13:48 <pingou> described on ralph's email: https://lists.fedoraproject.org/archives/list/infrastructure%40lists.fedoraproject.org/thread/PZWDQ5WFHHD6O2W5ZXDE2SL5LAYXJVXA/
18:14:07 <pingou> this debt might be as simple as documentation, or even just docstring
18:14:09 <threebean> oh, sorry.  back now!
18:14:22 <pingou> so the question is: do we want to tackle a few projects and fix them all
18:14:32 <pingou> or tackle a few debts and try to fix all the projects
18:14:45 <pingou> there are pros and cons to both approach
18:14:55 <pingou> so more opinions welcome :)
18:14:59 <puiterwijk> Well, I would say we first look at a smaller number of projects, and try to get those sorted out.
18:14:59 * threebean nods
18:15:20 <pingou> the idea being: we won't fix everything anyway in one week
18:15:31 <puiterwijk> Since that would mean there's a lot of eyes on those projects' codebases, which would point out other points of technical debt as well
18:15:35 <pingou> so we can fix few projects/debts this week, then plan the next week and so on
18:16:12 <pingou> puiterwijk: a cons I see about too many eyes is people stepping on each others' toe
18:16:17 <puiterwijk> Also, doing releases and such is easier and less overhead if we select a few projects for every week, instead of having to release about every project in that week
18:16:28 <threebean> puiterwijk: good point
18:16:42 <threebean> does anyone have any particular debts or projects they want to see addressed?  perhaps that can help us narrow in.
18:16:42 <nirik> well, we could also wait on releases...
18:16:50 <nirik> until there's other real need to do the release.
18:16:52 <pingou> puiterwijk: well, depending on the changes made, might not be release-worth
18:16:54 <nirik> but then it might languish
18:16:58 <puiterwijk> And if we decide we do a few kinds of tech debt every week across all projects, when are we going to say "Go for release"?
18:17:26 <pingou> I don't think we thought to do this every week
18:17:35 <pingou> but more like a week every month, or every 2 months
18:17:43 <puiterwijk> nirik: right. But then we might have some patches that clear up some text or errors in the logs that sit dormant for a month or so, and when you look for that string upstream you won't find it anymore
18:17:50 <pingou> or whatever rythm we're confortable with
18:17:53 <nirik> sure. its a balance.
18:18:12 <puiterwijk> Note: my opinion is not a very strong one, just a feeling
18:18:19 <nirik> I'd like to work on ansible playbooks/roles/docs... but not sure that really fits too well in this effort. This seems more apps...
18:18:33 <threebean> no, I think it does.
18:18:33 <pingou> nirik: I think it fits perfectly
18:18:35 <nirik> can we identify what apps might have more debt to work on ?
18:18:57 <pingou> and can be tackle from different angles, so ansible is quite a good one I think
18:19:11 <puiterwijk> And regarding "stepping on toes", you could just coordinate, at least to a certain extend, and tell on teh channel what kind of problem or what piece of the code you're tackling
18:19:16 <threebean> so - one debt I want to take on is code duplication.  a lot of our flask apps have some boilerplate that appears again and again.  it might be nice to consolidate that to its easier to read the "meat"
18:19:36 <threebean> our ansible playbooks have some of the same problem.  there's lots of boilerplate duplication at the top of all of our groups/ playbooks.
18:19:47 <threebean> the upgrade playbooks too, are mostly copy/paste with a few minor changes.
18:19:57 <pingou> a fedora-flask extension? :]
18:20:02 <threebean> could use includes to make that all more maintainable.
18:20:20 <puiterwijk> I would say that that would go perfectly in python-fedora.
18:20:21 <threebean> pingou: even before we break it out into a separate module, we could try it out isolated to a few apps and see how it goes.  ;)
18:20:23 <nirik> yeah, I did some work on that with the virt stuff
18:20:28 <pingou> threebean: sure :)
18:20:28 <nirik> but more woud be good.
18:20:43 <threebean> python-fedora needs work too.  it contains an old pkgdb1 client that hasn't been usable in over a year.
18:20:52 <threebean> nirik: we could tag team it :)
18:21:41 <threebean> I guess, let's start a wiki page with a list of things to consider.. and then revisit how we're going to narrow stuff down as we get closer?
18:21:43 <pingou> python-fedora clearly needs some love/work
18:22:03 <pingou> pkgdb1, fas2, flask_fas_openid that puiterwijk wanted to port to flask-openid
18:22:04 <nirik> sure. +1 wiki page brainstorming
18:22:13 * pingou has some more patches for flask_fas_openid
18:22:16 * threebean will create
18:22:17 <threebean> https://fedoraproject.org/wiki/Infrastructure/Debt
18:22:20 <nirik> I'm a bit worried if we try the first week everyone is back people will be sidetracked with catching up on things.
18:22:29 <puiterwijk> pingou: oh? Do send :)
18:22:51 <puiterwijk> pingou: since if you have important things, I would like to merge that with flask-openid
18:22:52 <threebean> nirik: eh, that's possible, yeah.  I guess my thoguht was that nobody would already be in the middle of something.
18:23:13 <pingou> puiterwijk: I just need to diff the version bundled in pagure
18:23:19 <puiterwijk> Okay
18:23:29 <pingou> nirik: true, maybe from Tuesday to Friday then?
18:23:38 <pingou> giving us a day to catch up on emails and things?
18:23:48 <nirik> threebean: true... and people aren't supposed to send emails or things over the break anyhow. ;)
18:23:54 <Southern_Gentlem> threebean, University folk use that  week for upgrades as well
18:24:08 <nirik> tue-friday sounds doable.
18:24:30 <threebean> +1
18:24:57 <nirik> lets plan for it. ;)
18:25:09 <odin2016> works for me. should able to find things....
18:25:11 <nirik> we can discuss more on list or in the next 2 meetings before break. ;)
18:25:23 <nirik> odin2016: yeah, there should be lots of things. ;)
18:25:47 <odin2016> nirik: good
18:25:54 <pingou> nirik: and things aren't supposed to break during the break... right? :)
18:26:05 <nirik> pingou: right. sure. you betcha.
18:26:09 <pingou> ^^
18:26:15 <nirik> anyhow, anything else on this? or shall we move on?
18:26:29 * puiterwijk wonders if we'll have a y2k16 bug ^^
18:26:48 <nirik> ha
18:27:17 <puiterwijk> I'm not excluding the idea, given that we had a month 12 issue recently :)
18:27:36 <nirik> software: always exciting. ;)
18:28:32 <nirik> #info planning on tech debt week 2016-01-05 to 2016-01-08. See https://fedoraproject.org/wiki/Infrastructure/Debt/
18:28:34 <nirik> #topic ntp servers - kevin
18:28:45 <nirik> so, I wanted to brainstorm a bit on ntp servers.
18:29:01 <nirik> We had set a list of them up on basically all machines a while back and life was fine.
18:29:34 <nirik> But it turns out that in phx2, the ntp servers that the firewall will let us talk to is only the ones the rhel ntp pool.
18:29:40 <nirik> and those change from time to time.
18:29:51 <nirik> which is anoying.
18:30:19 <nirik> So, we could just use the pool names in our ntp.conf, but that doesn't work for builders as they have output rules to block all but specific ones.
18:30:32 <nirik> or we could just setup our own ntp servers somewhere and use those.
18:31:24 <nirik> If we do setup our own, should they be new hosts? or should we stick them on bastion01/02 or something? and what do we do about non phx2 hosts?
18:31:53 <puiterwijk> I think that we should make sure to use the same NTP servers in phx2 and non-phx2.
18:32:17 <nirik> ok, but then we would need to have them externally reachable somehow.
18:32:24 <puiterwijk> but having them on the VPN might be tricky
18:32:38 <puiterwijk> Yeah, since servers need a correct time to connect to the VPN server
18:32:47 <nirik> right.
18:33:08 <puiterwijk> Or let me rephrase that: they need to have the same view on time as the VPN servers
18:33:13 <nirik> we could setup some in external sites, but then we would need to get phx2 firewall to allow them
18:33:19 <puiterwijk> But I am not sure we want to host publicly visible NTP servers.
18:33:26 <nirik> right, it's a pain.
18:33:31 <puiterwijk> Would it be an idea to setup a stratum 2 NTP server?
18:33:47 <puiterwijk> So setup one or two NTP servers in phx2 that sync from a pool of well-known ones
18:33:55 <puiterwijk> and then have the public hosts sync from the same external ones
18:34:08 <nirik> right, that was one suggestion above.
18:34:15 <odin2016> would be be too rough to say, use bastion in phx2 as a point to contact what everyone else is using and either have phx2 use bastion, or have a group that use bastion as a master?
18:34:46 <nirik> it seems overkill to me to have a entire vm only for ntp server.
18:35:03 <puiterwijk> nirik: right. But explicitly make sure that the ones we setup in PHX2 sync from the same public servers as our other sites do
18:35:26 <nirik> I am not sure how we can do that without lots of fiddling
18:35:31 <nirik> all the time
18:35:36 <nirik> and I am not sure it's that important.
18:35:40 <puiterwijk> Hmm, okay
18:36:06 <puiterwijk> Well, I think it's important to make sure all our servers have the same view on time
18:36:14 <puiterwijk> That makes combining and cross-checking log files possible
18:36:26 <puiterwijk> If they're off by even half a minute, cross-checking logs would be near impossible
18:36:52 <puiterwijk> (since we have quite a few requests per second)
18:36:53 <nirik> yes, but if they are using the same ntp pool of servers, even if they aren't the same exact servers, they should be pretty close.
18:37:05 <nirik> if one of those pool servers is 30s off, I think people wouldn't have it in the pool
18:37:20 <nirik> and they all use multiple servers.
18:37:25 <puiterwijk> Well, I'm not sure if there's any syncing within a pool, but I guess you're right with that
18:37:28 <nirik> so the one thats 30s off would be ignored
18:37:35 <puiterwijk> True there
18:37:41 <nirik> or weighted heavily against.
18:38:28 <nirik> anyhow, sounds like: setup ntp servers on bastion01/02 allowing phx2 and using the pool by dns, set non phx2 servers to just use the pool, and phx2/etc to use bastion01/02
18:38:54 <nirik> sound workable?
18:39:03 <puiterwijk> Yeah, that sounds like a good setup to me.
18:39:05 <odin2016> makes sense.
18:39:18 <nirik> ok.
18:39:29 <nirik> #info setup ntp servers on bastion01/02 allowing phx2 and using the pool by dns, set non phx2 servers to just use the pool, and phx2/etc to use bastion01/02
18:39:35 <nirik> ok, on to learn about...
18:39:52 <nirik> #topic Learn about: Learn about buildbot - tflink
18:39:56 <nirik> tflink: you around?
18:40:01 <tflink> yep
18:40:31 <tflink> I've usually done stuff like this in venues where I can use pictures
18:40:43 <tflink> if something isn't making sense, please stop me and ask
18:41:02 <tflink> I suspect that most people here have heard the term "Continuous Integration" (or CI) before. The term seems to be taking on a level of fuzziness akin to "cloud" or "web-2.0" but the core concept is nonetheless valuable. Current usage points to a definition of "do some action that's probably build or test related in response to external stimuli and keep track of results over time".
18:41:18 <tflink> Honestly, the point of this isn't to debate the current meaning of the term CI or what a "proper CI system" should contain and do. The point of this is to talk more about buildbot, how it works, the kinds of things it can do and go over some of what we're using it for within Fedora.
18:41:31 <tflink> so, why buildbot?
18:41:50 <tflink> The 300 lb gorilla in the room is Jenkins - its use has become very widespread to the point where it seems to be the de-facto default for CI that isn't done as a service.
18:42:01 <tflink> There are differences in language (Jenkins is Java, Buildbot is Python), method for configuration and so on but in my mind, one the primary differences between buildbot and jenkins is that jenkins is an extensible application for running automated builds and buildbot is an automation framework.
18:42:34 <tflink> The consequences of this are that Jenkins tends to set up more quickly but isn't nearly as easy to tweak unless there's already a plugin which does exactly what you need. Buildbot, on the other hand, can take longer to configure for a given build setup at first but the configuration file is python and most customization is done through modification of that configuration file.
18:43:10 * tflink waits for a bit to make sure folks have caught up since he's pasting stuff he wrote earlier
18:43:37 <threebean> cool :)
18:43:56 <tflink> so a bit on how buildbot is set up
18:44:05 <tflink> Buildbot has a master-slave architecture where the slaves poll the master for builds and report results upon completion of assigned builds. So long as a supported version of twisted and python can run on a machine, it doesn't much matter what the parent OS is but there seems to be a bit more support for *nix and linux distros in particular.
18:44:19 * pingou starts to wonder about buildbot for pagure
18:44:31 <tflink> A common buildbot setup consists of a buildmaster, buildslaves, builders, build factories, schedulers and change sources.
18:44:42 <tflink> The buildmaster is responsible for coordination, delegation and storing logs of the results.
18:44:54 <tflink> Buildslaves are responsible for polling the buildmaster and responding to any builds it is assigned.
18:45:07 <tflink> Build factories are a method for encapsulating the actual build process: checking out code, preparing it for build, running analysis, pushing artifacts to a common point etc.. Each build factory is composed of one or more build steps which specify the actual work to be done.
18:45:45 <tflink> you can use the build steps which are part of the upstream buildbot, you can modify those steps to do what you want or you can write your own build steps from scratch
18:45:59 <nirik> whats a builder in this context? is buildslave == builder?
18:46:06 <tflink> The remaining components i listed are used to pull everything together.
18:46:18 <tflink> nirik: I'm about to get to that :)
18:46:26 * nirik waits. ;)
18:46:30 <tflink> Builders associate buildslaves and build factories. If you have a python project, you might want to run your builds on f22, f23 and f24. In this case, you'd have at least one buildslave for each fedora you want to run builds on and create 3 different builders which reuse the same build factory: fooproject-f22 (fooproject-factory + f22-slave), fooproject-f23 (fooproject-factory + f23-slave), fooproject-f24 (fooproject-factory + f24-slave).
18:46:59 <tflink> The builder-buildslave relationship is not 1:1, if you have another barproject-factory, you can reuse the same buildslaves for barproject-f22, barproject-f23 and barproject-f24 builders. You can even take this farther and have slaves with different versions of a critical dependency, different databases, different versions of python or any other variation that you want to see builds done against.
18:47:20 <tflink> Buildbot's meta-buildbot is a good example of an insteance with a large set of builders: http://buildbot.buildbot.net/waterfall
18:47:51 <tflink> A change source is pretty much what it sounds like - it's the part of buildbot which initiates the build process. Change sources can be daemons polling repos for changes, listeners for SCM hooks, mail listeners or something custom - the common thread is that when an expected action occurs, the change source emits a signal which can be used to start off a build.
18:48:13 <tflink> Schedulers connect change sources with builders to control which change signals trigger which builders.
18:48:35 <tflink> any questions so far?
18:48:48 * tflink realizes that was a bit too much in the "wall-o-text" direction
18:49:00 * nirik is still digesting, but go on. ;)
18:49:08 <pingou> tflink: you can run 1 buildbot for multiple projects right?
18:49:28 <tflink> pingou: yeah, you can
18:49:42 <pingou> tflink: from the UI as well or only with text?
18:50:02 <tflink> in practice, you can start hitting limitations in how the buildmaster displays output, though
18:50:13 <tflink> pingou: you mean see them or add them from web ui?
18:50:16 <pingou> yes
18:50:20 <pingou> (like jenkins)
18:51:17 <tflink> at the moment, all configuration is file-based
18:51:35 <tflink> from what I understand, upstream isn't against adding web configuration but it's not a priority right now
18:51:51 <pingou> ok, thanks
18:52:00 <tflink> both should be somewhat easier once 0.9 is released (more on that later)
18:52:21 * tflink moves on since there's only 8 minutes left
18:52:33 <tflink> Let's go through a relatively simple example process for a build, using the previous "story" of fooproject which we want to build on f22, f23 and f24.
18:52:51 <tflink> 1) we push a change to fooproject's canonical git repo
18:53:00 <tflink> 2) the git poller on buildbot detects the change and waits a configurable amount of time before emitting a change
18:53:01 <tflink> 3) schedulers route the change to appropriate builders (fooproject-f22, fooproject-f23, fooproject-f24)
18:53:01 <tflink> 4) Each builder schedules a build using its build factory on one of the buildslaves associated with it
18:53:01 <tflink> 5) Each affected buildslave (f22, f23, f24 in this case) goes through the actions in the build factory specified by the scheduler
18:53:01 <tflink> 6) the buildmaster stores logs and the result of each step of the build
18:53:02 <tflink> 7) rinse, repeat as needed
18:53:52 <tflink> Any questions?
18:54:09 <tflink> Now that we've gone through a bunch of how buildbot works, I'll talk a bit about how buildbot is used with in Fedora.
18:54:17 <tflink> Buildbot is a core component of Taskotron, responsible for delegating tasks to clients and getting the individual tasks executed.
18:54:49 <tflink> The QA tools folks are starting to use buildbot for CI on their tools.
18:54:58 <tflink> The docs folks are looking to use buildbot to rebuild and push new documentation builds as they become available
18:55:27 <tflink> those are the uses I'm aware of, anyways
18:56:00 <tflink> if there aren't any questions, I'll talk a bit about buildbot 0.9
18:56:11 <tflink> The current stable version of buildbot hasn't changed a heck of a lot in recent memory, it's still very similar to the versions that I was using almost 10 years ago. However, there is a pretty big change in the works: buildbot 0.9
18:56:30 <tflink> Currently in beta (0.9b5 right now), 0.9 represents a big change and in my opinion, a big step forwards. One of the biggest changes is to decouple the user interface from the buildmaster. In 0.8, the master renders and serves the entire html interface in addition to coordinating and keeping record of buildslaves and builds.
18:56:50 <tflink> With 0.9, the buildmaster is a rest-ful service and the frontend is a separate service which uses the buildmaster as a datasource. Not only does this take some load off of the buildmaster but it opens the door for much more customization - you can build custom frontends without forking the entire project and don't have to worry so much about whether or not your use case can be shoehorned into the existing display paradigms used in the
18:56:50 <tflink> buildmaster.
18:57:02 <tflink> well, rest-ful-ish
18:57:24 <tflink> There are other changes to how the buildsteps work, how multiple masters communicate in a multi-master setup, how the buildslaves communicate with the master and so on but the decoupling of web frontend from the buildmaster is the biggest reason why I'm looking forward to being on 0.9.
18:57:35 <nirik> cool.
18:57:42 <tflink> randomuser and I have been working on getting the 0.9 betas packaged but that's still an ongoing task
18:58:04 <tflink> that's all I had prepared - any questions?
18:58:17 <nirik> is the thought to get 0.9 in epel7? or keep it out for now?
18:58:39 <tflink> for the moment, I'm focused on getting it to work in fedora
18:58:54 <nirik> fair enough
18:59:08 <tflink> I haven't even looked into whether all the deps needed by 0.9 are in or can go into epel7
18:59:18 <nirik> one step at a time. ;)
18:59:23 <nirik> is it python2? or 3?
18:59:24 <tflink> exactly :)
18:59:35 <tflink> as python3 as twisted is, afaik
19:00:00 <pingou> can it build builders on the fly ?
19:00:03 <tflink> it can do both, rather
19:00:17 <pingou> like on-demand builder in a cloud
19:00:18 <tflink> but I do believe that python2 is the primary supported platform right now
19:00:46 <nirik> alright. Thanks for the info tflink!
19:00:53 <tflink> pingou: if you're asking about using openstack to supply buildslaves, yes. the functionaltiy could use some love but it is there
19:01:16 <pingou> cool
19:01:23 <tflink> np, hope it was useful
19:01:32 * tflink is happy to answer questions or help with buildbot stuff
19:01:36 * nirik nods. will ponder on other uses. ;)
19:01:42 <nirik> #topic Open Floor
19:01:44 <pingou> certainly interesting :)
19:02:01 <pingou> I started a py3 in our infra discussion on the list, feel free to share your thoughts :)
19:02:02 <aldapetr> tflink: thanks for info
19:02:06 <nirik> Would anyone like to sign up to teach about other items upcoming? we have gone through our existing scheduled ones...
19:02:52 <nirik> Or are there any things people would in particular like to learn about/
19:03:02 <nirik> or does anyone have any items for open floor?
19:03:18 <puiterwijk> I have one small thing for open floor
19:04:04 <puiterwijk> Just a small reminder that OpenStack transient instances will not be restarted after OpenStack upgrade windows. If you want them to remain for longer than that, please ping me (until I wrote down how) on how to move them to a more persistent tenant
19:05:14 <nirik> ok, if nothing else will close out in a minute...
19:05:31 <pingou> pagure 1.0 should be out before Christmas :)
19:06:06 <puiterwijk> pingou: cool. Is the release name going to be "Christmas Present"? :)
19:06:18 <pingou> puiterwijk: "surprise surprise"
19:06:48 <puiterwijk> Works for me :)
19:06:51 <nirik> ha
19:07:09 <nirik> ok, thanks for coming everyone. Do continue in #fedora-admin, #fedora-noc, and #fedora-apps.
19:07:11 <nirik> #endmeeting