17:59:45 <smooge> #startmeeting Infrastructure (2017-05-04)
17:59:45 <zodbot> Meeting started Thu May  4 17:59:45 2017 UTC.  The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:59:45 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
17:59:45 <zodbot> The meeting name has been set to 'infrastructure_(2017-05-04)'
17:59:45 <smooge> #meetingname infrastructure
17:59:45 <zodbot> The meeting name has been set to 'infrastructure'
17:59:45 <smooge> #topic aloha
17:59:45 <smooge> #chair smooge relrod nirik abadger1999 dgilmore threebean pingou puiterwijk pbrobinson
17:59:45 <zodbot> Current chairs: abadger1999 dgilmore nirik pbrobinson pingou puiterwijk relrod smooge threebean
17:59:47 <clime> hi
17:59:59 * pingou around but likely not for long
18:00:02 * cyberworm54_ here
18:00:05 * threebean waves
18:00:15 <axk4545> here
18:00:22 <R3d_N0mad> hi
18:00:32 <jcline> .hello jcline
18:00:33 <zodbot> jcline: jcline 'Jeremy Cline' <jeremy@jcline.org>
18:00:57 <bowlofeggs> .hello bowlofeggs
18:00:58 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <randy@electronsweatshop.com>
18:01:15 <smooge> hello everyone
18:01:16 <nb> .hello nb
18:01:17 <zodbot> nb: nb 'Nick Bebout' <nb@nb.zone>
18:01:24 <nirik> morning
18:01:44 <smooge> #topic New folks introductions
18:01:51 <smooge> Hi do we have any new people here today?
18:02:03 <axk4545> smooge: I am new.
18:02:07 <R3d_N0mad> hi my first meeting but I joined in Feb
18:02:44 <smooge> hello to you both.
18:02:54 <cyberworm54_> hello!!
18:04:01 <axk4545> also my first ever irc meeting
18:04:23 <smooge> good to meet you all.
18:04:23 <nirik> welcome!
18:04:48 <threebean> axk4545: :)
18:05:07 <R3d_N0mad> great to finally meet all of you as well
18:07:29 <smooge> good to meet you also
18:07:31 <smooge> next up
18:07:41 <smooge> #topic announcements and information
18:07:42 <smooge> #info beta freeze will start 2017-05-17
18:07:42 <smooge> #info infra hackfest in RDU 2017-05-08 to 2017-05-12 - everyone
18:07:42 <smooge> #info Mass update/reboot cycle completed - everyone
18:07:42 <smooge> #info new nagios with templates deployed
18:07:51 <threebean> yay!
18:07:53 <smooge> we had a fun filled week here in infrastructure land
18:07:58 <nirik> indeed.
18:08:31 <nirik> with the hackfest next week a lot of folks won't be around as much as usual... so keep that in mind and file tickets or email if you can't find anyone.
18:09:01 <nb> I am rewriting /git/dns to remove built/* so everyone will have to re-clone when I am done
18:09:06 <nb> I will email the list when it finishes
18:09:29 <axk4545> nb++
18:09:46 <threebean> oh.  nb how are the contents of built/ going to be generated?
18:10:41 <puiterwijk> threebean: they have always been generated
18:10:46 <puiterwijk> but they contain dnssec signatures.
18:11:04 <puiterwijk> And that means basically the entire file gets rewritten on every small change, which makes the size grow very hard
18:11:17 * threebean nods
18:11:44 <smooge> thank you nb
18:11:46 <threebean> I guess my question is.. what is responsible for generating them now?  (we used to do it ourselves).  is it a post-receive git hook?  or a cron job? or..?
18:11:50 <smooge> my disk space thanks you also
18:11:54 <nb> threebean, will be the same way as normal, I will run do-domains when the commit finishes
18:11:59 <puiterwijk> threebean: the same do-domains is responsible for it
18:12:09 <nb> threebean, nothing changes, except removing old built/* from history
18:12:15 <puiterwijk> threebean: we are just cleaning up all the historical /build builds from the git repo
18:12:16 <threebean> oh!!! ok.  awesome :)
18:12:20 <puiterwijk> Yeah, what nb said
18:13:02 <pingou> from history but not disk then?
18:13:18 <smooge> No they will be removed from disk
18:13:27 <smooge> they never happened
18:13:40 <nirik> history... rewritten!
18:13:41 <smooge> we are never to talk about them again (until we grow to gigabytes)
18:13:46 <pingou> I guess I'm a little confused then
18:14:02 <smooge> pingou, there are 2 trees in the git repo
18:14:13 <smooge> there is /master and /built
18:14:26 <jlozadad> hello!
18:14:27 <smooge> the /built gets generated each time with the signing keys and committed
18:14:46 <smooge> so it is basically like keeping track of .o files
18:14:47 <nirik> right now the repo is 3.1GB
18:15:33 <smooge> currently we fix that by going through history and pruning out all those .o files regularly
18:16:08 <smooge> I am not sure that is the best fix, but we have usually 100,000 things to do before 'fixing DNS to be less disk wasteful' comes near to the top
18:16:33 <smooge> pingou, does that help muddy the waters any?
18:16:49 <pingou> so we keep /master and drop /built
18:16:55 <axk4545> smooge: why not set up a thing to run a cleanup after generation?
18:17:12 <nb> axk4545, it breaks existing clones when you do a cleanup
18:17:15 <nb> because it is a force push
18:17:18 <pingou> but we still rebuild /master
18:17:28 <pingou> and that's no longer going into /built?
18:17:29 <axk4545> nb: ah
18:17:38 <Skeer> Sorry Im late.
18:17:44 <smooge> pingou, no it is still going into /built
18:17:59 <nirik> we are just making the repo smaller for a while.
18:17:59 <smooge> pingou, all we are doing is basically removing 2 years of date from the history in /built
18:18:04 <pingou> smooge: so we're just cleaning the history and keeping the last version
18:18:10 <pingou> ok gotcha
18:18:14 <pingou> cool, thanks nb
18:18:16 <nirik> the repo also gets cloned to each dns server on any change
18:18:22 <smooge> you can get the history from /master changes
18:18:31 <nirik> so that starts getting slower and taking up more space over time
18:18:32 <smooge> but anything in /built is gone
18:19:41 <smooge> ok anything more on this?
18:19:59 <smooge> next up then
18:20:01 <smooge> #topic cleanups from mass update cycle - kevin
18:20:08 <smooge> nirik, you are up
18:20:24 <nirik> so yeah, this weeks mass update/reboot didn't go too great.
18:20:40 <nirik> I think partly because we haven't done one in about 3 months, so there was a lot of change piled up.
18:20:51 <nirik> There's a few process things we can change to help:
18:21:00 <nirik> * no updates pushes allowed on days we are updating things.
18:21:25 <bowlofeggs> +10
18:21:30 <nirik> There's also a few bugs we still need to fix
18:21:49 <nirik> I think puiterwijk already fixed/filed/whatever the fedimg and autocloud ones? They failed to start right after upgrade.
18:22:08 <nirik> There's also a bug with fedmsg-relay... it starts but doesn't make a socket so nagios cannot monitor it.
18:22:09 <puiterwijk> autocloud was not issue in the end, I filed fedimg, and a patch is now live
18:22:38 <nirik> There was an issue with our openvpn CRL... it was expired (I guess it never used to check it?)
18:22:52 <puiterwijk> Also, the fedmsg-datanommer package suddenly added an /etc/fedmsg.d/datanommer.py thatbroke datagrepper
18:23:26 <puiterwijk> (fedmsg-datanommer-models, that is)
18:23:36 <nirik> and of course we also had hardware croak. :) as it always does when rebooted.
18:23:41 <nb> can someone update /srv/web/infra/dns? /me does not have permission to
18:23:51 <pingou> puiterwijk: that one surprises me, did we do a release of some sort?
18:23:54 * nb will update the nameservers
18:24:03 <puiterwijk> pingou: I do not know of any. And it surprised me too
18:24:14 <puiterwijk> nb: wrong channel, but will do that
18:24:25 <nb> oops, sorry, i meant to talk in #fedora-noc
18:24:48 <nirik> I can file the relay bug... is that part of main fedmsg?
18:25:04 <jcline> I believe so
18:25:42 <nirik> oh, back to openvpn CRL... the default is that it expires every month. Do we really want to have to make it every month? or should we up that to something more reasonable like a year?
18:25:56 <axk4545> nirik: year
18:26:26 <axk4545> or maybe sync it with our mass upgrade cycle
18:26:33 <pingou> making this a January thing, or March thing (end of FY)
18:26:49 <nirik> year is fine with me.
18:27:24 <smooge> nirik, maybe it could be done like letsencrypt?
18:27:27 <puiterwijk> nirik: I have set it to a year for this generation
18:27:46 <puiterwijk> nirik: however, as I said last night, normally it gets regenerated when a new cert is generated
18:27:55 <puiterwijk> In other words: when we add a new server, we get a new CRL
18:28:02 <nirik> puiterwijk: ok, well, unless you put a new one in, I put one in that still had the 30 days.
18:28:13 <nirik> and changed to sha256
18:28:32 <puiterwijk> I had made those same changes, just not pushed them to the repo :). I just generated it and deployed to bastion01
18:28:32 <nirik> because after the outage still a bunch of machines couldn't connect.
18:28:46 * nirik can check which one is live after the meeting and we can sync git.
18:28:59 <smooge> git merge nirik
18:29:58 <smooge> ok any other items?
18:30:25 <nirik> I think thats it. I hope I didn't forget any...
18:30:39 <smooge> we can get them next mass update after beta
18:30:50 <smooge> #topic dist-git-1.0 released for epel7 - clime
18:31:02 <smooge> clime, you are up
18:31:16 <clime> Okay, I would like to say a few words about this new dist-git package
18:31:57 <clime> It basically contains: the upload.cgi script, setup_git_repo, and mkbranch script, then httpd configs and selinux
18:32:17 <nirik> nice.
18:32:21 <clime> there is a few more things like systemd service setup for git-smart-http but that's pretty much it
18:32:49 <clime> it does not contain gitolite or any pkgdb integration but they can be plugged in easily
18:33:00 <nirik> reminds me... there's a bug in upload.cgi... puiterwijk looked into it, but I don't think he solved it.... it uploads things twice
18:33:02 <axk4545> ths is what runs the lookaside?
18:33:17 <clime> oh ok, then that bug is also in the upstream package
18:33:30 <nirik> clime: likely, but I haven't checked for sure
18:33:49 <clime> It is very least common demoninator between Fedora Infra DistGit and any other DistGit
18:33:59 <clime> like copr-dist-git.
18:34:22 <clime> I would like to ask if we could adopt it into FedoraInfra
18:34:30 <puiterwijk> nirik: that is a client-side problem
18:35:26 <nirik> clime: I'm not opposed... we will have to be carefull of course to make sure we don't break anything.
18:35:38 <clime> It has configurable paths to git repos and lookaside
18:35:42 <axk4545> clime: the new package?
18:35:48 <clime> axk4545: yup
18:36:01 <clime> nirik: okay cool!
18:36:21 <clime> so I would like to make a very slow transition to it
18:36:52 <clime> with lots of testing alongside
18:36:58 <smooge> that is always good
18:37:26 <nirik> yep
18:38:30 <clime> okay, thank you. Can I talk to anyone later about details or show a small patches for a review?
18:39:14 <clime> It's really not so much work but I would like to have an extra eye if possible.
18:39:32 <smooge> I would say that an email to the list covering the changes would be a good start
18:40:01 <smooge> is this something on our side or on releng to mostly help implement?
18:40:07 <clime> Yeah, there is basically no change when compared to the current state of FedoraInfra.
18:40:34 <clime> but for sure, I can write up something
18:42:33 <clime> smooge: well, it's basically on our side to implement
18:42:42 <clime> but it's just few minor changes...
18:43:02 <smooge> ok thanks
18:43:09 <smooge> if there is nothing more on this..
18:43:13 <smooge> #topic service account for automated rebuilds? - threebean
18:43:29 <smooge> threebean, you are up
18:43:42 <threebean> #link https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/thread/YEONLKLQ2HBTCDL4OUIBYDOYL3L55JC7/
18:44:04 <threebean> just wanted to see if anyone here had any feedback on that request.
18:44:24 <threebean> the gist is that we're working on a continuous rebuild service to:
18:44:30 <nirik> I'm a little worried security wise...
18:44:36 <threebean> oh, yeah?
18:44:44 <nirik> since basically that account will be provenpackager right?
18:44:50 <threebean> as things stand, yeah.
18:44:56 <nirik> but as long as we are carefull and puiterwijk doesn't yell...
18:45:01 <threebean> hm.
18:45:05 <puiterwijk> threebean: I'm not sure I'm comfortable with the current way things are planned
18:45:14 <puiterwijk> Maybe we can discuss this next week?
18:45:14 <threebean> here's an alternative:
18:45:23 <threebean> no provenpackager for this account..
18:45:41 <threebean> but instead add some special casing to the gitolite config which gives it rights only on components in the modules/ namespace.
18:45:46 <threebean> like provenpackager-lite, for modules only.
18:46:28 <nirik> threebean: what initiates the changes?
18:46:28 <threebean> well, either way we can discuss it next week.  some of us will be together for the FAD.  maybe we can discuss it there?
18:46:49 <threebean> nirik: in this case, a human (kevin) commits to a specfile in dist-git.
18:47:18 <threebean> the system notices that on fedmsg and looks up the list of what modules have a dependency on that specfile/rpm.
18:47:51 <threebean> it then kicks off a rebuild of those modules (which necessitates doing an empty commit to those modules' yaml files in dist-git)
18:47:59 <nirik> and to push changes it will need a ssh key right?
18:48:03 * threebean nods
18:48:15 <threebean> yeah, that's it.  shouldn't need a pw.
18:48:29 <nirik> but we could lock it down to only be from the mbs machines or the like.
18:48:38 <threebean> we could, yeah.
18:48:57 <threebean> it would be nice to have stg dist-git be a little more open for devs to test.  but certainly prod.
18:49:06 <pingou> I think I like the gitolite approach
18:49:27 <pingou> pagure will be more complicated but we'll see when the time comes
18:49:53 <axk4545> pingou: you mean using it with src.fp.o?
18:49:58 * threebean nods
18:50:18 <pingou> axk4545: yup
18:50:40 <nirik> yeah, I'd say lets hash this out next week if we can... you are going to be in RDU threebean ?
18:50:47 <threebean> I'll be there!  :)
18:51:06 <smooge> OK thanks threebean
18:51:07 <nirik> oh nice. cool.
18:51:12 <smooge> it sounds like next week will be busy
18:51:13 <threebean> going to pitch a tent in the hallway outside the conference room.  :)
18:51:44 <pingou> \ó/
18:51:51 <threebean> ok.  thanks all!
18:51:56 <nirik> Good thinking... saves on hotel. ;)
18:52:19 <smooge> I want to remind people that transportation to and from the event is provided by Smoober... where riding in the back of a 20 year old truck is style
18:52:33 <smooge> ok back to the main topic
18:52:43 <smooge> #topic Apprentice Office 5 minutes
18:53:33 <smooge> Any questions from apprentices on tickets?
18:54:29 <axk4545> maybe
18:54:36 <axk4545> one sec.
18:54:38 <Skeer> Wonder if I could get some eyes on my bodhi combination template by this wekend?
18:55:29 <smooge> Skeer, what is the ticket?
18:57:36 <smooge> ok apprentices, we are coming up to the top of the hour. I am going to ask to move the questions over to #fedora-admin
18:57:45 <smooge> Thank you all for coming to the meeting
18:57:55 <Skeer> sorry smooge.. boss walked in ;)
18:57:56 <smooge> #endmeeting