18:00:18 <nirik> #startmeeting Infrastructure (2014-09-04)
18:00:18 <zodbot> Meeting started Thu Sep  4 18:00:18 2014 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:18 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:18 <nirik> #meetingname infrastructure
18:00:18 <nirik> #topic welcome
18:00:18 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
18:00:18 <zodbot> The meeting name has been set to 'infrastructure'
18:00:18 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean
18:00:51 * threebean is here
18:01:06 * pingou 
18:01:14 * relrod here
18:01:16 <janeznemanic> hi
18:01:30 <bochecha> hi there
18:01:41 <dgilmore> hola
18:02:16 <akshays> hello
18:02:21 * mirek-hm is here
18:02:50 <nirik> hello everyone. ;)
18:03:02 <nirik> #topic New folks introductions and Apprentice tasks
18:03:16 <nirik> any new folks like to introduce themselves?
18:03:26 <nirik> or apprentices with questions or comments?
18:04:21 * michel_slm here
18:04:26 <akshays> Hi, i am new here. I am a sys admin.'
18:04:58 <nirik> akshays: welcome.
18:05:32 * oddshocks pops in
18:05:45 * danielbruno here
18:06:29 * roshi listens
18:06:32 <nirik> #topic Freeze reminder
18:06:43 <nirik> Just a reminder we are in freeze until f21 alpha goes out...
18:06:57 <nirik> see the note on the list for more information.
18:07:05 <nirik> #topic Applications status / discussion
18:07:15 <nirik> any application news/status?
18:07:20 <pingou> http://209.132.184.188/spechub/
18:07:24 <threebean> oo
18:07:29 <pingou> based on progit, made for pkgs.fp.o
18:07:46 <pingou> would integrate quite easily with gitolite2
18:08:02 <pingou> we'll have to see when we figure out how to handle gitolite3 :)
18:08:04 <nirik> yeah, I still need to poke around on it.
18:08:45 <pingou> basically, for the integration w/ gitolite2, we can just call http://209.132.184.188/spechub/admin/gitolite/conf and embed the output in the conf file with our cron task
18:08:47 <nirik> this (or something like it) might meet the needs mirek-hm was looking for for copr dist-git perhaps.
18:08:48 <mirek-hm> we (as copr team) started thinking about dist-git for copr, I sent email to fedora-devel today, so please respond and discuss there
18:08:56 <pingou> oh, and it works fine on the 19,000+ git repos :)
18:09:15 <nirik> mirek-hm: yeah, I wonder if something like this might help that case. ;)
18:09:18 <threebean> pingou: looks neat
18:09:20 <nirik> but we will have to discuss.
18:09:31 <pingou> threebean: thanks ;-)
18:09:38 <tflink> we've been finding small issues/annoyances with taskotron in stg and fixing them as we go - no more pto for a while so taskotron production should be do-able before beta
18:09:44 <mirek-hm> pingou: but that is "just" fronted to dist-git, isn't it?
18:10:01 <pingou> mirek-hm: it's a front-end to our git repos, allows forking and merging
18:10:28 <pingou> well, I need to work with threebean for the auth part that requires caching the acls from pkgdb until they change
18:10:52 <mirek-hm> ok, that would be nice one we choose how to do our dist-git, will check that.
18:10:55 <nirik> pingou: forks and commits to those folks are stored where? in git with the project forked? or ?
18:11:20 <nirik> tflink: cool. ;) so, after alpha we can nuke autoqa and enable prod?
18:11:24 <pingou> nirik: in a new git repo under the forks/ folder
18:11:34 <mirek-hm> pingou: https://fedorahosted.org/spechub/ does not exist (taken from footer)
18:11:43 <pingou> mirek-hm: request pending in trac ;-)
18:11:48 <mirek-hm> ok
18:11:50 <nirik> mirek-hm: I haven't processed his request to add it yet. ;)
18:11:51 <pingou> mirek-hm: code is in github ftm
18:12:00 <threebean> tflink: any chance there will be a new release of resultsdb before you go to prod?
18:12:09 <tflink> nirik: that's the plan
18:12:17 <tflink> threebean: are there unreleased features?
18:12:20 * threebean nods
18:12:23 <threebean> unreleased bugfix
18:12:24 <nirik> pingou: so, you could fork say kernel, then commit whatever to it and others can clone that,e tc?
18:12:31 <tflink> I thought that the jsonp stuff was already part of stg
18:12:59 <pingou> nirik: we don't allow forking a fork, but you can clone locally a fork sure
18:13:02 <tflink> threebean: I'll look into that after the meeting, if we have unreleased fixes, I'll do a release
18:13:12 <threebean> tflink: yeah, this one in particular -> https://bitbucket.org/fedoraqa/resultsdb/commits/72ce429c1c68073780178566163927f75cdaec94
18:13:38 <nirik> pingou: ok.
18:14:47 <nirik> alright. Any other application news?
18:14:54 <pingou> oh
18:14:59 * danofsatx is here, but muy busy
18:15:02 <tflink> threebean: I think that's already been released - should be in dev and stg
18:15:04 <pingou> I'm loading datagrepper's data into mongodb
18:15:21 <lmacken> I upgraded bodhi in production this week with changes for EPEL-7
18:15:23 <pingou> on 209.132.184.235
18:15:42 <pingou> db.messages.count() : 92541 -- it's slow to load
18:15:59 <threebean> clarification -> slow to populate, right?
18:16:08 <nirik> #info http://209.132.184.188/spechub/ under development, feedback welcome (dist-git frontend)
18:16:25 <nirik> #info taskotron to hit prod after alpha hopefully and replace autoqa once and for all
18:16:28 <pingou> threebean: yes
18:16:33 <pingou> arg, it failed :(
18:16:34 <nirik> #info bodhi1 updates this week to handle epel7
18:17:10 <threebean> tflink: not sure.  I'll send a link in #fedora-apps
18:18:41 <threebean> pingou: yeah, it will be interesting to see if there's any significant difference in query speed when mongo and postgres have the same number of records/documents.
18:19:17 <nirik> #topic Sysadmin status / discussion
18:19:37 <nirik> we have had several mirrormanager blowups this week sadly.
18:19:38 <pingou> threebean: looking forward that :)
18:19:56 <mirek-hm> I would like to urge https://fedorahosted.org/fedora-infrastructure/ticket/4488 (/me is looking at puiterwijk)
18:20:02 <nirik> One seems to have been due to bad data in the db causing the update directory list to fail.
18:20:26 <pingou> a foreign key not enforced
18:20:29 <nirik> mirek-hm: yeah, not sure where puiterwijk is. ;( I will do that myself after the meeting.
18:20:32 <pingou> and a delete not propagate
18:21:13 <nirik> anyhow, we will keep things going along on it until we replace it. ;)
18:21:39 <nirik> we have also been having some vpn saturation issues...
18:22:06 <nirik> It's possible these are due to high fedmsg busgateway traffic. So, we moved that to not go over the vpn...
18:22:23 <nirik> but that won't entirely take effect until clients restart
18:22:54 <mirek-hm> nirik: and regards https://fedorahosted.org/fedora-infrastructure/ticket/4514 - nirik do you think 256 IPs per tenant is enough? I would rather go with 172.N.x.x Class B networks for every tenant. E.g. 172.1.x.x for copr, 172.2.x.x for permanent, 172.3.x.x for fooo....
18:23:07 <threebean> I'm preparing a freeze break request to push some of that traffic back to the proxies which should help.
18:23:35 <nirik> mirek-hm: well, if you like, sure. I really don't think we are going to have more than 256 per client, but I guess I could be wrong. Use Class B if you like...
18:24:23 <mirek-hm> that give us 16k networks and 65k IP per each network, that is more then enough in both directions
18:25:24 <nirik> sure, works for me
18:26:13 <nirik> #topic Nagios recap
18:26:26 <nirik> .tiny https://admin.fedoraproject.org/nagios/cgi-bin//summary.cgi?report=1&displaytype=3&timeperiod=last7days&smon=9&sday=1&syear=2014&shour=0&smin=0&ssec=0&emon=9&eday=4&eyear=2014&ehour=24&emin=0&esec=0&hostgroup=all&servicegroup=all&host=all&alerttypes=3&statetypes=3&hoststates=7&servicestates=120&limit=25
18:26:27 <zodbot> nirik: http://tinyurl.com/l3vjae8
18:26:34 <nirik> pretty much all of those are due to the vpn thing.
18:27:07 <nirik> #topic Upcoming Tasks/Items
18:27:07 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
18:27:20 <nirik> anyone have upcoming items they would like to note or schedule?
18:28:13 <pingou> threebean and I prepare an announce about FMN
18:28:21 <nirik> cool.
18:28:30 <nirik> just more widespread? or ?
18:28:35 <pingou> of the 1566 packagers we have in FAS, 1453 are not using FMN yet
18:28:50 <pingou> so we want to create their account automatically
18:29:06 <pingou> so we don't have a clear date for the change yet
18:29:14 <nirik> ok
18:29:15 <pingou> but I'd hope for early october
18:29:16 <mirek-hm> FMN?
18:29:25 <pingou> mirek-hm: https://apps.fedoraproject.org/notifications/
18:29:44 <nirik> basically a way to be notified as you like when fedmsg's happen.
18:29:55 <nirik> instead of always emails
18:30:30 <threebean> you can use it to get emails about coprs, for instance.
18:31:17 <mirek-hm> hmm can someone add it on list of trusted apps for fedoauth?
18:31:30 <nirik> I thought we already did?
18:31:46 <mirek-hm> It just asked me to review the login
18:31:54 <nirik> if not, we can... (althought we are in freeze)
18:32:09 <mirek-hm> true, it can wait
18:32:43 <nirik> anyhow, it's a handy setup. ;)
18:32:49 <nirik> #topic Open Floor
18:32:55 <nirik> ok, any items for open floor?
18:33:22 * tflink is curious if there has been any status change on the new app server and qa host
18:33:49 <nirik> tflink: qa09/virthost-comm03?
18:33:52 <tflink> yep
18:34:02 <nirik> smooge has been working to bring them up.
18:34:19 <nirik> I am not sure what the current status was. I know they are on the net and all, just need installing (which he was doign)
18:34:32 * tflink just wants to get some stuff moved over to the new hosts before warranties start expiring
18:34:43 <nirik> yeah, understandable.
18:34:54 <nirik> will poke him when he's around
18:35:04 <bochecha> nirik, what's the next step of moving distgit to ansible? I kind of lost track after my patches were merged (I see you fixed some errors I made, then it seems we're halfway through with moving to gitolite3?)
18:35:08 <tflink> we should be getting 2 more hosts in q2 and q3, still in process
18:35:45 <pingou> bochecha: nirik I can help until next week thursday if we want to move closer at gitolite3
18:36:10 <nirik> bochecha: so yeah, pkgs01.stg is in ansible now. It's rhel7. It has synced data on it. However, we have no gitolite2 in rhel7, so we wanted to look at moving to 3, but it's apparently a big jump. pingou looked into it, but I haven't had time yet.
18:36:30 <nirik> bochecha: also, cgit isn't working right, but thats likely seliux or something minor
18:36:49 <pingou> nirik: we don't need no cgit, we have spechub :-p
18:37:09 * pingou ducks
18:37:13 <nirik> pingou: :)
18:37:48 <bochecha> pingou, so what's missing to move to gitolite3?
18:38:12 <smooge> ugh sorry
18:38:28 <tflink> unless there's an issue I don't know of, moving to gitolite3 in the middle of a release that's already chaotic sounds like a bad idea
18:38:34 <nirik> hey smooge. tflink was asking about qa09/virthost-comm03.... any update?
18:38:34 <pingou> bochecha: basically in gitolite2 we have 1 conf file in /etc
18:38:46 <bochecha> tflink, right, we're only doing this on staging, not production
18:38:55 <pingou> bochecha: that conf file defines where the repos are located /srv/...
18:38:57 <tflink> ok, nvm then :)
18:39:08 <pingou> bochecha: in gitolite3 repos *must* be in ~/repositories
18:39:14 <nirik> tflink: well, mostly because we dont have the old version on epel7... but of course we can punt and add it
18:39:23 <smooge> The boxes got to mgmt working. I have to start setup and installation. I expect by Monday
18:39:38 <pingou> so since we use gitolite in such a way that each user has its own account on the machine, it means that we need each user to have ~/repositories
18:39:47 <pingou> oh and ~/.gitolite.rc btw
18:39:49 <tflink> nirik: I thought that the idea was to move production soon, missed the bit about stg
18:40:00 <bochecha> ah right
18:40:10 <pingou> so either we mess with $HOME
18:40:14 <bochecha> is there a reason why we don't use the recommended gitolite setup? (only one user)
18:40:24 <pingou> or we find a way to move to that ^
18:40:35 <nirik> tflink: well, I would like to move prod (but not in freeze of course). We are mostly just exploring the options in stg right now.
18:40:50 <pingou> which I expect means: 1) a new fedpkg version 2) break all the clones of all the packages out there
18:41:05 <nirik> yeah, those don't seem... feasable
18:41:06 <bochecha> fwiw, at my previous dayjob I was using the "one user only" recommended setup of gitolite for my internal distgit, it was working well
18:41:17 <bochecha> pingou, ah right, the fetch/push urls with the user in them :-/
18:41:22 <pingou> yup
18:41:33 <pingou> and that's where I stopped :)
18:41:39 <bochecha> well, a new fedpkg could rewrite those urls
18:41:45 <pingou> https://groups.google.com/forum/#!msg/gitolite/iSfd4b-UbhM/MDU2aP0DvCgJ
18:41:46 <pingou> ftr
18:42:04 <pingou> they do speak about us in there
18:42:29 <pingou> and they seem to advice to mess with $HOME
18:42:44 <nirik> yeah, so sounding like we should just get v2 branched for now at least
18:43:11 <bochecha> yeha, maybe gitolite2 (at least in th einfra repo) for el7 would make things simpler for now
18:43:12 <pingou> bochecha: which means, breaking all the old fedpkg around :/
18:43:30 <bochecha> pingou, well, yeah, forcing an upgrade
18:43:34 <pingou> I wanted to check how hard it would be to build gitolite2 on koji, but I haven't checked it yet
18:43:43 <bochecha> (which is coming anyway, when we move to sha512 for the lookaside hashes)
18:44:02 <bochecha> but yeah, doing it twice is certainly not nice
18:44:06 <pingou> nirik: ^ if we force a fedpkg upgrade already, that might be a good time to do that
18:44:19 <bochecha> pingou, I don't htink we can do both at the same time
18:44:27 <pingou> that == move to the one gitolite user
18:44:30 <pingou> bochecha: why not?
18:44:45 <dgilmore> what does move to one gitolite user actually mean?
18:45:11 <bochecha> dgilmore, git clone gitolite@pkgs.fedoraproject.org:foobar
18:45:11 <pingou> ssh://pingou@pkgs.fedoraproject.org/fedocal -> ssh://gitolite@pkgs.fedoraproject.org/fedocal
18:45:21 <dgilmore> bochecha: yeah no
18:45:31 <dgilmore> bochecha: never will that be okay
18:45:43 * pingou needs more info
18:45:43 <dgilmore> we want to know who commited the changes
18:45:52 <bochecha> we still do with the gitolite@ scheme
18:45:59 <pingou> dgilmore: we do know, gitolite handles that
18:46:03 <nirik> its just at a gitolite layer
18:46:08 <dgilmore> ewww
18:46:11 <dgilmore> no
18:46:12 <bochecha> all users are 'gitolite' on the server, but it's still mapped to the user's private key
18:46:26 <bochecha> dgilmore, it's the upstream recommended way, though
18:46:33 <dgilmore> bochecha: doesnt make it right
18:46:38 <bochecha> sure
18:46:42 <pingou> s/recommended/&designed/
18:46:52 <bochecha> it's just making it hard/troublesome to do differently
18:47:05 <dgilmore> then we should look at other options
18:47:12 <dgilmore> or work with upstream to make it sane
18:47:38 <pingou> dgilmore: they explicitely removed what allowed to do things the way we did
18:47:41 <pingou> https://groups.google.com/forum/#!msg/gitolite/iSfd4b-UbhM/MDU2aP0DvCgJ
18:47:47 <bochecha> knowing we were doing it this way
18:47:51 <dgilmore> pingou: then we find a replacement
18:48:03 <bochecha> pingou, you handle the acls directly in spechub, ok? :)
18:48:13 * pingou jumps
18:48:18 <bochecha> \o/
18:48:28 <pingou> no, ther other jump ;-)
18:49:09 <nirik> yeah, I think v2 for now and look for a longer term solution seems like what we will have to do.
18:49:10 <bochecha> oh /o\
18:49:25 <pingou> HOME=/srv/... might work
18:49:54 <pingou> I mean, we need that in a couple of places (in the ~/.ssh/authorized_keys) that we already handle
18:50:11 <nirik> pingou: feel free to play with it and see if you can get it to work. ;)
18:50:19 <nirik> any other open floor items? :)
18:50:35 <mirek-hm> dgilmore: why you do not like those user tracking in gitolite layer #justasking
18:50:36 <pingou> nirik: for some reason I didn't feel like playing alone with my $HOME just in case I can't login anymore :)
18:52:05 <dgilmore> mirek-hm: a clearer seperation, we already have the logging of ssh logins, we would need to setup some additional logging to know. also the authorized_users file would be massive, i can see that effecting performance
18:52:38 <bochecha> pingou, play with mine! I don't have shell access to that server ;)
18:52:45 <pingou> bochecha: let's do that :)
18:53:47 <nirik> ok, if nothing else will close out in a minute...
18:54:24 * mirek-hm would like to benchmark affect of size of authorized_users on performance
18:54:54 <dgilmore> right now there would be 1481 keys in the authorized_users file
18:55:42 <nirik> on the other hand it would mean we wouldn't need to give packagers accounts. ;)
18:56:23 <pingou> yes
18:56:25 <nirik> anyhow... we will keep looking into it. ;)
18:56:30 <nirik> thanks for coming everyone!
18:56:34 <nirik> #endmeeting