19:00:46 <nirik> #startmeeting Infrastructure (2014-02-13)
19:00:46 <zodbot> Meeting started Thu Feb 13 19:00:46 2014 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:46 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:47 <nirik> #meetingname infrastructure
19:00:47 <nirik> #topic welcome y'all
19:00:47 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
19:00:47 <zodbot> The meeting name has been set to 'infrastructure'
19:00:47 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean
19:00:52 * relrod here!
19:00:54 * mariocav is here
19:01:04 * threebean is here
19:01:08 * mirek is here
19:01:30 <lmacken> me
19:01:38 * willo present
19:01:49 * webpigeon is here
19:01:52 * adimania is here
19:02:13 * puiterwijk is here
19:02:30 <nirik> cool. :) welcome everyone.
19:02:36 <nirik> #topic New folks introductions and Apprentice tasks.
19:02:44 <nirik> any new folks like to introduce themselves?
19:02:52 <nirik> or apprentices with questions or comments or ideas?
19:03:42 <janeznemanic> hi i'm present too
19:04:28 <nirik> ok
19:04:36 <nirik> #topic Applications status / discussion
19:04:50 <nirik> any application news or plans or discussion.
19:04:58 <puiterwijk> #info FedOAuth 2.0 is currently in staging, please test it and report serious issues!
19:05:00 <threebean> all kinds of good stuff happening.  maybe too much.  I can't keep track of it ;)
19:05:18 <nirik> threebean: yeah, I was just going to mention a few, and couldn't decide what to mention. ;)
19:05:19 <lmacken> I pushed out bodhi 0.9.8 to prod this week, https://lists.fedorahosted.org/pipermail/bodhi/2014-February/000751.html lots of bugfixes
19:05:34 <lmacken> on to 2.0 now :)
19:05:37 <nirik> #info bodhi 0.9.8 to prod this week, https://lists.fedorahosted.org/pipermail/bodhi/2014-February/000751.html lots of bugfixes
19:05:47 <nirik> lmacken: cool. ;)
19:05:55 <threebean> lmacken: awesome.  that opens the door to a couple new badges that are in the queue :)
19:06:07 <lmacken> threebean: yep, those should be good to go now.
19:06:21 <nirik> #info plans are afoot for us to get bugzilla fedmsg's sometime in the next 3-4months. This is really good news! :)
19:06:29 <threebean> \ó/
19:06:31 <webpigeon> puiterwijk: linky?
19:06:53 <puiterwijk> webpigeon: id.stg.fedoraproject.org
19:07:01 <webpigeon> thanks
19:07:07 <mirek> I have some updates for Copr in queue, but since I will be on holidays next week I will postpone deployment till i return
19:07:34 <nirik> mirek: sounds good. You saw that idea to have some sort of fedorahosted -> copr integration?
19:07:35 <tflink> puiterwijk: does that include persona provider support?
19:07:40 <puiterwijk> tflink: yes, it does
19:07:54 <puiterwijk> oh
19:08:02 <threebean> #info new datagrepper deployed earlier this week with "html cards" enhancements from Charul
19:08:07 <nirik> mirek: also, we can look at backups... we can have our backup server reach out to the cloud and backup stuff for copr.
19:08:11 <threebean> https://apps.fedoraproject.org/datagrepper
19:08:13 <puiterwijk> #info if no issues are found, FedOAuth 2.0 will be promoted to production on saturday
19:08:20 <mirek> nirik: not sure which one you think, there was a lot of ideas on devconf and around
19:08:24 <threebean> #info new fedora-packages deployed earlier this with, integrating that datagrepper feature
19:08:31 <threebean> https://apps.fedoraproject.org/packages
19:08:33 * pingou late
19:08:42 <mirek> nirik: I'm backing up copr into OS1 right now
19:09:01 <nirik> mirek: ah ha. cool. so, if need be you could fire it up there?
19:09:19 <pingou> fedocal 0.4.6 is in stg since today (0.4.5 was in since last week-end)
19:09:28 <mirek> nirik: but just rpms, because that is plain disk, db is on normal storage (i.e. I hope that is at least two disk in mirror)
19:09:44 <lmacken> threebean: oh excellent, I didn't know that went live already. So good.
19:09:47 * dgilmore arrives
19:09:50 <mirek> nirik: err, I'm backing up even db
19:09:53 <nirik> mirek: could do a db dump and back that up.
19:09:56 <nirik> ok, cool.
19:10:05 <mirek> yes, db dump
19:10:34 * nirik wonders if we shouldn't have some kind of timeline or list for things... we have so many apps in the air now.
19:10:56 <mirek> and yes, we have copr probe for nagios based on fedmsg.
19:10:56 <pingou> we do indeed
19:11:18 <pingou> mirek: which timeframe did you set finally?
19:11:32 <nirik> mirek: yeah, we should probibly bump up the time some. we did bump it to 3 hours, but even thats low.
19:11:56 <nirik> perhaps 6 ?
19:11:57 <mirek> i'm right now preparing really small package, which I plan to build every hour
19:12:07 <threebean> oo
19:12:09 <mirek> so every hour should come notif
19:12:18 <pingou> that's one option :)
19:12:24 <nirik> hum...
19:12:29 <nirik> I was wondering about that...
19:12:47 <nirik> could we get fedmsg to do pings? some way to submit a message thats just a 'I am alive and processing ok'/
19:12:56 <nirik> or could we never be sure it really hits all the messages it should?
19:13:12 <threebean> pingou and I talked about that some over the weekend..
19:13:26 <abadger1999> nirik: Like haproxy, does, you mean?
19:13:31 <mirek> well sometimes backend is alive, but workers are stuck, so you are not sure unless you really try to build package
19:13:48 <mirek> small package, which already has been build should take just 45 sec.
19:14:10 <nirik> abadger1999: sort of, but more application side... it emits all the messages it can/normally does, but with a 'test' or 'ping' set so it doesn't show up in the regular stream
19:14:16 <threebean> nirik: the short answer is no.  I'd be glad to talk about the longer answer later if anyone would like..
19:14:32 <nirik> fair enough. Just a thought, I trust your folks judgement.
19:15:32 <nirik> any other application side news?
19:15:38 <pingou> summershum? :)
19:15:45 <nirik> pingou: where were we on pkgdb2? waiting on new fedoauth?
19:15:47 <nirik> ah, that too.
19:16:05 <pingou> nirik: pkgdb2 waits on pkgdb-cli that waits on python-fedora that waits on fedoauth
19:16:26 <nirik> nice trail. ;)
19:16:27 <nirik> ok
19:16:29 <pingou> yup :)
19:17:01 <pingou> the svg on thisweekinfedora should be a little more readable next week
19:17:19 <pingou> (can hardly do worst atm)
19:17:43 <pingou> otherwise, for summershum, what do you guys think?
19:17:55 <nirik> could be nice. ;)
19:18:11 <mirek> what is summershum?
19:18:11 <puiterwijk> pingou: well, you don't need to wait for FedOAuth anymore, right? :)
19:18:30 <puiterwijk> at least for dev purposes
19:18:35 <nirik> mirek: a planned app to index/checksum sources.
19:18:58 <nirik> mirek: so, we could then query for 'how many copies of COPYING are in what packages' or 'how many have this md5.c file'
19:19:00 <pingou> mirek: indexes the sha1sum of every file on every package
19:19:18 <mirek> aha thanks
19:19:28 <pingou> I thought building the API for this would make a good GSoC project
19:19:45 <pingou> well, s/I/Ralph and I/
19:20:00 <nirik> (well, if we are in GSoC this year)
19:20:04 <pingou> true
19:20:31 <willo> sounds so awesome.  great stats to come from that
19:20:48 <abadger1999> might want to see if you can get a little dmalcolm time to ask about that as well -- I think he worked on something similar at one point.
19:20:49 <threebean> willo: oh, you were asking about busmon last week?
19:20:51 <nirik> it might be nice to have something like that debian source searcher too...
19:20:58 <nirik> I don't know if that could be on top of this or not.
19:21:01 <threebean> willo: I have a rewrite in progress I'd be glad to share with you if you'd like to hack on it
19:21:23 <willo> threebean: yeah, cool.  would be great
19:21:25 <pingou> nirik: sounds doable
19:21:51 <nirik> https://github.com/Debian/dcs and http://codesearch.debian.net
19:22:41 <nirik> I can see several uses for something like that...
19:22:54 <nirik> "find out of any other packages are using this thing I just fixed"
19:22:55 <pingou> nirik: this is searching within the files
19:22:59 <nirik> yeah.
19:23:06 <nirik> so, different/harder
19:23:12 <threebean> willo: cool.  I'll push it on a branch and let you know
19:23:24 <pingou> which is something that was requested on the list, but imho a little different from the current idea of summershum
19:23:33 <willo> threebean: great
19:23:40 <willo> pingou: shummershum-ng
19:23:41 <nirik> right, it might be seperate, or related, or on top, not sure.
19:23:42 <willo> :)
19:23:46 <pingou> willo: \ó/
19:24:08 <nirik> anyhow, just something to think on...
19:24:30 <nirik> even just a search of spec files/patches would be interesting... "how many packages use BuildRoot:" or whatever
19:24:51 <pingou> yeah that sounds quite nice
19:25:26 <nirik> anyhow, any other applications news ?
19:26:15 <nirik> #topic Sysadmin status / discussion
19:26:23 <nirik> so, on the sysadmin side of things...
19:26:35 <nirik> smooge is out at our datacenter doing bunches of hands on things.
19:27:01 <nirik> #info smooge on site until friday
19:27:13 <nirik> #info unbound instances moved to ansible
19:27:15 <mirek> are the ARMs in fedora cloud network already?
19:27:25 <nirik> mirek: not yet, but very soon. ;)
19:27:27 <nirik> on that...
19:27:39 <nirik> we are going to move one of our arm chassis over to the cloud network.
19:27:45 <nirik> thats 24 SOCs.
19:27:58 <threebean> #info gluster module ported to an ansible role
19:28:03 <nirik> we can then use some of them for copr and some for qa/packagers
19:28:31 <nirik> mirek: we need to decide if we want copr to install them clean each build, or just use existing installs.
19:28:52 <nirik> threebean: yes! Thank you. Although it needs a bit of tweaking to be idempotent it seems like...
19:29:11 <mirek> I will have use for 8 of them (right now), rest of them can be used for other purposes
19:29:13 <adimania> I guess unbound is ported fully as well.
19:29:31 <mirek> nirik: I would rather see reinstall before each build
19:29:32 <nirik> mirek: sounds good. I can let you know once we have it setup.
19:29:55 <nirik> mirek: ok, that just would involve ipmitool to set it to pxeboot, power on, wait for install to finish and use.
19:30:05 <dgilmore> nirik: we can script rebuilds using ipmitool
19:30:06 * mirek nod
19:30:06 <nirik> adimania: yep. ;) Thanks for that work.
19:30:15 <nirik> dgilmore: yep. ;) that was my plan.
19:30:17 <adimania> :)
19:30:25 <threebean> nirik: cool, I'll have a look at it again when I get around to tagger/packages -> ansible.
19:30:43 <pingou> we added --extra-vars="testing=True" to the updates_packages and sign_and_import playbooks
19:30:44 <nirik> janeznemanic: I still need to look at your bodhi and smtp-mm playbook work. ;) it's on my list... thanks for working on it.
19:30:50 <dgilmore> nirik: mirek came and saw me today and I let him know its underway
19:30:57 <adimania> i can pick up another module. Any easyfixes?
19:30:58 <nirik> excellent.
19:31:22 <nirik> adimania: I was just going to file one on paste/sticky-notes. will do so when I get a chance.
19:31:34 <janeznemanic> nirik: no problem
19:31:44 <adimania> nirik, great. I'll pick it up.
19:31:59 <nirik> also, I was going to file another easyfix for someone to check our phx2 dns... see what hosts are not responding to ping (and likely gone) so we can recover their ips
19:32:18 <lmacken> pingou: nice, yeah I used the testing var yesterday. Real handy :)
19:32:44 <pingou> I had a problem with the update_packages today, the yum clean all didn't perform
19:32:49 <nirik> finally, I have been updating machines (but only rebooted stg). I am going to continue that today and get everything updated... should not hopefully have any outage causing issues.
19:33:21 <nirik> #info updates being applied to all instances (but no reboots needed)
19:34:13 <nirik> On our netapp slowness... there was work tuesday night to try and upgrade it. It failed to work. There is supposed to be a new plan from netapp folks later today and likely another attempt and updating. :(
19:35:21 <nirik> lets see....
19:35:54 <nirik> I'm sure there's more stuff, but not sure what. ;)
19:36:36 <nirik> #topic Upcoming Tasks/Items
19:36:36 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
19:36:47 <nirik> anything coming up folks would like to note or schedule.
19:37:10 <threebean> We still have no ETA on a F21 release cycle?
19:37:17 <mirek> I'm next week on holidays
19:37:30 <pingou> enjoy :)
19:37:32 <threebean> mirek: great; enjoy :)
19:37:46 <nirik> threebean: aug/sept... but nothing definite.
19:38:07 <nirik> Hopefully things will come more into focus in the next few weeks as working groups submit what they want to actually deliver
19:38:19 <nirik> mirek: going somewhere fun I hope?
19:38:45 <mirek> skiing with kids - if there will be some snow
19:38:48 <willo> do we have an infrastructure freeze associated with release?
19:38:57 <nirik> mirek: fun. ;)
19:39:09 <willo> if so, how long before?
19:39:10 <mirek> we have very hot winter this year :(
19:39:12 <nirik> willo: yeah, we do freezes 2 weeks before each release (alpha/beta/final)
19:39:35 <nirik> mirek: I think folks in the south of the us would happily send you some snow. ;)
19:39:54 <mirek> I would be glad to exchange :)
19:39:57 <willo> nirik: great
19:40:13 <threebean> oh, some of us have been discussing the potential for FADs in the coming few months, but nothing serious or focused yet.
19:40:29 <nirik> willo: http://infrastructure.fedoraproject.org/infra/docs/fedora-releases.txt has some more detailed info
19:40:46 <nirik> threebean: yeah, I would really like to do one. :)
19:40:48 <puiterwijk> threebean: I propose a FAD for moving everything to OpenID once FedOAuth 2.5 hits prod
19:40:49 <pingou> there would be budget on OSAS to make an infra FAD
19:40:52 <threebean> some ideas of things to get together one:  FAS3, bodhi2, taskotron maybe?
19:41:23 <nirik> ansible migrating, openid switching
19:41:30 * threebean nods
19:41:32 <threebean> yes
19:42:08 <threebean> some of the final ansible pieces will be tricky (like the proxy stuff) and would definitely benefit from multiple people being present
19:42:11 <nirik> just for the 'lets actually get it done' I am liking the bodhi2 idea. ;)
19:42:25 <nirik> yeah.
19:42:30 <threebean> yeah, well, fedora.next I think will put some new ideas into the mix for bodhi2
19:42:42 <nirik> it could well do so, yep.
19:42:44 <threebean> the more complicated karma system, and more feedback from taskotron to gate updates?
19:43:01 <pingou> I'm thinking maybe two groups
19:43:01 <threebean> anyways, that's just more reason to get together to brainstorm and implement
19:43:09 <pingou> FAS3 in one room, Bodhi2 in the other?
19:43:11 <nirik> theres also talk of a 'less constrained' repo of packages, possibly bodhi would need to be used to mash/manage that too...
19:43:20 <threebean> right, right
19:43:43 <nirik> and scls, etc
19:43:53 <nirik> (alhough they may just be rpms like any other)
19:44:39 <nirik> pingou: we could try multiple goals, but fads seem to do best when there's one specific goal thats reachable...
19:44:53 <nirik> if people get pulled from a to work on b and back they might not finish a or b
19:45:38 <nirik> but some things might also be very related.
19:45:49 * threebean nods
19:45:54 <nirik> if bodhi2 and taskotron are interacting, it might be good to work on both at the same fad.
19:46:09 <nirik> and... tflink and lmacken are both here in colorado, so if we did one in denver...
19:46:11 <puiterwijk> like openid intersects with about every other app we come up with :)
19:46:18 <threebean> nirik: heh, I was just about to say as much.
19:46:41 <threebean> puiterwijk: a lot of openid interactions we can implement/deploy on a case by case basis though, no?
19:46:57 <pingou> nirik: we need to figure out wrt costs, but Denver is clearly a sensible place ;-)
19:47:01 <dgilmore> nirik: i can get a direct flight to Denver
19:47:02 <threebean> as in, once FedOAuth 2.5 is deployed, we can update each application one after another over time.
19:47:14 <puiterwijk> threebean: yep. but I think we should try to move that forward.
19:47:15 <nirik> yeah.
19:47:23 <threebean> puiterwijk: agreed.
19:47:48 <puiterwijk> threebean: especially when the new OpenID specs land, that should give us a great reason to push forward more
19:47:49 <lmacken> +1 for Denver FAD :)
19:47:52 <tflink> any thoughts on when to do the FAD?
19:47:54 <pingou> puiterwijk: most app will switch when being re-written, so pushing bodhi2 should make you happy ;-)
19:48:03 <nirik> lmacken: you said your place had some facilities?
19:48:05 <puiterwijk> pingou: :D
19:48:06 <lmacken> also, logging is another potential item. centralized app logs, logstashery, etc
19:48:13 <nirik> oh yeah, logging++
19:48:15 <threebean> +1
19:48:15 * pingou thought end May beginning June
19:48:17 <lmacken> nirik: yeah, I can host a small FAD for sure
19:49:00 <nirik> pingou: might work.
19:49:26 <nirik> tflink: would that be something good from your side?
19:49:28 * masta looks in & lurks
19:49:50 <tflink> nirik: yeah, I'm definitely interested
19:50:03 <threebean> tflink: will you be at pycon again this year?
19:50:11 <nirik> when is pycon again?
19:50:15 * nirik goes to look
19:50:16 <threebean> aprilish?
19:50:23 <pingou> early April
19:50:27 <tflink> threebean: haven't decided yet, still need to figure that out
19:50:32 <nirik> ok, yeah, early apr
19:50:52 * willo is curious how people can assist remotely, if that's feasible
19:50:54 <threebean> cool, if you are and its no biggie, we could use that time to discuss and lay out some things to think about before hand
19:51:04 <puiterwijk> I need to head out for now
19:51:20 <nirik> so, how about I toss up a fad page on the wiki for a bodhi2/taskotron fad in late may/early june and we see from there if it will work out and if we can get everyone we need there.
19:51:23 <threebean> willo: surely we can do some kind of teleconference.  g+ hangout at least ;)
19:51:33 <adimania> +1 to willo
19:51:38 <adimania> and threebean
19:51:49 <nirik> yeah... we have #fedora-fad we use for irc to fads as they are running too...
19:51:56 <nirik> and remote help is always great too.
19:52:18 <willo> all: great, long, long way to come for a day :)
19:52:45 <pingou> willo: that wouldn't be a single day ;-)
19:53:12 <willo> :)
19:53:14 <threebean> yeah, probably a whole weekend or maybe more?  depends on people's schedules once we know more what we want to accomplish
19:53:53 <nirik> yep. usually 3ish days
19:54:13 <abadger1999> plus travel days
19:54:33 <willo> is plan documented in wiki usually or outlined in mailing list
19:54:48 <nirik> usually setup in the wiki first and discussed on list.
19:54:50 <relrod> both usually
19:54:50 <willo> was wanting to check out what was done last time
19:54:54 <relrod> yeah
19:54:57 <willo> awesome
19:55:19 <nirik> look at our 2012 security fad...
19:55:26 <nirik> should be on wiki and on list. ;)
19:55:28 <nirik> #topic Open Floor
19:55:32 <nirik> any items for open floor?
19:55:45 <threebean> I just need to share that..
19:55:50 <threebean> ..for lunch, I ate only pickles.
19:56:06 * pingou needs to go pick up his dinner (pizza \ó/!)
19:56:28 <nirik> ha.
19:56:40 * willo wil be having hotdogs todays
19:56:48 <willo> several
19:57:06 <nirik> today's triva: does anyone know (without looking at wikipedia or the like) what a blue eagle sign in a store would have meant? :)
19:57:37 <pingou> that the store owner is american? :)
19:57:50 <nirik> true.
19:58:03 <pingou> something to do w/ the army?
19:58:04 <nirik> and that their store was around in the us great depression. ;)
19:58:04 <willo> pingou: \o/
19:58:17 <pingou> didn't get that one
19:58:19 <nirik> anyhow, look it up if you like. ;)
19:58:20 <pingou> nirik: nice :)
19:58:25 <mariocav> O.o
19:58:30 <nirik> anything else? or will close out in a minute...
19:59:19 <nirik> thanks for coming everyone!
19:59:22 <nirik> #endmeeting