19:00:39 <nirik> #startmeeting Infrastructure (2013-09-26)
19:00:39 <zodbot> Meeting started Thu Sep 26 19:00:39 2013 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:39 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:39 <nirik> #meetingname infrastructure
19:00:39 <zodbot> The meeting name has been set to 'infrastructure'
19:00:39 <nirik> #topic welcome everyone
19:00:39 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
19:00:39 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean
19:00:52 * relrod here
19:00:52 * tflink is lurking
19:00:53 * mirek waves
19:00:54 * pingou here
19:00:57 * threebean here
19:01:00 * anshprat here
19:01:02 * adimania is here
19:01:24 * robyduck is also here
19:01:38 <frankieonuonga> i am here as always
19:01:40 * croberts croberts is here
19:01:40 * ausmarton is here
19:01:59 <Xeladem> here too :)
19:02:08 <frankieonuonga> i dont know how to do that i am here thing
19:02:20 <pingou> /me\
19:02:21 <nirik> frankieonuonga: /me whatever
19:02:24 * pingou sorry
19:02:30 <pingou> /me sorry # damn it!
19:02:41 * ynchen 
19:02:42 <mirek> :)
19:02:47 <nirik> :)
19:02:48 * ynchen is here
19:02:51 * Xeladem waves
19:02:54 <nirik> ok, lets go ahead and get started then...
19:03:05 * frankieonuonga is here
19:03:08 <nirik> #topic New folks introductions and Apprentice tasks.
19:03:18 <nirik> any new folks like to introduce themselves?
19:03:25 <nirik> or apprentices with questions or comments?
19:03:38 <ynchen> hi i just joined this week, i looking to join the sysadmin-noc
19:03:51 <robyduck> Yes, me, I'm Robert Mayr from Italy, would like to help out also in the infra team :)
19:04:04 <anshprat> am working on speeding up askbot presently..
19:04:13 <pingou> robyduck: awesome, welcome :)
19:04:15 * croberts waves hi to robyduck
19:04:19 <nirik> welcome everyone. ;)
19:04:29 <robyduck> Hi pingou, nice to meet you here :)
19:04:32 <threebean> anshprat: thanks for that.. that will be nice :)
19:04:47 * threebean waves "hello" to robyduck
19:04:56 * pingou wonders how threebean does that
19:05:02 <robyduck> hey threebean :)
19:05:04 * frankieonuonga waves to the new crew
19:05:12 <robyduck> pingou ^^
19:05:45 <nirik> as always do chime in with any questions as we go... and #fedora-admin, #fedora-noc, and #fedora-apps are the place(s) to watch for things to do...
19:06:18 <nirik> #topic Applications status / discussion
19:06:26 <nirik> what do we have application wise this week or upcoming?
19:06:37 <pingou> nuancier
19:06:58 <pingou> we have a stagging version working: https://apps.stg.fedoraproject.org/nuancier
19:07:07 <nirik> #info nuancier is rolling out soon! :)
19:07:18 <pingou> and pkgdb2 w/ pkgdb1 data http://209.132.184.188/
19:07:27 <nirik> cool.
19:07:29 <pingou> please test, break and let me know how it behaves :)
19:07:34 <nirik> what are the next steps on pkgdb2?
19:07:40 <mirek> https://fedoraproject.org/wiki/Infrastructure/RFR/Copr - any question to this?
19:08:03 <pingou> for pkgdb2: polishing a little still, finish the API authentication question, pkgdb-cli update
19:08:09 <pingou> more unit-tests
19:08:23 <pingou> and that should be it
19:08:29 <nirik> pingou: wow. cool.
19:08:51 <nirik> mirek: yeah... I still have marked to reply to your message about resources...
19:08:54 <pingou> but it takes ~3h to convert the db to the new model :(
19:08:54 <nirik> just been swamped. ;)
19:09:08 <pingou> so we'll need to schedule some down time when we do the migration
19:09:15 <nirik> thats fine.
19:09:32 <pingou> I'm also thinking to send a request for testing to the devel list
19:09:44 <pingou> that I can get bug report and RFE as early as possible :)
19:09:45 <nirik> pingou: +1
19:09:48 <threebean> good idea
19:09:59 <mirek> I'm not sure what is next in process of RFR, should you vote about it? or what? or can I ask for that DB...?
19:10:18 <nirik> mirek: I have a number of questions. ;)
19:10:41 <nirik> are the frontend/backend things we want to run in our normal infrastructure? or are they best still being in the cloud?
19:10:42 * oddshocks late
19:10:49 * pingou has still a point about nuancier, but will wait for after the copr section
19:10:57 * anshprat too
19:11:00 <mirek> nirik: still in cloud.
19:11:22 <mirek> nirik: but there is nothing cloud specific
19:11:36 <nirik> https://fedorahosted.org/fedora-infrastructure/ticket/4023 and https://fedoraproject.org/wiki/Infrastructure/RFR/Copr
19:12:11 <nirik> so, I wonder if this would be a good fit for something to be in the other RH cloud that we were offered resources in... since it needs more disk than we have around.
19:12:49 <nirik> also, can you expand on whats in the currently used disk?
19:12:58 <nirik> is it keeping everything ever built? or ?
19:12:58 <mirek> nirik: I did not get response from rvokal about that, I'm asking him every other day thou
19:13:40 <pingou> (how would the "new" cloud interact with the db?)
19:13:45 <nirik> mirek: well, I can talk with Dan Yocum, who is who I was talking to about it... They don't have external ip's yet tho, which we would need.
19:13:48 <pingou> our db, a dedicated node db?
19:13:55 <mirek> nirik: right now it keeps all builds. I have to come with some policy which build will be deleted
19:14:29 <nirik> if it's in the cloud, it would need it's own db also in the cloud most likely.
19:14:37 <nirik> because our db host is not externally reachable.
19:14:57 <mirek> nirik: well ask him :) you can make it only better :)
19:15:01 <nirik> which I guess we could get around via vpn, but I'd rather not.
19:15:25 <pingou> so we can also make it more easily accessible to mirek
19:15:31 <nirik> mirek: will do. I can cc you on email.
19:16:17 <mirek> I can prepare playbook for frontend (which use db) to be deployed as normal fedora service and backend can stay in cloud
19:16:29 <frankieonuonga> nirik:can the cc be to copr-devel
19:16:37 <mirek> we would need just 22  and 443 ports between those two machines
19:16:39 <frankieonuonga> keeps everyone in the loo[
19:16:43 <frankieonuonga> loop i mean
19:16:57 <nirik> well, this is an implementaion detail, but I can send whatever I find out there too.
19:17:07 * nirik will subscribe
19:17:48 <lmacken> I did a bodhi masher update yesterday which will hopefully fix the duplicate notice updateinfo.xml issue
19:18:03 <nirik> mirek: that might work... the frontend would need credentials to talk to the cloud, but we could probibly setup that.
19:18:20 <nirik> lmacken: cool. ;) pesky timestamps.
19:18:24 <pingou> mirek: but then the frontend should be packaged :)
19:18:33 <mirek> nirik: ok, I will still count with 3 options, ordinary fedora service, staying in fed.cloud, or that hosting, we will see what will look better next week
19:18:44 <mirek> pingou: it is already
19:18:53 <nirik> pingou: happily it just passed review. ;)
19:18:53 <mirek> pingou: even with selinux :)
19:19:06 <mirek> nirik: thanks for that btw
19:19:17 <pingou> \ó/
19:19:17 <nirik> no problem.
19:19:21 <pingou> champagne!
19:19:25 <threebean> hooray!
19:19:47 <nirik> mirek: where is the storage used for packages? off the frontend?
19:19:57 <mirek> nirik: on backend
19:20:08 <nirik> ok
19:20:08 <pingou> mirek: I still owe you a couple of patch for the API/CLI
19:20:43 <mirek> pingou: and I'm still waiting :)
19:20:54 <nirik> I think we may want a pretty agressive pruning policy... but we can see I suppose.
19:21:44 <mirek> yes, I will discuss it on copr-devel and copr deployment is delayed to next week, do you agree?
19:21:46 <nirik> mirek: any other coprs questions we should look at for now?
19:21:57 <nirik> yep. I need to subscribe there, but will do so. :)
19:22:05 <nirik> abadger1999: did you have any other copr questions?
19:22:41 <abadger1999> nope
19:22:56 <nirik> cool. Any other upcoming application news?
19:23:03 <pingou> about nuancier
19:23:20 <pingou> so it generate thumbnails for the pictures 'candidates' to an election
19:23:30 <pingou> atm these pictures are installed by hand on each node
19:23:49 <pingou> while the cache is generated by the frontend and thus only on one node
19:23:59 <nirik> dumb question: are these the same size/format the wiki uses?
19:24:10 <anshprat> pingou:  I found a bug and an issue in nuancier.. ping you after the meeting..
19:24:21 <pingou> but we can easily generate the cache by hand on each nodes, so this might be the simplest solution for the moment for nuancier
19:24:23 <pingou> anshprat: please :)
19:24:58 <pingou> in the long term, when nuancier will handle upload of the candidates, then we should probably look into a shared storage
19:25:06 <pingou> nirik was proposing glusterfs
19:25:19 <pingou> (eof)
19:25:21 <nirik> we could do gluster or just do a nfs netapp volume...
19:25:40 <nirik> we already have one for the wiki, we could also just reuse part of that...
19:26:06 <pingou> either way is fine for me, my question is more, do we want to have this already or should we wait for nuancier full feature?
19:26:13 <nirik> but If you are ok with manual for now, we can just revisit it later.
19:26:37 <nirik> when is the go-live date?
19:26:40 <pingou> nirik: the pictures themselves are pretty big (not sure what the wiki uses, but likely the same), the thumbnails are 128x128px
19:26:43 <nirik> for the upcoming election?
19:26:45 <pingou> nirik: Monday
19:27:09 <anshprat> pingou: is there a way to download and try out the images ?
19:27:22 <nirik> yeah, I'd say manual for now, revisit after this election.
19:27:24 <anshprat> (full res I mean)
19:27:43 <pingou> anshprat: yes, right click on the 'google' icon when hovering over the image
19:27:48 <pingou> nirik: wfm, thanks :)
19:28:18 <nirik> #info bodhi fix for duplicate updates hopefully out
19:28:32 <nirik> #info coprs going to discuss best deployment path and revisit next week
19:28:55 <nirik> #info nuancier going to go with manual cache/thumbnailing for now, revisit after the upcoming voting.
19:29:03 <lmacken> mashing in koji discussions underway :)
19:29:10 <threebean> cool :)
19:29:19 <nirik> yeah. :) I added some thoughts to that thread.
19:29:25 <nirik> would be great to move that into koji
19:29:49 <lmacken> from a bodhi perspective, it has to end up in /mnt/koji anyway, so bodhi could simply flip symlinks
19:30:14 * lmacken should chime in on the thread :)
19:30:17 <nirik> yeah, it's a lot of data to shlep around too... and needs enough space to write it.
19:30:44 <lmacken> yeah, moving it from the builders to the hub, etc, would be a lot of work
19:30:47 <nirik> lmacken: how would we handle signing? just like now? get a list, sign, push and hopefully everything in there is signed?
19:31:08 <lmacken> signing doesn't have to change at all, right?
19:31:18 <nirik> yeah, it could be the same
19:31:57 <nirik> #topic Sysadmin status / discussion
19:32:03 <nirik> ok, on the sysadmin side... lets see.
19:32:14 <nirik> #info new ansible on lockbox with some nice new features.
19:32:29 <nirik> One of the new nice things in ansible 1.3.x is 'accelerate' mode.
19:32:44 <threebean> its fireball but without the zeromq libs :)
19:32:47 <nirik> I think we should just look at enabling that everywhere and using it. It makes things much much faster
19:32:53 <threebean> +1
19:32:56 <nirik> yep. It's more simple... no zmq
19:33:14 <pingou> what was zmq used for already?
19:33:24 <nirik> it does need python-keyczar on each node.
19:33:30 <nirik> but thats got only a few deps.
19:33:33 <lmacken> 2-6x faster than ssh apparently
19:33:55 <lmacken> ah, all we do is add 'accelerate: true' to our playbooks??
19:34:07 <nirik> gallery playbook: with accel: 32sec, without: 1minute 50 seconds.
19:34:13 <relrod> wow
19:34:17 <threebean> its fun :)
19:34:32 <nirik> lmacken: and install python-keyczar on the hosts being playbooked
19:34:41 <lmacken> cool
19:35:06 * abompard coming in late
19:35:26 <nirik> #info close to having the new cisco ucs blade server working
19:35:44 <nirik> I have one of the blades asking for dhcp correctly now... but then it ignores the ip. ;(
19:36:03 <nirik> I'm going to schedule 2 outages for next week:
19:36:47 <nirik> 1. on tuesday for buildsys stuff. Will move some guests around on bvirthosts, and also will move lookaside cache on pkgs01 to netapp storage, and also apply updates/reboots
19:37:23 <nirik> 2. on wed for class a/non buildsys stuff. moving some guests to a new virthost15 and updates/reboots.
19:37:36 <nirik> I'll file outages for them and send announcements later today.
19:37:50 <nirik> If there's anything else someone needs an outage for let me know.
19:38:39 <nirik> We are trucking thru the backlog of stuff waiting for the freeze to be over...
19:38:51 <nirik> if anyone sees tickets that were waiting on that that haven't been applied yet, please note them. ;)
19:39:13 <nirik> anything else on the sysadmin side?
19:39:20 <nirik> #topic Upcoming Tasks/Items
19:39:21 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
19:39:33 <nirik> calendar should be up to date on our milestones now. ;)
19:39:38 * nirik figured out how to edit it
19:39:45 <pingou> \ó/
19:40:17 <nirik> just to note: I am going to be out next friday, all the following week and then back the monday after that.
19:40:32 <nirik> I'll possibly check emails and say hi, but won't be on vastly much. ;)
19:40:42 <oddshocks> nirik: will someone else run the meeting?
19:40:56 <frankieonuonga> nirik: enjoy
19:41:00 <pingou> nirik: have fun :)
19:41:01 <frankieonuonga> the break
19:41:07 <nirik> yeah, need to line someone up. ;) Usually smooge does.
19:41:13 <oddshocks> cool :)
19:41:55 <nirik> our next freeze is after I get back...
19:42:01 <nirik> #topic Open Floor
19:42:02 <relrod> I should be around if he can't be
19:42:14 <nirik> anything else anyone would like to discuss or note?
19:42:20 <nirik> relrod: thanks.
19:42:47 <croberts> I am going to send in a commit for the FAS newslink and get that ticket closed
19:42:59 <croberts> and file the upstream bug that you suggested nirik
19:43:01 <nirik> croberts: excellent.
19:43:03 <frankieonuonga> i can help relrod
19:43:16 <nirik> croberts: there's a few more wiki related tickets if you are looking for more. ;)
19:43:35 <croberts> nirik: nice :) I will check out the trac and look at them
19:43:58 <nirik> cool.
19:44:58 <relrod> I'm in the process of trying to set up a dpsearch instance and get frankieonuonga access to it, but our temp instance playbooks are running into various issues with the ansible update
19:45:17 <smooge> sorry been cleaning up.
19:45:47 <nirik> relrod: I can assist some too... I cleaned up some various issues with other playbooks.
19:46:16 <nirik> smooge: no worries.
19:46:22 <nirik> smooge: anything to note for the meeting?
19:46:27 * frankieonuonga agreeing that relrod has been of great assistance
19:47:08 <relrod> frankieonuonga: :)
19:48:19 <nirik> ok, will close out in a min if nothing else...
19:49:18 <nirik> Thanks for coming everyone. Do continue in our normal channels. :)
19:49:20 <nirik> #endmeeting