18:00:08 <nirik> #startmeeting Infrastructure (2014-04-03)
18:00:08 <zodbot> Meeting started Thu Apr  3 18:00:08 2014 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:08 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:08 <nirik> #meetingname infrastructure
18:00:08 <nirik> #topic welcome y'all
18:00:08 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
18:00:08 <zodbot> The meeting name has been set to 'infrastructure'
18:00:08 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean
18:00:16 <dgilmore> hola
18:00:25 * relrod_ here
18:00:47 * danofsatx-work is waiting for the official "roll call" call.....
18:01:06 <smooge> here
18:01:34 * threebean is here
18:02:29 * tflink is here
18:02:43 <nirik> ok, lets go ahead and get started. ;)
18:02:52 <nirik> #topic New folks introductions and Apprentice tasks.
18:03:00 <nirik> any new folks like to introduce themselves?
18:03:06 <nirik> Or apprentices with questions or comments?
18:03:10 <nirik> emedentsii_: ^
18:03:46 <danofsatx-work> last week, smooge mentioned maybe setting up some 'easyfix' ticket in regards to documentation. And thoughts on that?
18:03:46 <jmosco> I am new, hello everyone
18:03:49 <emedentsii_> hi
18:04:26 <nirik> welcome jmosco and emedentsii_. Can you each give a short intro about yourselves? and are you more interested in sysadmin or application development
18:04:27 <emedentsii_> I'm new too. jmosco , go ahead=)
18:04:48 * willo is here, sorry too many things open.  laptop was hung for a good couple of mins
18:05:03 <jmosco> I would say around 60/40
18:05:06 <jmosco> im trying to learn more dev
18:05:27 <jmosco> I have been working in a devops style role now with a focus on automation
18:05:55 <jmosco> ive been in infra now for around 5 years
18:06:33 <jmosco> :), and very happy to start working with the fedora team
18:06:33 <nirik> awesome. welcome. See me after the meeting if you want to be added to our apprentice program... also #fedora-noc (for sysadmin) and #fedora-apps (for application devel) are good channels to ask questions in.
18:06:52 <jmosco> thank you
18:08:21 <nirik> cool. Welcome again. I talked with emedentsii_ some earlier also
18:08:53 <nirik> any other new folks ?
18:09:22 <nirik> #info all easyfix trac tickets in fedora-infrastructure trac have been reset to new and milestone corrected.
18:09:34 <nirik> So, if you are looking for an easyfix, do take another look today...
18:09:40 <emedentsii_> My name is Eugene. I have 2 year expirience in Automation QA, writing tests using selenium WebDriver and python. Also i involved in DevOps process as a member of our contin integration team. I would like to develop application for fedora, also i think i can be usefull for some infrastructure tasks.
18:10:30 <threebean> oo, selenium and continuous integration?  that gives me an idea..
18:10:32 <nirik> emedentsii_: great. :) You may also be interested in our qa devel stuff... see tflink on that
18:10:55 <nirik> s/see/talk to/
18:10:59 <nirik> threebean: oh?
18:11:08 <threebean> emedentsii_: real quick -> we have a project called 'rube' that is supposed to test our staging infrastructure with selenium https://github.com/fedora-infra/rube/
18:11:35 <threebean> emedentsii_: right now, we just run it by hand every now and then.. but it would be neat to run it periodically via some automatic means.
18:11:44 <emedentsii_> ok, great. i will check it
18:12:09 <nirik> yeah. :) it could definitely be expanded to cover more too.
18:12:43 <nirik> cool.
18:12:52 <nirik> #topic Applications status / discussion
18:13:00 <nirik> Any application status or news this week?
18:13:12 <nirik> pingou is off at a conference without much coverage.
18:13:27 <threebean> one small thing.. our little landing page at https://apps.fedoraproject.org got some new behavior this week.
18:13:58 <threebean> it now has 'deep-linking' where you can click on an item, share the link, and it should navigate to the same position in the graph
18:14:02 <threebean> like this: https://apps.fedoraproject.org/#ForPackagers
18:14:06 <nirik> oh nice.
18:14:12 <relrod> I just got the jenkins-fedmsg plugin working locally (with threebean's help :D) - Need to talk to threebean later about how to deploy it, because our jenkins master node (a cloud node) will need to be able to access fedmsg somehow...but it is working fine locally, so that is good.
18:14:29 <threebean> relrod: that's awesome :P
18:14:49 <nirik> cool.
18:15:04 <nirik> #info apps now has deep linking support: https://apps.fedoraproject.org/#ForPackagers
18:15:29 <nirik> #info jenkins-fedmsg plugin is now working. Just needs deployment work/access.
18:15:45 <threebean> oh, and we got janeznemanic's genacls + fedmsg updater deployed, too.
18:15:50 <threebean> http://threebean.org/blog/pkgdb-acls-speedup/
18:16:04 <janeznemanic> not just mine
18:16:09 <threebean> :D
18:16:10 <janeznemanic> yours too
18:16:15 <nirik> #info fedmsg triggered genacls is now live.
18:16:35 <threebean> it is a neat proof-of-concept for replacing other cronjobs we have.
18:16:54 <threebean> fasclient, for example, syncs shell credentials from the FAS webapp every hour.
18:16:58 <nirik> yeah. ;) its nice to only run things when they need to run and not midlessly every hour or whatever
18:18:09 <relrod> I see charul just made some commits to https://github.com/charulagrl/mw-Datagrepper - would like to work with her on getting that deployed soon
18:18:20 <nirik> we talked about a lot of ways to make fasClient better... not sure if we are doing all of them or have picked some or what. :) We should figure out the actual roadmap
18:19:01 * tflink wonders if it would be worth looking into making taskotron's trigger more generic
18:19:23 <willo> question though about fedmsg generated events during outages
18:19:26 <threebean> relrod: she actually had a puppet patch to deploy it to production.. it just needs to be reworked to be tested in staging, I think.  very soon :)
18:19:48 <relrod> threebean: awesome!
18:20:36 <willo> what happens when the gitolite side of things that's consuming the fedmsg is not running and a pkgdb event triggers a message to be produced and place on the bus
18:20:49 <threebean> yeah, it will just miss it.
18:20:52 * relrod remembers the day he deployed mw-FedoraBadges, and haproxy saving his butt a few times... fun day that was. :P
18:20:53 <willo> how does gitolite know the event occured in the past
18:21:15 <willo> ah, ok, so cron job might stay as the fall back
18:21:16 <threebean> heh, it won't.  as things stand now, it would either need to be run by hand.. or the next acl change would trigger both to be picked up.
18:21:47 <threebean> willo: hm, so an idea was talked about but never implemented..
18:22:05 * abadger1999 arrives from the fpc meeting
18:22:12 <threebean> .. to have fedmsg consumers leave some sort of file around indicating the time that they were last running.
18:22:18 <nirik> I wonder tho, perhaps we could have fedmsg emit a 'I am shutting down now' message when a machine is stopped...
18:22:32 <nirik> and a start and then we could see between them?
18:22:42 <threebean> and when they start up, to look for that file and then query the history for all messages (of whatever type) since then.
18:22:46 <willo> yeah, was thinking about something like that last week as well
18:22:52 * threebean nods
18:23:07 <nirik> in practice I don't think this is a big deal right now, but would be nice to think about...
18:23:31 <willo> so, I guess network outages are not a common occurance. :)
18:23:43 <threebean> yeah, I'm for it.  It just needs some sustained attention from some people to implement/test/deploy. :p
18:24:07 * SmootherFrOgZ here
18:24:20 <nirik> willo: not too often... and most of the fedmsg emitting things are all in the same datacenter
18:25:19 <willo> nirik: ok, yeah cause I was wondering about fedmsg across the vpn's that might drop ocassionally
18:25:19 * smooge wonders if we could cram this in the journal and have systemd-fedmsg
18:25:32 <nirik> :)
18:25:59 <willo> damn internet, why can't it be more reliable.  :)
18:26:07 <nirik> indeed.
18:26:16 <dgilmore> smooge: ewwwwwwwww
18:26:19 <dgilmore> ewwww
18:27:08 <nirik> abadger1999 / SmootherFrOgZ: so do you recall the exact plans we had for fasClient? I know we talked about push mode, running from ansible, etc... but did push mode get implemented?
18:27:09 <smooge> I think if we do this by fed22 we could have use everyone's unused cycles for bitcoin mining and profit
18:27:55 * threebean puts smooge in charge
18:28:08 * dgilmore takes threebean and smooge out back
18:28:13 <nirik> fedmsg-bot: bitcoin.mine.start -- smooges server started mining bitcoins.
18:28:14 <SmootherFrOgZ> nirik: I've a branch with a 1rst-step implementation where fasClient listen to fedmsg to do stuff.
18:28:36 <nirik> SmootherFrOgZ: awesome. ;) This is 2.0?
18:28:50 <willo> :)
18:28:53 <nirik> so wait, fasClient runs all the time and listens? or ?
18:29:53 <SmootherFrOgZ> nirik: for now, yes. but we could work from this implementation to tell ansible do fire specific fasClient.
18:30:08 <dgilmore> id rather push via ansible
18:30:28 <dgilmore> not make something extra listen on every box
18:31:09 <nirik> yeah, I think... thing on lockbox01 listens for fedmsgs related to accounts. When they come in, it looks to see if they matter for any. If they do, it fires off to run fasClient on those machines. I think we could implement this today.
18:31:25 <dgilmore> nirik: yep
18:31:47 <nirik> but that would be specific to our setup, might be nice for a more generic thing in fas upstream... but not sure how much where.
18:32:15 <dgilmore> nirik: would need some kind of policy engine
18:32:26 <dgilmore> that has a config with all teh amppings
18:32:36 <dgilmore> I wish i could spell
18:33:10 <dgilmore> nirik: I think it could be easily done today
18:33:22 <dgilmore> and would be generic enough to go upstream
18:33:35 <threebean> .. and, the mappings/policy.  what is that?
18:33:43 <nirik> so wait... the only things we care about are: ssh key changed and user added/deleted right?
18:33:47 <threebean> a mapping of "changes to these fas groups mean push changes to these machines?"
18:33:57 <dgilmore> threebean: today its spread through puppet and ansible
18:34:21 <dgilmore> which fas groups land on which boxes
18:34:26 <nirik> user added/deleted from group we care about that is.
18:34:35 * threebean nods
18:34:49 <dgilmore> passwords or ssh keys changed
18:34:52 <nirik> so, really we could do a first cut of this that is dirt simple:
18:34:54 <dgilmore> users added or removed
18:35:08 <nirik> watch for ssh key, user added/removed from group. Run fasClient on all machines.
18:35:17 <nirik> because that doesn't happen that often at all anyhow.
18:35:22 <nirik> it would cut down runs vastly
18:35:31 <SmootherFrOgZ> nirik: yep. my changes handle what you want. you could look at it as sample
18:35:37 <nirik> dgilmore: don't need passwords, fas doesnt store them anymore
18:35:40 <dgilmore> nirik: and passwords?
18:35:47 <nirik> well, fas does
18:35:56 <dgilmore> nirik: how are they getting on the boxes for sudo?
18:35:56 <nirik> but fasClient doesn't put them in the nss db
18:36:15 <threebean> boxes ask fas via pam_url, correct?
18:36:17 <nirik> sudo uses 2fa
18:36:19 <nirik> yeah
18:36:27 <nirik> except for some small number of weird hosts.
18:36:31 <SmootherFrOgZ> nirik: fasClient does
18:36:33 <SmootherFrOgZ> actually
18:36:44 <nirik> I thought we disabled that?
18:37:28 <SmootherFrOgZ> hm...is this upstream changes or just config updates?
18:37:47 <nirik> can't recall, but when we moved to 2fa we saw no need to expose passwords on all hosts.
18:37:57 <nirik> I can double check that out of meeting
18:38:03 * SmootherFrOgZ nods
18:38:20 <dgilmore> nirik: i know that the secondary sigul boxes dont do 2 facter
18:38:32 <dgilmore> and one of the compose boxes doesnt
18:38:36 <nirik> dgilmore: yeah, there's about 4 or 5 weird hosts that don't
18:38:48 <nirik> but we should move them to it too. ;)
18:39:08 <dgilmore> anyway
18:39:22 <nirik> right, sorry... I will file a ticket on the fasClient thing and we can see what we can get going
18:39:34 <nirik> any other application news?
18:39:40 <threebean> https://fedorahosted.org/fedora-infrastructure/ticket/3925
18:39:46 <nirik> #action nirik to file a ticket for fasClient changes
18:40:00 <nirik> #info fasClient could likely be made to run MUCH less often without too much work
18:40:11 <nirik> ha too fast. ;)
18:40:14 <nirik> thanks threebean
18:40:29 <webpigeon> Evening
18:40:43 <nirik> #topic Sysadmin status / discussion
18:40:49 <nirik> morning webpigeon. ;)
18:40:56 <nirik> ok, on the sysadmin side of things...
18:41:08 <nirik> we have new machines racked, we just need networking on them and can get them installed.
18:41:15 <nirik> smooge: is there a ticket on networking those new boxes?
18:41:54 <nirik> #info easyfix tickets have been updated and cleaned up some.
18:41:55 * axil42 sneaks in late
18:42:09 <nirik> we did a mass reboot on tuesday. It went pretty well.
18:42:19 <SmootherFrOgZ> threebean: for the record: https://github.com/fedora-infra/fas/blob/feature/new_fasClient/client/fas_client/daemonize.py
18:42:20 <axil42> nirik, thanks for the cleanup ;)
18:44:02 <nirik> no problem
18:44:38 <nirik> I still need to make some sundries servers and look at the ansible wiki migration diffs.
18:44:53 <nirik> I did also file a few new easyfix tickets
18:45:02 * webpigeon looks
18:45:17 <smooge> I need to put in some documentation tickets..
18:46:37 <nirik> smooge: cool. Docs on what?
18:47:10 <danofsatx-work> From what I see, a lot of the how-tos on the Fedora docs page haven't been updated since 14 to 16.
18:47:12 <smooge> well I figured we needed to go over documentation we have already.. and see if it actually meaans anything
18:47:27 <smooge> well that is different documentation
18:47:40 <nirik> danofsatx-work: docs.fedoraproject.org? yeah, thats the fedora docs group that does those.
18:47:40 <danofsatx-work> oh, ok...wrong docs ;)
18:48:32 <nirik> we have these: http://infrastructure.fedoraproject.org/infra/docs/
18:48:38 * danofsatx-work was befuddled
18:48:38 <threebean> SmootherFrOgZ: cool :)
18:48:39 <nirik> updates to which are always welcome.
18:48:52 <nirik> #info updates to http://infrastructure.fedoraproject.org/infra/docs/ always welcome
18:48:53 <smooge> I am talking about 'how are we set up?' 'what do we expect people to be able to do on people/pkgs/hosted', 'what commands should we do in git', 'why is the sky blue and vegemite black', 'who was on the grassy knoll'
18:49:08 <axil42> smooge, +1
18:49:10 * nirik makes a 'jfk.txt' doc
18:49:24 <willo> :)
18:49:36 <smooge> then take the csi docs and see if we still 'follow' them,
18:49:55 <nirik> yeah, csi also needs love. Just not had time from my side.
18:51:06 <nirik> #topic Upcoming Tasks/Items
18:51:07 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
18:51:14 <nirik> anything upcoming folks would like to note or schedule?
18:51:40 <nirik> looks like I will be giving an ansible thing at the centos dojo in denver next week. ;) Need to write that up...
18:51:53 <threebean> nirik: cool :)
18:51:55 <nirik> #info nirik at centos dojo in denver next week
18:52:15 <nirik> #info bodhi2/taskotron fad planning ongoing, in june in denver
18:52:26 <threebean> the Bodhi2 FAD is still coming up in June..
18:52:47 <threebean> I'm still trying to figure out some of the budget stuff but I haven't had luck getting in touch with rsuehle this week.
18:52:48 <nirik> I might be away first week of may... still havent decided my plans there yet, but will let folks know when I do.
18:53:09 <nirik> #info flock talk submissions closes very very very soon.
18:54:15 <nirik> #topic Open Floor
18:54:21 <nirik> any items for open floor? questions, comments/
18:55:34 <smooge> not from me
18:55:46 <smooge> nirik, ah man now I want to go to the denver dojo
18:55:55 <threebean> random item ->
18:56:01 <danofsatx-work> all's quiet on the southern front
18:56:07 <nirik> smooge: come up. :) It's only a 5-6 hour drive...
18:56:13 <threebean> github users, I packaged up a cli tool: https://bugzilla.redhat.com/show_bug.cgi?id=1083344
18:56:42 <threebean> it is fun to use.. EOM
18:56:42 <willo> threebean: cool
18:57:13 <nirik> nice
18:57:24 <webpigeon> shiny :)
18:57:41 <nirik> ok, if nothing else, will close out the meeting in a minute here.
18:58:17 <willo> a lot less typing to clone github repo, sweet
18:58:30 <threebean> willo: yeah, it can interact with pull requests and stuff too
18:58:39 <emedentsii_> Thank you!
18:58:53 <willo> nice
18:59:25 <nirik> ok, thanks for coming everyone. :) Lets go continue over in #fedora-admin, #fedora-apps and #fedora-noc. See you all there.
18:59:27 <nirik> #endmeeting