18:00:00 <nirik> #startmeeting Infrastructure (2015-06-18)
18:00:00 <zodbot> Meeting started Thu Jun 18 18:00:00 2015 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:01 <nirik> #meetingname infrastructure
18:00:01 <nirik> #topic aloha
18:00:01 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk pbrobinson
18:00:01 <nirik> #topic New folks introductions / Apprentice feedback
18:00:01 <zodbot> The meeting name has been set to 'infrastructure'
18:00:01 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pbrobinson pingou puiterwijk relrod smooge threebean
18:00:11 * threebean 
18:00:12 <dgilmore> hola
18:00:13 <puiterwijk> hello
18:00:16 <nirik> any new people around or apprentices with questions or comments?
18:00:19 <cydrobolt> hello!
18:00:23 * sborza waves hello
18:00:25 * oddshocks pops in
18:00:26 * relrod here
18:00:54 * Corey84 is apprentice but  not any questions coming  to moind
18:01:07 <nirik> #topic GSoC student update - kushal
18:01:18 <nirik> any GSoC students want to give a update on their project?
18:01:20 <kushal> Who all are here?
18:01:26 * sonalkr132 is here
18:01:26 <kushal> nirik, thanks :)
18:01:37 <nirik> kushal: no problem. :)
18:01:53 <Corey84> been active with the tox  stuff mostly on github tho (not student tho)
18:01:57 <AnuradhaW> Hi, I'm here.
18:02:10 <kushal> sonalkr132, AnuradhaW your updates please.
18:02:29 <sonalkr132> @rahulrrixe and @sshagarwal was here a moment ago
18:02:35 <prth> i have completed the basic implementation of dynamically created multi cropping overlays.
18:02:39 <AnuradhaW> I have worked on getting a better layout for Q/A page of askfedora.
18:02:43 <sshagarwal> I am still here
18:02:43 <prth> next, I'll be polish the ui
18:02:53 <rahulrrixe> Kushal: Written a blog for this week update https://medium.com/@rahulrrixe/becoming-git-pro-by-getting-into-under-the-hood-417054b3f4aa
18:03:15 <AnuradhaW> And the mentors have pointed out some issues with the detailed coding of the page and I'm currently fixing those.
18:03:22 <kushal> Okay
18:03:46 <sonalkr132> As of I was working on setting up git server for GG
18:04:08 <sonalkr132> Only last part of puzzle is left, ie integration of sparkleshare
18:04:18 <sonalkr132> #link https://chasingcrazydreams.wordpress.com/2015/06/18/awesomeness-of-rack/
18:04:26 <prth> s/polish/polishing
18:04:27 <rahulrrixe> I have completed all the backend code for the git integeration and working on UI. I think first milestone will be complete by today as I have to fix some minor bugs in pull-request and merging code.
18:04:45 * danofsatx is back
18:04:53 <kushal> We will talk to your mentors for the midterm next week.
18:05:06 <Corey84> rahulrrixe,  nioce  blog
18:05:14 <kushal> I will also count the presence here.
18:05:22 <sshagarwal> We needed a protocol to demonstrate the Skinkglue way of adding protocols to the client using addon approach. So, I am in the process of writing the protocol to display folder emails (email files placed locally in a folder).
18:05:33 <kushal> nirik, you can continue I guess.
18:05:38 <rahulrrixe> Corey84: Thanks :)
18:05:51 <nirik> kushal: thanks.
18:06:07 <nirik> on to announcements/info
18:06:11 <nirik> #topic announcements and information
18:06:11 <nirik> #info Got the runroot plugin working in our production koji - threebean / nirik
18:06:11 <nirik> #info arm03-packager and arm03-qa instances announced and reinstalled with f22 - nirik
18:06:11 <nirik> #info production koji store mounted ro on koji01.stg - nirik
18:06:11 <nirik> #info grew storage for secondary arch kojis - nirik
18:06:12 <sshagarwal> kushal: hi --^
18:06:12 <nirik> #info fixed fed-cloud09 playbook to be idempotent for check/diff at least - nirik
18:06:13 <nirik> #info initial people01 setup in ansible/rhel7. Still needs work - nirik
18:06:15 <nirik> #info mote pushed to production at https://meetbot.fedoraproject.org - still working out some kinks - cydrobolt / threebean
18:06:18 <nirik> #info next trip to PHX2 will be July 27->July 30th. Hardware to be installed needs to be there by the 27th.
18:06:23 <nirik> any further discussion on those or things to add?
18:06:47 * roshi is here
18:07:00 <nirik> on to a discussion item added by mdomsch...
18:07:03 <nirik> #topic - mdomsch - Retire MM 1.4.4 from Fedora and EPEL repos
18:07:05 * roshi wonders why 1800 UTC always sneaks up on him...
18:07:15 <nirik> from the gobby doc:
18:07:18 <nirik> Not sure I can attend, but I see no reason to keep mirrormanager 1.4.4 in the Fedora 23
18:07:18 <nirik> repositories.  We have a couple choices:
18:07:18 <nirik> 1. transition owner to someone and let them put mirrormanager 2.x into the repos
18:07:18 <nirik> 2. orphan and delete MM 1.4.x from FC23, let the releases in FC21-22 remain as-is
18:07:19 <nirik> Happy to do whichever the team desires, and if someone with pkgdb admin rights wants
18:07:20 <nirik> to take care of it too, don't wait on me. Thanks! -mdomsch
18:07:21 <nirik> (P.S. it's in EPEL repos too, follow the same logic).
18:07:34 <nirik> IMHO, I'd like to see us take over the package and update it to mirrormanager2.
18:07:52 <puiterwijk> yeah, agreed
18:07:54 <nirik> pingou / adrian: any thoughts on that idea?
18:08:17 <threebean> I think pingou is away..
18:08:25 <nirik> yeah...
18:08:30 <Corey84> agree with  taking it  in
18:09:18 <threebean> I dunno if it would be easier or harder to do it with a separate package and provides/obsoletes?
18:09:29 <threebean> or, if that makes a difference.
18:09:31 <nirik> yeah, not sure. Would need a re-review...
18:10:34 <threebean> eh, maybe table this for next week so pingou and adrian can comment.
18:10:38 <nirik> ok, I'll try and move this forward someway. yeah.
18:10:45 <puiterwijk> it doesn't matter either way. Except that with a new package, we'd be more clear that it's a rewrite
18:10:49 <Corey84> obsoletes would kinda make senses (to me at least)
18:11:07 <nirik> puiterwijk: well, yeah, but 1.x vs 2.x might say that too
18:11:24 <Corey84> not everyone checks that tho sadly
18:11:30 <puiterwijk> nirik: fair enough. But anyone else runniung it and just doing "yum update" might except it to Just Work(R)
18:11:40 <Corey84> that too^
18:12:02 <Corey84> would need some  hefty docs i'd think to go that route
18:12:03 <nirik> puiterwijk: sure, we could also just update rawhide
18:12:10 <nirik> and epel7 (which doesn't yet exist)
18:12:25 <puiterwijk> fair enough
18:12:34 <nirik> then f23+ will get it.
18:12:44 <puiterwijk> people won't expect to be able to just update from F22->F23 without any work, I guess :)
18:12:55 <nirik> but then we will need to keep epel6 on the old one...
18:13:04 <nirik> but I doub't there's too many users.
18:14:27 <kushal> nirik, I have an open question for the open floor.
18:14:31 <nirik> ok, anything more to discuss, or shall we move on to letting threebean teach us about fedmsg
18:14:43 <smooge> well if it is side packages mirrormanager1x mirrormanager2x , could both be in EPEL?
18:14:45 <nirik> kushal: ok, can wait for that, or you want to do it now?
18:14:48 <Corey84> why would that requires epel6 still what am i missing
18:14:58 <kushal> nirik, I can wait :)
18:14:59 <nirik> smooge: they could, then we would need to maintain them both f
18:15:21 <nirik> Corey84: epel6 has mirrormanager 1 in it. Upgrading it to mirrormanager2 is not allowed in the epel guidelines.
18:15:49 <Corey84> ah im not following the epel as closely apparently
18:16:06 <nirik> #topic Learn about: fedmsg - threebean
18:16:13 <nirik> threebean: the floor is yours. :)
18:16:16 <threebean> cool :)
18:16:36 <threebean> so, if there's questions about stuff as I go, just ask.
18:16:48 <threebean> i'm not sure what corners people are more interested in, so.. I'll just overview.
18:16:59 <threebean> http://fedmsg.com/
18:17:20 <threebean> so, fedmsg is our in-house message bus that connects most of the 40-some services we run.
18:17:32 <threebean> it's built on top of zeromq for the underlying message passing.
18:18:05 <threebean> it's a pub/sub style system.. so it can't do things like request-reply or other more special-purpose patterns.
18:18:22 <Corey84> would it need to tho?
18:18:30 <threebean> Corey84: not yet it hasn't.
18:19:03 <threebean> one of the design guidelines going in was to make it as 'resilient' as possible, so that if some or any of the pieces go down, the others can function as gracefully as possible.
18:19:35 <threebean> this means that, if you stand up one of our fedmsg-enabled webapps on your laptop, it can run and publish fedmsg's without having to setup any other services.
18:19:55 <threebean> but.. this has no effect, because nothing else is configured to listen to that lonely service running on the laptop.
18:20:36 <threebean> we don't have a central broker, like other messaging systems do.  but we do have a variety of optional services that can play portions of the role played by a central broker.
18:20:36 <Corey84> so each services while connected can be standalone?
18:20:38 <rahulrrixe> threebean: Is there any available software architecture diagram of fedmsg? I am interested in learning its underlying components.
18:20:43 <threebean> Corey84: right, right.
18:21:16 <threebean> rahulrrixe: I think this might be the best that we have -> http://www.fedmsg.com/en/latest/topology/
18:21:19 <roshi> what services aren't currently on the bus and is there a roadmap to get them on the bus?
18:21:24 <Corey84> sounds alot like how tox is  architected non  central and  standalone (or even popcorntime)
18:21:46 <threebean> rahulrrixe: but there are many more services now that aren't in that diagram.. it would just get too big.
18:22:07 <Corey84> threebean,  this is the same fedmsg  in git2fedmsg yes?
18:22:09 <threebean> roshi: there's this doc for the status of different apps -> http://www.fedmsg.com/en/latest/status/
18:22:14 <Corey84> github*
18:22:25 <threebean> Corey84: yes.  :)
18:22:42 <Corey84> g2k  thats what side i use most not the webapp
18:22:46 <threebean> roshi: but it needs a little love.  for instance, we need to add 'zanata' to it.
18:23:08 <rahulrrixe> threebean: It suffiecient for now. BTW thanks :)
18:23:12 <roshi> is querying datanommer the best means to gather historical data from the bus?
18:23:17 <Corey84> zanata is the reminder stuff ?
18:23:28 <threebean> Corey84: it's a translation platform at fedora.zanata.org
18:23:39 <Corey84> ah
18:23:52 <threebean> the *big* missing piece from the bus right now is bugzilla content.  I can only assure you that we've been working on it, it's slow going, and unfortunately there's no ETA yet.
18:24:00 <threebean> it would be pretty valuable information to have in the stream.
18:24:11 <threebean> the best way to query for history stuff, yes, is (technically) datagrepper
18:24:18 <threebean> https://apps.fedoraproject.org/datagrepper/
18:24:38 <threebean> if you have an app that lives inside fedora infrastructure, you can circumvent that webapp and query the datanommer postgres db directly
18:24:43 <Corey84> what are the biggest snags on bz  implementation then
18:24:49 <threebean> which would be slightly faster than having to setup and teardown HTTP requests to datagrepper
18:25:34 <threebean> Corey84: just stuff on the RH bz side.  they need to publish messages to many other communities besides fedora (like jboss) and they're working on how best to do that in a way that makes everyone happy.
18:25:37 <roshi> is there a batch archive of the database we can populate something local with? So we don't have to send queries over the network for testing?
18:25:39 <Corey84> brb, coffee  refill
18:25:42 <threebean> roshi: yes
18:25:47 <roshi> awesome
18:25:55 <threebean> http://infrastructure.fedoraproject.org/infra/db-dumps/
18:26:01 <threebean> roshi: that snapshot is updated nightly
18:26:06 <roshi> awesome
18:26:31 <threebean> so, we have our Fedora Infrastructure deployment and there are others at debian and data.gouv.fr.
18:26:53 <roshi> wow, 5 gigs of data in it already...
18:26:59 <threebean> roshi: that's compressed, too ;)
18:27:04 <roshi> haha
18:27:19 <threebean> you should be able to get your local box subscribed to both the Fedora infra and the debian buses by uncommenting a line in your local /etc/fedmsg.d/endpoints.py file.
18:27:32 <threebean> release-monitoring.org and pagure.io, although they are run by us, have "their own" buses.
18:27:42 <threebean> but we loop their feeds back into the Fedora bus to make things convenient.
18:27:44 * Corey84 back
18:27:55 <threebean> also -- centos infra started discussing adopting fedmsg in the last few months.
18:28:02 <threebean> In the news, there was a talk on fedmsg data at the Spark Summit just recently:
18:28:04 <threebean> http://chapeau.freevariable.com/2015/06/summit-fedmsg.html
18:28:10 * nirik notes #fedora-fedmsg has a (mostly) live stream of our fedmsgs (with some of the noiser ones filtered).
18:28:21 * threebean nods
18:28:38 <threebean> roshi: if you're looking at trawling through all that data, it could be handy to have the list of topics and example messages from the docs which can be found here:
18:28:40 <threebean> http://fedora-fedmsg.readthedocs.org/en/latest/topics.html
18:29:07 <roshi> sure
18:29:10 <threebean> for kicks, we have a visualization of some of the earlier activity from the bus (2012)
18:29:13 <threebean> http://threebean.org/so-i-turned-the-fedmsg-data-into-a-git-log-and.webm
18:29:15 * Corey84 adds to channels
18:29:39 <threebean> you should be able to generate your own modern version of that with the script in this repo
18:29:41 <threebean> https://github.com/ralphbean/fedmsg2gource
18:30:04 <nirik> might be cool to do one for this week with the mass rebuild going
18:30:12 <nirik> if someone wants a fun project
18:30:16 <threebean> agreed :)
18:30:42 <Corey84> no time here but would be interested
18:31:05 <threebean> so, if you're interested in how fedmsg is deployed in our infra, the majority of the config is held here:
18:31:07 <threebean> http://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/fedmsg
18:31:47 <threebean> so, in a system like this, services have to know about each other.  in a broker-centered setup, the broker manages introducing everybody to everybody...
18:32:01 <threebean> .. so, as we've grown over time, our configs have gotten a little hairy
18:32:09 <threebean> http://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/fedmsg/base/templates
18:32:20 <threebean> in particular, the "endpoints" files are a lot to wade through.
18:32:28 <Corey84> the intent is to kick the broker then?
18:32:44 <threebean> Corey84: right.  no broker to crash and then bring down the whole infrastructure with it.
18:33:04 <threebean> one of the things I've been working on recently, is trying to automatically generate our fedmsg configs from our ansible inventory definition
18:33:15 <Corey84> makes sense  BUT  how would they stay synced  manual  sync  on connecting?
18:33:33 <Corey84> assuming such a crash
18:33:46 <threebean> Corey84: oh, I guess I don't understand the question.  can you rephrase?
18:34:43 <lmacken> the hubs have the ability to query datanommer to see if they missed anything. I think that is what you are asking?
18:34:54 <Corey84> assuming a crash (one that in broker  sys  would crap out everything in the new system what  sync / re-sync  mechanism is there)
18:35:07 <Corey84> indeed thanks  lmacken
18:35:11 <threebean> ah, yeah.  that's it.  :)
18:35:59 <Corey84> so the hub would essentially call home to the voiceamil machine to check for  missed alerts?
18:36:10 * threebean nods
18:36:14 <Corey84> ah cool
18:36:36 <threebean> yeah - in our case, the badges.fedoraproject.org awarder daemon checks for its last message when it restarts and then asks datanommer for all the messages between then and now.
18:36:46 <threebean> so it can award badges to people it missed while it was down.
18:36:57 <Corey84> seems it would allow for offline updating  too and  fetching /pushing new stuff  in that fashion
18:37:23 * threebean nods
18:37:56 <threebean> as for new things being built on top of fedmsg, the #fedora-apps team is working on at least two
18:38:32 <Corey84> so bootstrapping in essentially for new stuff
18:38:35 <threebean> there's a new statscache daemon we've been working on that should allow quicker access to statistics that you'd have to compute yourself from big datagrepper qureies
18:38:37 <threebean> http://threebean.org/blog/statscache/
18:39:06 <threebean> and there's the new fedora-hubs project that people are collaborating on over in #fedora-hubs
18:39:38 <threebean> it has a smart-cache mechanism driven that will be driven by fedmsg
18:39:39 <lmacken> there's also tool called fedmsg-notify that lets you get fedmsg desktop notifications ;) http://lewk.org/blog/fedmsg-notify
18:39:39 <threebean> https://pagure.io/fedora-hubs/blob/develop/f/README.rst#_131
18:39:48 <threebean> http://fedoraproject.org/wiki/Fedora_Hubs
18:39:57 <threebean> oh yeah - the desktop notifications are amazing :p
18:40:06 <threebean> lmacken++
18:40:19 <usernkey> Hi guys, This is my first meeting. I introduced myself a couple of weeks ago in the mailing list and I would like to know if I can get the apprentice access so  I can start exploring the infrastructure.
18:41:15 <nirik> usernkey: welcome. Sure we can get you added...
18:41:23 <jcvicelli> Me 2
18:41:52 <nirik> see me in #fedora-admin after the meeting.
18:42:00 * nirik goes back to listening to threebean on fedmsg
18:42:05 <usernkey> thx
18:42:16 <threebean> I don't really have anything else prepped.
18:42:20 <threebean> if anyone has other questions?
18:42:25 <Corey84> +1 fedmsg-notify sweet  option I use it
18:42:29 <threebean> dreams?  plans?
18:42:53 <nirik> now that we have a pretty sold fedmsg we should try and go back and replace things we have now that are cron jobs.
18:42:58 <nirik> see if we can reach 0 cron job
18:43:10 <nirik> (or very low anyhow)
18:43:17 <threebean> would be cool, yeah.
18:43:41 <nirik> one thing we talked about ages and ages ago... was allowing users to submit messages...
18:43:44 <threebean> as we think of them, filing tickets is the best way.
18:43:53 <nirik> is that still on the roadmap? or no longer?
18:44:11 <threebean> eh, no longer.  but only because the use-case isn't clear, at least to me.
18:44:46 <threebean> if we have a solid use case, I think we could do it with a HTTP -> fedmsg bridge service, kind of like how github2fedmsg works.
18:44:57 <nirik> well, the case I guess is trying to get users more involved. So, if we have a fedmsg from some user that they run foo all the time or update foo in testing we can offer them foo related stuff...
18:45:09 <nirik> but yeah, needs a lot more thought.
18:45:32 <nirik> and probibly talking to the workstation folks (at least I think it's more a workstation thing than a server or cloud)
18:45:47 <threebean> yeah - the way we kind of do that now is that they 1) update foo in testing and then 2) provide karma feedback in bodhi and *then* they get a fedmsg message published and a badge and hooray!
18:46:20 <nirik> yeah.
18:46:30 <nirik> perhaps that was a poor example. :)
18:46:33 <threebean> nirik: who should I ping in workstation land?  any suggestions? :p
18:47:16 <nirik> I guess it would be more: 1) user uses foo a lot (launches, windows? dunno), 2) we have a updates-testing one, 3) we offer to let them test it if they want, 4) they do and provide karma
18:47:22 <randomuser> oh, if you're taking http->fedmsg questions, is zanata messaging getting any traction?
18:47:50 <nirik> threebean: not sure. I guess I can try and come up with a more articulate proposal and run it by you and then them?
18:47:55 <threebean> randomuser: yeah!  we have 1/2 of it solved.
18:48:01 <randomuser> nice
18:48:05 <threebean> nirik: sounds good :)
18:48:38 <threebean> randomuser: at our request, they added a webhooks interface (like github has) and it works on a local test with https://github.com/fedora-infra/zanata2fedmsg
18:48:58 <threebean> but.. they forgot to cryptographically sign their messages.  so, we can't stand it up because it would open our bus to abuse.
18:49:04 <randomuser> heh
18:49:07 <threebean> so, they're adding that feature.. and then we can play.
18:49:23 <randomuser> awesome.  big news for translators, nice work!
18:49:31 <threebean> woot woot
18:49:49 <Corey84> eta on the stand up?
18:49:50 <puiterwijk> threebean: can you tell me with whom you're in contact? I've also got in contact with some of the Zanata devs recently
18:50:08 * threebean finds the bz ticket
18:51:20 * threebean waits and waits on bz search
18:51:36 <threebean> https://bugzilla.redhat.com/show_bug.cgi?id=1213630
18:52:10 <threebean> we're just waiting on that.. then we could deploy, I dunno, a month afterwards.  just need to tweak our bridge software for their changes, setup a node, etc..
18:52:26 <nirik> cool
18:52:46 <puiterwijk> threebean: cool, thanks for the link
18:53:00 <nirik> ok, anything else? shall we go to open floor?
18:53:16 <nirik> thanks much threebean!
18:53:32 <nirik> #topic Open Floor
18:53:38 <nirik> anyone have any items for open floor?
18:53:42 <kdas_> nirik, I am looking at https://fedoraproject.org/wiki/Changes/Two_Week_Atomic
18:54:00 <lmacken> threebean++
18:54:01 * nirik nods.
18:54:22 <kdas_> nirik, to have a full test system to test the atomic/cloud images, we will need a baremetal box to run the tool.
18:54:33 * nirik sighs.
18:54:46 <kdas_> nirik, just wanted to inform that, and get your thoughts about how to skip that :)
18:54:54 <nirik> well, thats a major pain.
18:54:57 <Corey84> much appreciated threebean
18:55:04 <Corey84> very informative
18:55:05 <nirik> we don't have random idle hardware sitting around doing nothing. ;)
18:55:13 <nirik> and asking for such out of the budget cycle is not fun
18:55:14 <kdas_> nirik, basically I will have to run qemu-kvm to boot and run some images.
18:55:32 <kdas_> nirik, I know, that is why I am asking how to go ahead with this.
18:55:41 <nirik> kdas_: can it be done as part of a koji plugin?
18:56:06 <kdas_> nirik, can be, but I really do not want take down koji accidentally
18:56:34 <nirik> sure. I guess open a discussion about it on the releng list?
18:56:35 <kdas_> * want to :)
18:56:39 <kdas_> nirik, Yup.
18:56:44 <kdas_> nirik, thanks for the input.
18:56:46 <nirik> and perhaps folks there can suggest the best way forward
18:56:59 <Corey84> opening in releng sounds good
18:57:02 <nirik> it might be possible to get a machine, but it will be painfull I am sure.
18:57:06 <kdas_> nirik, also do you if there is any nested virt which is good enough,
18:57:16 <kdas_> * do you know
18:57:24 <nirik> last I tried nested virt it was completely unusable. ;(
18:57:33 <Corey84> ^^ same
18:57:40 <nirik> but it was a while back. ;)
18:57:43 <Corey84> but that  was > 6 months ago
18:57:48 <Corey84> may be better now
18:58:26 <nirik> ok, anything else for open floor?
18:58:31 <nirik> thanks for bringing it up kdas_
18:59:00 <puiterwijk> kdas_: I've heard it might work reasonable nowadays
18:59:07 * randomuser is wrapping up an RFR ticket right now
18:59:08 <puiterwijk> at least in some situations
18:59:29 <dgilmore> nirik: kdas_: we should be able to use nested virt to do automated testing
18:59:32 <puiterwijk> kdas_: I can forward you to the person that told me that after the meeting, he might have ideas
18:59:47 <nirik> well, we can give it a go...
19:00:06 <dgilmore> its apparenly improving rapidly
19:00:27 <dgilmore> kdas_: we can talk about it in person next week
19:01:16 <nirik> ok, thanks for coming everyone.
19:01:18 <nirik> #endmeeting