18:00:21 <nirik> #startmeeting Infrastructure (2016-06-02) 18:00:21 <zodbot> Meeting started Thu Jun 2 18:00:21 2016 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:21 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:21 <zodbot> The meeting name has been set to 'infrastructure_(2016-06-02)' 18:00:21 <nirik> #meetingname infrastructure 18:00:21 <zodbot> The meeting name has been set to 'infrastructure' 18:00:21 <nirik> #topic aloha 18:00:21 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore threebean pingou puiterwijk pbrobinson 18:00:21 <nirik> #topic New folks introductions / Apprentice feedback 18:00:21 <zodbot> Current chairs: abadger1999 dgilmore lmacken nirik pbrobinson pingou puiterwijk relrod smooge threebean 18:00:26 * pingou here 18:00:29 <dgilmore> hi 18:00:32 * skrzepto is here 18:00:37 * puiterwijk here-ish 18:00:38 * lousab here 18:00:41 * aikidouke present 18:00:45 * pcreech here 18:00:51 <munjeli> here 18:01:11 * tflink is here 18:01:20 <nirik> any new folks like to give a short one line introduction of themselves? 18:01:39 <jflory7> .hello jflory7 18:01:40 <zodbot> jflory7: jflory7 'Justin W. Flory' <me@justinwflory.com> 18:01:43 * devyani7 is here too 18:01:49 <devyani7> .hello devyani7 18:01:50 <zodbot> devyani7: devyani7 'Devyani Kota' <devyanikota@gmail.com> 18:02:39 <odin2016> Here 18:02:41 <odin2016> Ish 18:02:45 * cverna here 18:02:58 <munjeli> I'm new. I'm a devops engineer in seattle. wanted to make some infrastructure yay! 18:03:09 <nirik> welcome munjeli. :) 18:03:28 <nirik> were you more interested in sysadmin type things? or application devel? or both? 18:04:01 <jflory7> Welcome, munjeli :) 18:04:01 <munjeli> nirik: I would like to pick up that easyfix ticket on building a script for infrastructure mapping 18:04:30 <munjeli> Mostly I dev, I can crawl servers too, but I came into infrastructure through devops more than sysadminning 18:04:32 <nirik> munjeli: sure. I think linuxmodder was working on that, you might try and get a hold of him and see if you can colaborate. 18:04:41 <munjeli> ok 18:04:59 <nirik> do feel free to ask questions, etc. ;) 18:05:12 <nirik> any other new folks? or shall we move on to status/info dump? 18:05:39 <smooge> info dump 18:05:42 <nirik> #topic announcements and information 18:05:42 <nirik> #info We are now in Fedora 24 Final freeze - everyone 18:05:42 <nirik> #info fix_arm_soc.yml playbook ready for use, see it for docs - kevin 18:05:50 <skamath> .hello skamath 18:05:51 <zodbot> skamath: skamath 'Sachin S Kamath ' <sskamath96@gmail.com> 18:05:53 * skamath is late 18:05:57 <nirik> thats all that was in gobby. ;) anything else folks would like to note ? 18:05:59 <jflory7> o/ 18:06:06 <skamath> jflory7: o/ 18:06:07 <pingou> #info new pagure deployed just before freeze 18:06:07 <nirik> no worries skamath. welcome 18:06:13 <pingou> (w/ a bugfix right after) 18:06:20 <skamath> thanks nirik :) 18:06:24 <puiterwijk> #info two new Basset releases to add trac attachments and user blocking 18:07:08 <nirik> the trac user blocking isn't live yet tho is it? or is it? 18:07:22 <skamath> puiterwijk++ I see spam storm in #fedora-fedmsg 18:07:45 <puiterwijk> nirik: no, I was about to bring it live whe nthis meeting started 18:07:57 <nirik> yeah, the spammers are hitting hard the last few days. Very sad they waste so much time and energy on spamming 18:08:07 <puiterwijk> Basically, the trac plugin is updated and a script to grant permissions is running. 18:08:11 <pingou> and us as well 18:08:13 <pingou> puiterwijk++ 18:08:13 <nirik> great. 18:08:15 <jflory7> I thought it was fascinating they're trying to hit WordPress with new posts 18:08:24 <puiterwijk> There's new permissions for this: ANTISPAM 18:08:28 <skamath> lol, innovative. 18:08:30 <pingou> fascinating isn't quite the word I'd use :) 18:08:35 <puiterwijk> jflory7: if they are, we might want to integrate Basset there too... 18:08:35 <jflory7> Heh, yeah, true 18:08:55 <nirik> all the wordpress sites moderate comments right? 18:08:58 <jflory7> puiterwijk: Wouldn't be a bad idea. I opened the CommBlog to find two new drafts that were spam posts last week. They can't publish it, but we still have to clean up. 18:09:02 <jflory7> Comments are moderated, yeah. 18:09:03 <puiterwijk> As far as I know, yes. 18:09:16 <puiterwijk> jflory7: you mean new posts? 18:09:20 <nirik> ask also moderates new posters and they ave hit there too. 18:09:20 <jflory7> puiterwijk: Yeah, literal posts. 18:09:25 <sayan> .hello sayanchowdhury 18:09:26 <zodbot> sayan: sayanchowdhury 'Sayan Chowdhury' <sayan.chowdhury2012@gmail.com> 18:09:30 <puiterwijk> Huh, okay... 18:10:01 <nirik> puiterwijk: oh, speaking of wordpress, did we ever get one of them using that CDN? we should get something up on it soon or they might decide we don't want it. ;( 18:10:10 <puiterwijk> Also, I've been thinking about making Basset a more integral part and blocking them at the proxies. I do not yet have clear how I would do that though 18:10:35 <puiterwijk> nirik: It's supposed to be configured for Magazine. 18:10:39 <skamath> puiterwijk: Maybe the IP's? 18:10:49 <nirik> cool. Will check the dashboard when I have a chance. 18:10:54 <puiterwijk> skamath: well, the IPs change a lot. 18:10:59 <pingou> tor? 18:11:02 <skamath> puiterwijk: I found this the other day : http://www.village-idiot.org/spammers 18:11:05 <puiterwijk> nirik: we should work with Ryan to make sure it works with our theme 18:11:10 <skamath> It keeps updating 18:11:13 <puiterwijk> pingou: no, VMs at various places 18:11:17 <pingou> k 18:11:41 <puiterwijk> skamath: yeah, the problem with most of hte IPs I've seen is that they're from all kinds of home connections, but also things like AWS and digitalocean. 18:11:45 * threebean is here 18:11:49 <puiterwijk> And blocking those would do a lot of harm for certain things 18:12:06 <nirik> anyhow, shall we go on to discussion items? or is there anything more for status/info? 18:12:36 <smooge> summary: it is painful. it hurts. patrick needs sleep 18:12:50 <jflory7> smooge++ 18:13:03 <pingou> smooge++ 18:13:14 <nirik> #topic supybot-gribble fork: https://github.com/ProgVal/Limnoria plans - kevin 18:13:42 <nirik> so, our friend zodbot is a supybot-gribble. It's pretty much dead upstream.... but I found a fork that seems active... 18:13:58 <nirik> I want to look at getting packages for it in and moving zodbot to it. 18:14:40 <jflory7> +1 for Limnoria. I know other Linux projects (e.g. Arch Linux) use this for their bot now too, and it seems pretty active upstream. I've seen positive things about it, 18:14:44 <nirik> It can actually work fine with python3, but to do that all plugins would also need python3 versions 18:15:03 <nirik> and for epel7 it would need a bunch of new python3 packages. 18:15:21 <nirik> I know for sure the koji plugin imports koji (so it would fail in python3) 18:15:43 <nirik> so I guess we just stick with python2 for now? or python2 for epel7 and python3 for fedora? 18:16:07 * pingou is fine with py2 for now 18:16:17 <jflory7> I think py2 makes sense for EPEL 18:16:38 <nirik> we still do need one package for python2 in epel7, but thats it... 18:17:59 <nirik> they have SocksiPy-branch in requirements, but I think we can use python-SockyiPy 18:18:58 <nirik> or rather python-pysocks 18:19:30 <nirik> anyhow, I will look at submitting for review, then we can update stg and do some testing. 18:19:43 <nirik> any obections or further comments on that plan? 18:20:02 * jflory7 would be happy to help test 18:20:19 <nirik> cool. I can make a call for testing once it's in stg... 18:20:27 <odin2016> I like it. Might be willing to help if needed. 18:20:40 <odin2016> All for supported things. ;) 18:20:48 <decause_pycon> .hello decause 18:20:49 <zodbot> decause_pycon: decause 'Remy DeCausemaker' <decause@redhat.com> 18:21:02 <nirik> #topic split notifications from scm-commits to new list? - kevin 18:21:24 <nirik> so, our scm-commits list also gets notifications for all acl changes, etc... 18:21:36 <nirik> and some folks are finding this a bit much recently 18:21:56 <nirik> we had things like perl maintainer dropping acls on all branches for thousands of packages... etc 18:22:19 <nirik> so, the idea has been floated of moving them to another list. scm-notifications? 18:22:26 <pingou> sounds good to me 18:22:33 <pingou> that's just a config change for pkgdb 18:22:36 <nirik> also want to get releng's perspective here... dgilmore: any thoughts on that? 18:22:49 <pingou> (together with the email override) 18:23:20 <nirik> I thought it was a change in notifications app? 18:23:40 <pingou> oh could be 18:23:51 <dgilmore> nirik: a seperate list would be good 18:24:01 <dgilmore> nirik: or just a better interface on datagrepper 18:24:27 <nirik> probibly we should wait until after freeze, but we can get things lined up... make new list, figure out where changes need to be made, etc 18:24:34 <dgilmore> where you can search for things with greate ease 18:24:48 <nirik> that would be great, but probibly beyond our scope right now. ;) 18:24:53 <dgilmore> indeed 18:25:46 <nirik> so, if no objections we can move forward with the new list idea for now. 18:26:28 <nirik> #info will look at splitting notifications traffic to another list 18:26:38 <nirik> threebean: you around to talk some modularity? 18:27:49 <nirik> or wait, we can do some apprentice office hours first 18:27:56 <nirik> #topic Apprentice office hours 18:28:14 <nirik> any apprentices looking for things to do? have questions on setup or anything? 18:28:29 <lousab> i have two questions 18:28:38 <nirik> we probibly need to go over the easyfixes again and make some more 18:28:42 <nirik> lousab: fire away. 18:28:47 <lousab> watching at #5290 i'd like to ask if it's possible also to create a visual ascii or html map like infratructure map of the servers? 18:29:06 <odin2016> Not I... Should be digging in again soon. (tm) 18:29:06 <nirik> .ticket 5290 18:29:08 <zodbot> nirik: #5290 (Generate infrastructure map) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/5290 18:29:52 <nirik> I would think either or both would be ok. as long as we had a script to make them 18:30:08 <nirik> note that munjeli also wanted to look at this too. 18:30:17 * pcreech is looking for some low hanging coding/development fruit to get back into things 18:30:20 <munjeli> yup I did 18:30:35 <lousab> nirik: ok 18:30:36 <munjeli> At my last job I built an infrastructure mapper for aws with d3 18:30:59 <nirik> one useful file for this is on batcave01 in /var/log/virthost-lists.out it has all our virthosts and what guests are running on those 18:31:05 <lousab> munjeli: cool :) 18:31:10 <munjeli> nice, thx 18:31:11 <jflory7> No questions from me at the moment. 18:31:29 <nirik> you can also find that out via the ansible git repo... every hosts variables has what their datacenter and virthost is 18:32:10 <lousab> nirik: the second question is: is there a possibility to have apprentice meetings to share "how to" and things to do? do we have also mentors or sponsors ? 18:32:45 <nirik> well, sure but the last time that was suggested, we just agreed to have a section of every meeting for such things. ;) 18:32:51 <nirik> this very section. 18:33:07 <lousab> ok 18:33:25 <nirik> we don't really have mentors, because it seems most of us don't have time to do 1 on 1 mentoring all the time and also all the things we need to get done. 18:34:01 <nirik> so we try and just help folks with questions as much as we can... I wish we did have a more formal mentoring setup, but it's hard to do with our personpower 18:34:16 <munjeli> nirik: is there a place we should ping, the list? if we need guidance? 18:34:29 <nirik> but everyone I think is happy to answer questions anytme they can. 18:34:44 <nirik> munjeli: sure, thats fine or #fedora-admin usually has some active folks 18:34:47 <jflory7> A lot of folks hang out in #fedora-admin and #fedora-apps too, which can be a good place for quick questions 18:35:02 <lousab> ok thanks all 18:35:05 <nirik> and also #fedora-noc for sysadminy things 18:35:15 <munjeli> cool. Is there onboarding documentation beyond what we've been pointed to? particularly around setting up a dev env? 18:35:58 <nirik> unless it's a outage/crisis or release day, it's usually fine to chime in with questions if you see people talking about something that interests you too... thats often a good way to pick up some work.. 18:36:19 <nirik> well, most of our apps have their own docs on setting up things... 18:36:40 <nirik> for sysadmin we have CSI, but it's a bit high level / fonal. 18:36:42 <nirik> formal 18:36:51 <munjeli> cool. Is there a pre-packed vm or anything for infrastructure dev? 18:37:34 <nirik> nope, but most of our stuff is packaged in fedora/epel, so you can install via packages or if not, virtualenv's and pip, etc. 18:38:05 <nirik> There was some talk about making containers or the like avaiable, but not sure what happened with that 18:38:09 <munjeli> ok cool. Is there test coverage for any of the ansible stuffs? What do you use? 18:38:17 <nirik> likely we didn't find much time. :) 18:38:27 <lousab> nirik: for who is interested in sysadmin work you suggest to stay tuned on fedora-noc and fedora-admin? what other things we can do? 18:39:01 <nirik> munjeli: there's not much, just syntax checking... I'd be open to some kind of testing, but not sure how it would fully work out. 18:39:19 <nirik> lousab: yep. And ask questions and also look at easyfix tickets. ;) 18:39:45 <lousab> nirik: yeah :) 18:40:23 <nirik> munjeli: oh, we also do have a daily check/diff job that runs and mails reports 18:40:46 <munjeli> nirik: ok I'll just keep it in mind. I like to dev in a docker so I'll try to make a container to share. 18:41:01 <munjeli> Or something sharable. 18:41:14 <nirik> sure. 18:41:33 <nirik> ok, anything else from anyone? 18:41:41 <nirik> threebean: you around now? 18:42:21 <devyani7> alright, seems like I have a lot of homework to do. no questions from me this week. thanks :) 18:43:43 <jflory7> Me too. :) 18:43:45 <threebean> oof, sorry. just got back. 18:43:57 <threebean> pycon sprints just kicking off here. 18:43:58 <nirik> no worries 18:44:14 <nirik> you still want to talk modularity? 18:44:22 <threebean> sure. lemme pull up the page.. 18:44:31 <threebean> https://fedoraproject.org/wiki/Modularization/Infra 18:44:31 <nirik> #topic Learn about: modularity from a high level - threebean 18:45:10 <threebean> hey, so, like I mentioned last week there's a number of changes we think we're going to need to make to infra pieces to make modularity possible 18:45:54 <threebean> one of the things we've noticed is that if we allow rpms to be built and shipped in modules with these "independent lifecycles", then we're going to have an explosion of complexity if we don't mitigate it somehow. 18:46:37 <threebean> in that infra page there's some discussion about what it would take for a packager to backport a security patch on a package to all the modules that consume it (by hand) that shows some of that. 18:46:47 <nirik> thats always been a problem of different lifecycle plans. ;( 18:47:11 <threebean> so.. we're thinking that we're going to need to 1) limit the number of modules that will be supported with policy that we haven't figured out yet but 2) provide a ton of automation in the infrastructure to make what's left over still possible for our human brains to handle. 18:47:19 * threebean nods 18:47:26 <threebean> nirik: totally. 18:48:00 <threebean> I guess I can walk through (real quick, since we're short on time) some of the pieces? 18:48:07 <nirik> sure. 18:48:21 <threebean> dist-git is unsurprising I think. we want to store module definitions there just like we're planning to do for dockerfiles and taskotron checks. 18:48:22 <nirik> is there any targeting for this stuff? or is it too early to tell? 18:48:29 <threebean> targetting? like, timelines? 18:48:32 <threebean> if so, yes. 18:48:32 <nirik> yeah... 18:48:50 <threebean> by f25 we want to "be able to produce f25 as one big module" 18:48:56 <threebean> but not actually produce the real f25 that way. 18:49:06 <threebean> so, off in some silly VMs on the side, we want the proof of concept to be there. 18:49:22 <threebean> then we can argue over it and vet it and hopefully do f26 as a first real composition of the distro in terms of modules. 18:49:58 <tflink> isn't the f25 target "unofficial" 18:50:00 <nirik> ok, thats pretty short ramp. ;) but it's good to have goals 18:50:03 <threebean> (would love to hear if anybody thinks that's unreasonable..) 18:50:05 * threebean nods 18:50:13 <tflink> in the sense that it's not released as a fedora deliverable? 18:50:19 <nirik> that could be a staging setup? 18:50:24 <threebean> tflink: exactly. that's what I meant by "on the side" 18:50:37 <threebean> nirik: it could be. I imagine 50% of it will be in staging by then, and the other 50% in cloud nodes. 18:50:44 <tflink> threebean: yeah, just making sure that's clarified 18:50:47 <nirik> ok. 18:51:07 <nirik> do we need to look at beefing up our cloud/virthosts before this? or shouldn't be too bad... 18:51:33 <threebean> shouldn't be too bad. much of it could even live on modularity team members personal VMs in other third party clouds. just development-style stuff. 18:51:42 <nirik> ok. 18:52:02 <threebean> one of the most interesting pieces of the whole thing is the loop in the middle of the diagram between koji, taskotorn and this 'orchestrator' process. 18:52:16 <threebean> "orchestration" is kind of overloaded, so we may need to change the name.. but the idea is to orchestrate rebuilds. 18:52:43 <threebean> so when we have a glibc vuln that gets patched, we can auto-rebuild all the modules that pull that in, and all the other modules that pull those modules in, and all the compose artifacts that pull in those modules 18:53:00 <threebean> and "voila", when we go to create the "compose" with pungi, all of the artifacts and repos are already done and ready to be shipped. 18:53:24 <threebean> it involves moving much of the functionality currently in pungi out into the orchestrator, which turns it into a kind of "ad hoc" pungi. 18:54:06 <threebean> the hopes here is that we do autoqa on those artifacts earlier, with feedback to package and module developers earlier. 18:54:23 <threebean> all before we do formal requests to releng humans. 18:55:09 <threebean> one last interesting part to mention is how we're looking at doing module builds in koji. 18:55:10 <linuxmodder> .hello linuxmodder 18:55:11 <zodbot> linuxmodder: linuxmodder 'Corey W Sheldon' <sheldon.corey@openmailbox.org> 18:55:16 * linuxmodder is super late sorry 18:55:24 <threebean> modules will get their own build roots, separate from others. 18:55:30 <nirik> np. welcome linuxmodder 18:55:39 <threebean> which hopefully gets us out of the problem of having one great big giganto build root shared by all packages. 18:55:55 <threebean> that's one half of the "independent lifecycles" problem that we think we have a good idea of how to solve. 18:56:10 <nirik> threebean: so each module might have it's own target/tags, etc? not sure how that will scale. 18:56:21 <threebean> the other half is whether to do parallel installability of these things, or just relegate them to delivery via containers. that's unsolved for the moment. 18:56:40 <dgilmore> threebean: I have not heard anything about how things will build 18:56:54 <dgilmore> threebean: nor the associated costs of them 18:56:57 <threebean> nirik: yeah. it would be something like 100 times the targets/tags we have now.. unmaintainable by humans. we'd require automation of that for it to be possible. 18:57:35 <threebean> dgilmore: psabata/contyk is working on experimenting with that now. we don't have a working prototype of those builds in koji yet, but that's the thinking. 18:57:45 <threebean> ...special targets/tags for each module. 18:58:30 <dgilmore> threebean: thats going to take a lot of management, and its going to have to be automated, today all teh tag cretaion is manual, and the syncing of packages and owners is semi manual 18:58:35 * threebean nods 18:58:39 <nirik> so then a patched glibc would build as a rpm, then koji could see that 10 modules buildroots need that tagged glibc and would rebuild those and then the superpungi would rebuild the actual modules? 18:58:47 <linuxmodder> threebean, so each would be a chroot of its own ? or am I misunderstanding 18:58:53 <threebean> nirik: more or less, yes. 18:59:13 <threebean> linuxmodder: yeah, each with its own repo for its special snowflake buildroot. 18:59:44 <linuxmodder> that seems coool but highly voilate at scale I fear too 19:00:08 <pingou> mirror wise, do we split the modules or will all the rpms be in the same folder? 19:00:14 <threebean> dgilmore: yeah. we simply cannot allow it to be manual anymore. we'd need to create new constellations of tags/targets for new modules once they pass "module review and approval" (kind of like package review we do now, but may require some kind of further signoff since the cost of a module is much higher. perhaps fesco involvement? we dunno..) 19:00:16 <linuxmodder> however in theory would that not also cut build times and possibly the loads for such builds down massively? 19:00:23 <pingou> or different folder with hardlink into a main one? 19:00:39 <dgilmore> threebean: indeed, there is a high cost to newRepos and buildroots 19:00:44 <dgilmore> that has to be made lower 19:00:50 <nirik> this kind of thing is always hard to see without a prototype to poke at. ;) 19:01:09 <threebean> pingou: I think in different folders with hardlink yes. we may not get much benefit from hardlinking if the rpms are different on a binary level (one built against one buildroot with another built against another). 19:01:25 <nirik> pingou: has to be split I would think. we don't want to assemble on the client... 19:01:26 <pingou> threebean: for similar versions indeed 19:01:27 <threebean> dgilmore: yes. that's part of the motiviation behind the koji tech debt cleanup we're working on now at pycon. 19:01:32 <dgilmore> threebean: given how thinsg work today I will push back really hard, we have to make the cost lower 19:02:04 <threebean> nirik: yes. but if we put too much work into a prototype without talking then we might end up all mad at each other down the road. thanks for entertaining the handy-wavy talk ;p 19:02:21 <nirik> threebean: sure. it's a balance. ;) 19:02:25 <threebean> dgilmore: agreed. 19:02:42 <nirik> but I think it could be pretty awesome come f26. ;) 19:02:44 <linuxmodder> so this would all be in stg or just on the builders? 19:02:47 <threebean> dgilmore: the performance-of-the-build-system is definitely on our minds. 19:03:05 <threebean> linuxmodder: half in stg and the other half on private VMs while we develop and try things. 19:03:34 <nirik> anyhow, we are over time now... 19:03:37 <threebean> another interesting problem (that we don't have answers to) is how do we store the EOL/SLA information for modules... 19:03:40 <threebean> oop - for another time then. 19:03:42 <linuxmodder> trying to think how to map this for the infra host mapping ticket with as much pre-planned spacing and templates so later its mostly fill in th eblanks 19:03:42 <nirik> threebean: thanks for talking about it. 19:03:50 <dgilmore> threebean: its not just performance 19:04:00 <nirik> linuxmodder: well, for now it doesn't exist, so don't worry about it. ;) 19:04:00 <threebean> (we currently store that information for the "distro release" in pkgdb. but if these module things are going to be independent it kind of breaks that model...) 19:04:46 <nirik> well, we may be able to extend pkgdb... or make a moduledb that does that 19:04:50 <threebean> dgilmore: the requirement to not-explode-the-complexity-onto-releng-or-qa-or-packagers is also on our minds. :p 19:04:53 <pingou> threebean: on pkgdb modules will always be rawhide only, right? 19:05:16 <threebean> pingou: yeah, that was our first thought. but some in modularity disagree about it. still being worked out I guess.. 19:05:17 <nirik> pingou: well, that seems artificial... 19:05:31 <pingou> nirik: as in? 19:05:36 <nirik> it's not really any 'fedora branch' ir could be using parts from several, no? 19:05:41 <dgilmore> threebean: well there is also disk cost, bandwidth cost, mirror cost 19:06:04 <dgilmore> pingou: I think the answer to that is no 19:06:38 <nirik> perhaps: module yyz would be/show 19:06:46 <nirik> 'module yyz's buildroot' 19:07:03 <nirik> as it's branch/whatever... and when that buildroot is retired it is too. 19:07:09 <nirik> dunno, too many possibilities. 19:07:14 <threebean> yeah yeah 19:07:35 <threebean> ok. I'll try to be back here next week too if we want to talk about it again. :) 19:07:58 <threebean> ther'es also a modularity working group meeting also on thursdays if infra wants to come :) 19:08:09 <pingou> what time is it? 19:08:09 <dgilmore> threebean: its tuesdays now 19:08:13 <threebean> oop! 19:08:17 * threebean is out of the loop 19:08:18 <dgilmore> 15:00 UTC 19:08:37 <tflink> threebean: well, it was just changed earlier today :) 19:08:38 * nirik has seen them, can try and attend... 19:08:39 <threebean> #fedora-modularization is the day-to-day chat place. 19:09:04 <dgilmore> https://apps.fedoraproject.org/calendar/modularity/2016/6/6/#m3924 19:09:07 <nirik> thanks for the info threebean! you want to talk again next week or play it by ear? 19:09:22 <threebean> nirik: let's say yes, and we can cancel if it doesn't make sense for some reason. 19:09:30 <nirik> sounds good. ;) 19:09:33 <nirik> #topic Open Floor 19:09:41 <nirik> anyone have any quick items for open floor? 19:09:57 <nirik> questions, comments, ideas, favorite types of bread? 19:10:06 <pingou> I have a large redesign of FMN's backend coming up 19:10:28 <threebean> I'm at the pycon sprints! people are just getting set up on fedora-hubs, and there's a crew of new people looking at the test suite for koji. :) 19:10:28 <pingou> things start to align, still fixing some issues but I start to see the end of the wood 19:10:32 <nirik> pingou: thanks for working on that. would be good to not get behind. 19:10:35 <threebean> pingou++ that's huge 19:10:52 <nirik> threebean: where is the hubs working going to happen? #fedora-hubs? 19:10:54 <pingou> threebean: I've been meaning to ask you, do you have an idea of the current rate? 19:11:07 <threebean> lmacken: ^^ yeah, #fedora-hubs for hubs chatter? 19:11:22 <threebean> pingou: not off hand, but we could try to measure it. collectd could give some idea. 19:11:23 * nirik also likes sourdough and nann. 19:11:26 <pingou> I'm getting about 1 sec/message on the core logic and the backends are pretty much immediate atm 19:11:35 <pingou> threebean: ok then I'll try locally 19:11:41 * threebean is all about the nann 19:11:45 * pingou hopes there is improvements 19:11:54 <threebean> home-made tortillas are also super good. 19:12:06 <pingou> earlier I got 10K messages digested in under 2h 19:12:27 * pingou likes threebean 's home-made tortillas 19:12:39 <threebean> :P 19:12:49 <threebean> pingou: with how many workers? if you add more, how does it scale? 19:12:53 * langdon just wandered by.. 19:12:58 <langdon> threebean++ 19:13:29 <pingou> threebean: curiously the number of workers doesn't seem to change much 19:13:38 <nirik> we had a spike recently up to 110k or so... 19:13:56 <pingou> nirik: yes but messages were arriving little by little no? 19:14:02 <pingou> not 110K in one go? 19:14:19 <threebean> pingou: hm, that is odd. 19:14:38 <pingou> threebean: I need to check all this still 19:14:43 <nirik> yes, the backlog got to 110k before it started to go down 19:15:08 <nirik> we are actually at about 18k backlog right now 19:15:20 <pingou> :/ 19:15:29 <puiterwijk> Right, part of that I think is caused by the COPR rebuild of rubygems... 19:15:45 <puiterwijk> I have a hotfix to ignore those (just like pypi when they were doing that). Will send that as an FBR in a few 19:15:55 <nirik> took about 5 days for that 110k spike to process down 19:16:28 <nirik> https://admin.fedoraproject.org/collectd/bin/graph.cgi?hostname=notifs-backend01.phx2.fedoraproject.org;plugin=fedmsg;plugin_instance=hub;type=queue_length;type_instance=FMNConsumer_backlog;begin=1463806220;end=1464411020 19:16:57 <pingou> nice :/ 19:16:59 <nirik> puiterwijk: also there was another flood of pkgdb acls notifications from someone last night I think... the orphaned a bunch of apckages or something 19:17:27 <pingou> yes (which triggered the ticket about scm-commits) 19:17:29 <puiterwijk> nirik: right. I guess we do wnat those to get through FMN though 19:17:52 <nirik> anyhow, anything else? we are way over... :) 19:17:52 <puiterwijk> We can also hotfix that stuff out, but not sure we want to 19:18:18 <nirik> yeah, we don't want to 19:18:24 <nirik> ideally we want to process fast. ;) 19:18:28 <pingou> agreed 19:18:46 <puiterwijk> go pingou! You can do it! :-) 19:19:05 <pingou> fingers crossed :) 19:19:19 <nirik> alright. Thanks for coming everyone. ;) Do continue in #fedora-admin, #fedora-apps and #fedora-noc. 19:19:24 <nirik> #endmeeting