18:00:25 <nirik> #startmeeting Infrastructure (2016-06-09) 18:00:25 <zodbot> Meeting started Thu Jun 9 18:00:25 2016 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:25 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:25 <zodbot> The meeting name has been set to 'infrastructure_(2016-06-09)' 18:00:25 <nirik> #meetingname infrastructure 18:00:25 <zodbot> The meeting name has been set to 'infrastructure' 18:00:25 <nirik> #topic aloha 18:00:25 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore threebean pingou puiterwijk pbrobinson 18:00:25 <zodbot> Current chairs: abadger1999 dgilmore lmacken nirik pbrobinson pingou puiterwijk relrod smooge threebean 18:00:26 <nirik> #topic New folks introductions / Apprentice feedback 18:00:30 <nirik> morning everyone. 18:00:37 <linuxmodder> .fas linuxmodder 18:00:37 <zodbot> linuxmodder: linuxmodder 'Corey W Sheldon' <sheldon.corey@openmailbox.org> 18:00:52 <munjeli> good morning 18:01:07 * lousab is here 18:01:11 * skrzepto is here 18:01:11 <lousab> hi all 18:01:18 <athos> hello :) 18:01:22 <athos> .fas athoscribeiro 18:01:22 <zodbot> athos: ribeiro 'Athos Ribeiro' <athoscribeiro@gmail.com> 18:01:32 <linuxmodder> packing the seats ...good to see 18:01:34 <jflory7> .hello jflory7 18:01:37 <zodbot> jflory7: jflory7 'Justin W. Flory' <me@justinwflory.com> 18:01:55 <nirik> any new folks around today that would like to give a short one-line introduction? 18:02:01 <athos> ! 18:02:22 <nirik> athos: feel free to chime in anytime. 18:02:31 <athos> sure: 18:02:48 <linuxmodder> tammy when she gets here too nirik 18:03:30 <tammy_> Hello everyone 18:03:37 <athos> My name is Athos, I'm a brazilian software engineer and CS graduate student. I am getting more involved with the project and would like to join the team :) 18:03:47 <nirik> morning tammy_. Care to give a short introduction of yourself? 18:03:54 <nirik> athos: welcome. 18:03:57 <jflory7> Hi athos, welcome :) 18:04:36 <linuxmodder> athos, what is your fasid? 18:04:41 <athos> Just became an embassador for LATAM and my 1st package is under review :). I also tried sending PRs for fedora-review and copr once or twice 18:04:42 <linuxmodder> tammy_, same to you 18:04:45 <athos> ribeiro 18:04:51 <athos> linuxmodder: ^ 18:05:10 <nirik> athos: great. Are you more interested in application development type tasks then? 18:06:20 <athos> nirik: I am interested both in app development and in infrastructure maintainance... I am currently learning how ansible works and have some experience with chef 18:06:37 <tammy_> Sure @nirik I am Tammy and I work as a helpdesk/junior system administrator I just started the junior sys administration role about six months ago and want to gain more experience in doing so 18:06:54 <nirik> athos: cool. See me after the meeting in #fedora-admin and I can get you setup in the apprentice group... 18:06:59 <linuxmodder> nice ..welcome tammy_ what is your fasid 18:07:07 <athos> thanks! :) 18:07:21 <tammy_> My fasid is tammyb5 18:07:42 <jflory7> Hi, welcome tammy_! 18:07:51 <nirik> tammy_: great! Sorry that many of us have been busy this week... I've seen you asking for assistance, but always at times I am busy or not around. :( But we will get you setup... never fear. 18:07:54 <tammy_> I also signed up for the apprentice group 18:08:05 <linuxmodder> and approved for it :) 18:08:17 <tammy_> No worries totally understand 18:08:36 <nirik> it's been spam fighting this last few weeks. ;( Pesky spammers 18:08:40 <linuxmodder> tammy_, on that note do oyu have a ssh key in fas setup yet (noticced no gpg keys mentioned in fas) 18:08:42 <nirik> ok, any other new folks? 18:09:04 <linuxmodder> seemed like no other new nicks to me 18:09:25 <tammy_> Wow 18:09:27 <nirik> ok, on to info and announcements... 18:09:35 <nirik> #topic announcements and information 18:09:35 <nirik> #info We are still in Fedora 24 Final freeze - everyone 18:09:35 <nirik> #info spam attacks ongoing. Many thanks to Patrick for cleaning them up via basset 18:09:35 <nirik> #info spam attackers have opened over 10,000 accounts which have been cleaned by basset 18:09:35 <nirik> #info Added ru lang to ask.fedoraproject.org - kevin 18:09:36 <nirik> #info faf and retrace services added in stg via proxy01.stg - kevin 18:09:38 <nirik> #info Notification bounces now going to /dev/null - kevin 18:09:42 <nirik> #info alt.stg.fedoraproject.org setup for websites - kevin 18:09:43 <linuxmodder> tammy_, worst case find me after meeting we can fix that pesky batcaveo01 issue 18:09:54 <nirik> anything else anyone would like to note or discuss from those? 18:10:04 <nirik> tammy_: also happy to help in #fedora-admin after the meeting. 18:10:52 <linuxmodder> when is alt goign strictly webisites ? 18:11:12 <linuxmodder> alt.fp.o that is not alt.stg 18:11:16 <nirik> should be at release day... 18:11:18 <munjeli> nirik: is the ongoing spamming long ongoing? Typical? 18:11:28 <nirik> which is not next week sadly (we slipped a week) 18:11:29 <linuxmodder> planned or actual GA was a n/g 18:11:38 <sayan> .hello sayanchowdhury 18:11:39 <zodbot> sayan: sayanchowdhury 'Sayan Chowdhury' <sayan.chowdhury2012@gmail.com> 18:11:45 <nirik> munjeli: started a few months ago... hasn't really let up much since then. ;( 18:12:10 <nirik> it's really pretty sad. People doing that for a living. ;( 18:12:30 <tammy_> Ok 18:12:59 <linuxmodder> brightside its helping us tighten up fas3 18:13:10 <nirik> we can't really lock things down too much or it would affect legit new contributors. ;( So, we have to mark and clean up... 18:13:18 <nirik> yep. :) 18:13:27 <munjeli> yah, it's a difficult problem. 18:13:28 <jflory7> Hopefully it will relent after a while. I wonder if it will fade off after the GA, if that's the reason for targeting. 18:13:32 * jflory7 shrugs 18:13:44 <linuxmodder> puiterwijk, you around later have a few ideas for basset 18:13:56 <nirik> jflory7: I don't think it's anything really other than we are somewhat open so they can spew their spam and hope search engines will pick it up. 18:14:06 <jflory7> True. 18:14:08 <nirik> they are also hitting bugzilla some. 18:14:10 * pingou late 18:14:15 <jflory7> pingou: o/ 18:14:16 <linuxmodder> jflory7, doubtful the attention is not seemingly GA centric from what I've seen 18:14:18 <patricku> linuxmodder: you can also file tickets. But let me know if you have ideas 18:14:18 <nirik> and I heard they are hitting kernel.org bugzilla too now. ;( 18:14:41 <nirik> anyhow. 18:14:49 <linuxmodder> and mozilla is getting traction from them too 18:15:04 <nirik> I didn't see any discussion items on the gobby. Did anyone have any? or shall we move on to apprentice open office hours... 18:15:13 <patricku> Again: if people know people in those communities under attack, feel free to send them my way 18:15:25 <linuxmodder> patricku, noted 18:15:54 <patricku> I'd be glad to help more people with setting up Basset or any other cross distro anti spam project 18:16:08 <nirik> sure. :) 18:16:30 <nirik> #topic Apprentice office hours 18:16:59 <munjeli> lousab and I are looking at the infrastructure mapping ticket and have questions 18:17:00 <nirik> any apprentices have questions or issues? or need things to work on ? perhaps we should hit the easyfix tickets and see if there's some that could use new folks working on them 18:17:09 <nirik> excellent. :) ask away. 18:17:10 <linuxmodder> munjeli, shoot 18:17:49 <munjeli> I would like to make a map that exposes useful information interactively, does this mean service dependencies, networking? 18:17:56 <munjeli> What is useful to the team? 18:17:58 <linuxmodder> my connection is getting unstable fyi advance apoligies if i d/c 18:18:19 <nirik> munjeli: well, IMHO I would start with something pretty basic at first. you can always add to it after. 18:18:20 <chakal> hi, i start few weeks ago and i need some guidance about what have to do 18:18:22 <linuxmodder> munjeli, useful information being what in your workflow 18:18:49 <nirik> ie, just datacenters, what machines are in each... 18:18:52 <linuxmodder> chakal, you looked at and done stuff on the Join page yet? 18:18:55 <linuxmodder> welcome btw 18:19:10 <chakal> yes 18:19:29 <munjeli> linuxmodder: I did this last time as an html page with d3 that showed routes, acls, and showed error conditions like black holes on AWS 18:19:29 <nirik> chakal: take a look at https://fedoraproject.org/easyfix/ and the fedora-infrastructure section... 18:19:35 <lousab> nirik: what do you think if we start mapping machines using the code of https://apps.fedoraproject.org/ 18:19:35 <chakal> i have logged on ssh server and already installed ansible 18:19:38 <linuxmodder> munjeli, I am busy next week but email still works too if you need a hand ( I was the one who originally handed that off to you) 18:19:38 <nirik> see if any of those tickets look like something interesting to work on 18:20:04 <chakal> yes y have fas accound and registered on infrastructure list 18:20:44 <nirik> lousab: you mean make the map use that same interface? 18:20:49 <lousab> yes 18:20:53 <linuxmodder> chakal, ssh key setup and / or easyfix tickets look good for you ? 18:21:01 <nirik> thats not a bad thought... :) 18:21:26 <linuxmodder> might get a littel busy but I like the idea 18:21:32 <nirik> it's simple, (json) and might work nicely for a map 18:21:41 <tammy_> Are there other things am apprentice can do 18:22:08 <linuxmodder> tammy_, what you have in mind? 18:22:09 <munjeli> Yes, I don't like the layout of the graphing there - I used a sankey diagram last time to show flow, but the graph is not authentic to most infrastructure diagramming. 18:22:15 <linuxmodder> there is plenty 18:22:25 <lousab> nirik: exactly and it should be quite simple to start from there 18:22:34 <linuxmodder> sankey diagram? 18:23:03 <lousab> munjeli: what if we change a little bit the graphic? 18:23:23 <munjeli> Yes, a sankey diagram is a dataviz that describes flow. But it was also too busy. I think authentically, we use things closer to circuit diagrams to map datacenters 18:23:28 <nirik> tammy_: well, the main thing is easyfix tickets (proposing solutions/patches), but also if you can hang out in irc or on the list there will often be discussions about apps or things and you can chime in that you would like to help out with those. Also we always need docs cleanup/help, and there's a few things listed on the main apprentice page as "quests" that anyone can work on. 18:23:58 <nirik> https://fedoraproject.org/wiki/Infrastructure_Apprentice#Longer_term_quests 18:24:04 <chakal> linuxmodder, I dont know what i have to do on easeyfix 18:24:06 <nirik> updating CSI variables and docs work 18:24:06 <munjeli> lousab: sure, a good first pass would be a graph diagram 18:24:47 <linuxmodder> munjeli, seen the infra_hosts file on batcave01? 18:24:51 <nirik> chakal: it depends on the ticket. Usually its something you can provide a patch for against the ansible repo... 18:24:52 <munjeli> linuxmodder: where is the best place to understand network and service dependencies? the data? 18:25:03 <munjeli> Yah, I have the list of hosts.. 18:25:18 <nirik> munjeli: nagios has some of that, and ansible has host/vhost dependencies and datacenter info 18:25:25 <linuxmodder> service dependencies ? as in what each needs or is used for? 18:25:30 <munjeli> Yup, nagios is what I want. 18:25:53 <linuxmodder> nagios01.phx2 18:26:00 <munjeli> Yeah, understanding what will fail and how. spoffiness, etc 18:26:01 <linuxmodder> should help ^ 18:26:36 <nirik> https://admin.fedoraproject.org/nagios/ 18:26:39 <munjeli> What ports are open, how the infrastructure is scoped (private and public subnets) 18:26:42 <nirik> and https://admin.fedoraproject.org/nagios-external 18:26:56 <linuxmodder> munjeli, take read at infrastructure.fedoraproject.org/infra/docs for the SOPs on those 18:26:58 <nirik> anyone should be able to login there. ;) 18:27:29 <munjeli> Even if we build for just seeing the basics, we should be open to extending for monitoring interactively, sec team loves stuff like this. 18:27:38 <nirik> BTW, nagios also has a map, but not all hosts are in nagios... and the map is... not so great 18:27:44 <munjeli> Gtk 18:27:49 <linuxmodder> munjeli, indeed we sec folks do :) 18:28:08 <lousab> well let's start :) 18:28:09 <munjeli> :) 18:28:17 <linuxmodder> nirik, does that nagios map have a undate script like ambassador map? 18:28:36 <munjeli> where is information on how to repo and code review? 18:28:37 <nirik> it generates it live from the data it has 18:29:16 <linuxmodder> looks like something out of the matrix :) 18:29:27 <linuxmodder> munjeli, repo what ? 18:29:37 <linuxmodder> the nagios and ansible stuff? 18:29:40 <nirik> munjeli: well, if this is goign to be like apps, you can look at the ansible roles/apps 18:29:41 <munjeli> The code for this map.. 18:29:52 <munjeli> where should we put it? 18:30:03 <linuxmodder> batcave01 ? 18:30:17 <linuxmodder> or somewhere on stg 18:30:18 <munjeli> but where do you git? is github? 18:30:25 <nirik> I'd say a patch against ansible in the ticket... call it roles/netmap or something? 18:30:33 <linuxmodder> munjeli, some things you could 18:30:38 <tammy_> I will work on the docs and questions 18:30:40 <linuxmodder> but we have pagure :) 18:30:44 <nirik> munjeli: ssh to batcave01, git clone ansible 18:30:50 <munjeli> sweet. 18:31:03 <linuxmodder> nirik, you know off hand if any SOPs need love like the guides did? 18:31:18 <nirik> ansible repo is public, but the base location for it is batcave. It's not on pagure or github or anything. 18:31:28 <nirik> linuxmodder: I am sure they do. 18:31:32 <nirik> tammy_: sounds good. 18:31:35 <linuxmodder> munjeli, ssh to MOST if not all hosts drops you in /home/fedora/<username> 18:32:04 <linuxmodder> nirik, any reason we couldn't throw the public bits for ansible on pagure? 18:32:05 <tammy_> How does Nagios work interested in that as well 18:32:11 <munjeli> nirik: gtk. 18:32:26 <linuxmodder> tammy_, its a monitorign appliance kind of like a lookout 18:32:50 <munjeli> I'm pretty committed to an agnostic interface as I want this tooling for my own job and my open source project. 18:32:56 <linuxmodder> iirc its what status.fedoraproject.org is polled from 18:33:04 <tammy_> OK 18:33:12 <nirik> linuxmodder: yes, we need to not depend on a remote repo. So, until we have things setup so we can merge changes from both pagure and batcave into the batcave repo, we cannot have it at pagure. 18:33:24 <linuxmodder> munjeli, pagure is agnostic you can do your own local instance even 18:33:40 <nirik> status is actually manually updated. 18:33:57 <linuxmodder> that is still an issue ah didn't realie the sync thing was still issue 18:34:02 <nirik> there is a pending easyfix ticket to add a host to nagios if anyone wants to take that on. :) 18:34:18 <linuxmodder> might look at that 18:34:39 <nirik> well, if our ansible repo was on pagure, and pagure was down, we could redeploy it with ansible... execept we couldn't because there would be no ansible repo to use. ;) chicken and egg. 18:34:47 <lousab> nirik: i could start from there also 18:35:00 <lousab> wich number 18:35:06 <pingou> nirik: there is a RFE to allow mirroring in pagure :) 18:35:21 <pingou> something which is slowly getting higher in my list :) 18:35:22 <nirik> pingou: i know. ;) That will be nice when it arrives. 18:35:28 <nirik> lousab: let me look... 18:35:30 <pingou> fingers crossed :) 18:35:52 <nirik> https://fedorahosted.org/fedora-infrastructure/ticket/5337 18:36:03 <linuxmodder> pingou, that needs a secure db return tho no? 18:36:04 <lousab> ok 18:36:12 <linuxmodder> for the rsync'ing part 18:36:26 <pingou> linuxmodder: I don't follow you, what do you mean? 18:36:30 <nirik> pingou: once we have that it would be nice to move to a PR model for a lot of the infra stuff too... although not in all cases. 18:36:46 <pingou> nirik: def 18:36:52 <linuxmodder> pingou, somewhere or some host that the backend trusts coming back thru bastion 18:37:03 <nirik> would be nice for freezes and for new folks to submit things. 18:37:10 <linuxmodder> ^^ 18:37:22 <pingou> linuxmodder: it would be using ssh just like everyone 18:37:35 <pingou> pagure will provide you with a public ssh key to put on the receiving side 18:37:36 <linuxmodder> pingou, ok 18:37:38 <pingou> and a git hook 18:37:40 <pingou> et voila :) 18:37:51 <nirik> chakal: did you see anything in this discussion you wanted to work on? did you look at the easyfix tickets? 18:38:06 <linuxmodder> se3parate ssh key then the one on pagure already to git clone git@pagure? 18:38:20 <nirik> chakal: or perhaps docs and CSI changes would be a good way to start? 18:38:22 <linuxmodder> pingou, ^ 18:39:12 <pingou> linuxmodder: I don't think the system ssh keys can be used to ssh into anything, am I wrong? 18:39:29 <athos> /5/5 18:39:37 <linuxmodder> pingou, the one you setup for pagure for git over ssh can't be used? 18:39:43 <linuxmodder> if so why not ? 18:40:24 <pingou> linuxmodder: pagure knows your public key, how could it push to something w/ a public key? 18:40:31 <chakal> nirik, i am a litle beat lost 18:40:34 <pingou> and no pagure does not want to know your private key 18:41:21 <linuxmodder> no I was thinking setting it pagure pubkey + fas pass or something 18:41:32 <linuxmodder> using a OAUTH style ssh auth 18:41:44 <nirik> chakal: BTW, I am kevin... you just emailed me. ;) So, perhaps best would be working on docs/CSI variables? 18:41:50 <nirik> take a look at: https://fedoraproject.org/wiki/Infrastructure_Apprentice#Longer_term_quests 18:42:06 <chakal> nirik, ok for me perfect 18:42:09 <pingou> linuxmodder: I don't think pagure wants to do anything w/ your fas password 18:42:28 <linuxmodder> well then maybe I'm thinking something weird 18:42:37 <pingou> and I don't see how this would work to mirror projects outside Fedora (for example on github) 18:42:43 <linuxmodder> how does it handle git clone git@pagure.io atm then 18:42:48 <nirik> chakal: Cool. There's usually someone around in #fedora-admin that can answer questions too, so ask if you get stuck or need more info... 18:42:56 <linuxmodder> and couldn't we do same or similar for the mirroring 18:43:05 <pingou> linuxmodder: it has your public key, the rest is ssh/gitolite 18:43:36 <linuxmodder> right but couldn't the mirroring over ssh be done like the tunneling thru bastion is 18:44:07 <pingou> I'm not sure how that would work 18:44:11 <linuxmodder> auth to pagure and then to the external repo and vice versa on retrun 18:44:31 <athos> I think CSI variables would be a good introduction point for me as well :) if that's ok 18:44:34 <linuxmodder> use pagure similar to how bastion is for (only with mirroring) 18:44:36 <pingou> linuxmodder: tunneling through bastion is done on your side, there is nothing to do on bastion itself 18:44:52 <nirik> anyhow, we could close out now, or if folks would like I could pontificate about some application or setup in infrastructure... I think I was going to talk about mailman3 a few weeks back. Does anyone have interest in that? 18:44:53 <linuxmodder> that i get 18:44:56 <pingou> so if that were to work, it wouldn't be something to be done in pagure 18:45:10 <nirik> athos: great. We do have a lot of hosts to update still I am pretty sure. 18:45:15 <linuxmodder> ! i do 18:45:53 <linuxmodder> mm3 has been kicking my ass lately :( 18:46:01 <nirik> athos / tammy_ / chakal: also, if you see any problems or improvements to the CSI doc that apprentice page points to, that would be great to submit as well. ;) 18:46:36 <nirik> #topic Learn about - mailman3 18:46:44 <athos> ack 18:46:45 <nirik> ok, I will talk just a bit about mailman3. 18:46:50 <linuxmodder> i.e broken links or outdated info ( login with fas creds on most wiki pages to edit them( 18:47:31 <nirik> abompard is our mailman3 guru. He wrote large parts of hyperkitty (the archiver) and works upstream on the admin and core bits as well. :) Hopefully if I say something incorrect he will correct me. 18:48:05 <nirik> We are now 100% on mailman3 for our lists. All of them run on mailman01.phx2.fedoraproject.org internally. Thats both fedoraproject.org lists and fedorahosted.org lists. 18:48:41 <linuxmodder> so mailman02 and 03 are decom'd? 18:48:48 <nirik> mailman3 has a number of parts that fit together. There's a core list handler setup, hyperkitty (the archiver and web poster) and postorius (the admin interface) 18:49:17 <nirik> we do have a mailman02, but it's not doing anything. It's not easy/possible to load balance mailman3. 18:49:37 <nirik> emails come in via our smtp-mm hosts (we have 3 of them spread around the world) 18:50:04 <nirik> from there they go to mailman01. If mailman01 is down, the smtp-mm hosts just queue the incoming emails until they can be processed. 18:50:34 <linuxmodder> is that what was happening a few weeks back when things were delaye liek crap 18:50:37 <nirik> the web part of things goes via our proxy network into mailman01. 18:51:35 <tammy_> Great nirik will do 18:51:40 <nirik> Not sure which thing you mean, but that was likely due to someone on the perl sig changing acls on all their packages, which generated an email for each branch of each package and each acl and was going to the scm-commits list. 18:51:58 <nirik> That generated a pretty vast backlog of emails and notifications. 18:52:11 <linuxmodder> likely 18:52:15 <pingou> which has pretty much nothing to do with mailman :) 18:52:52 <nirik> yeah. but hopefully that will be fixed in FMN soon. ;) 18:53:13 <nirik> anyhow, lets see. mailman3 allows you to login via persona, fedora openid or yahoo currently. 18:53:13 * pingou ducks 18:53:41 <linuxmodder> ho0w many yahoo id folsk do we still have? 18:53:44 <nirik> if you login via fedora openid, you are username@fedoraproject.org... which may not be the address you subscribe to lists with 18:53:56 <linuxmodder> seems so '90s to me 18:54:04 <nirik> but you can add additional emails and manage them from your fedoraproject.org account once you login 18:54:08 <tammy_> I have yahoo 18:54:21 <nirik> no idea. abompard kept it because yahoo supports openid. ;) 18:54:23 <pingou> linuxmodder: people using yahoo are probably grateful to your kind words 18:54:47 <nirik> persona is also going away toward the end of the year... 18:55:41 <nirik> ok, thats a high level overview. Anyone have any more specific questions on mailman3 (if they are too specific, may defer to abompard :) 18:56:16 <linuxmodder> mozilla is still supporting openid so why is persona going away? 18:56:17 <pingou> nirik: any idea how easy/hard it is to add new domains? 18:56:29 <munjeli> nirik: is there a client running on the host that sends stuff out by email? 18:56:30 <pingou> linuxmodder: lack of resources 18:56:30 <nirik> linuxmodder: mozilla is dropping persona. 18:56:39 <puiterwijk> linuxmodder: because persona is totally unrelated to openid. 18:56:41 <nirik> pingou: I think it's pretty easy. 18:56:53 <linuxmodder> nvm was thinking oneandedone.mozilla which is openid 18:56:54 <nirik> munjeli: yeah, thats part of core mailman3. 18:57:01 <pingou> nirik: cool, at one point we may want to see if we want @lists.pagure.io lists or so :) 18:57:28 <nirik> pingou: +1 18:57:41 <linuxmodder> or jsut hooks for existing lists ? 18:59:08 <nirik> pingou: or... we could setup a new server with that near pagure and tie them more closely... ie, allow it to add/remove them, perhaps some kind of patchwork thing to watch for PRs on list? 18:59:14 <nirik> but thats all down the road. ;) 18:59:18 <nirik> #topic Open Floor 18:59:43 <nirik> anyone have any items for open floor? anything we didn't cover? questions, comments, favorate kind of rice? 18:59:56 <pingou> yeah, down the road :) 19:00:19 <linuxmodder> down the road? 19:00:24 <jflory7> Nothing from me! 19:00:31 <jflory7> nirik++ 19:00:45 <jflory7> Thanks for chairing. 19:01:00 <nirik> linuxmodder: something to be determined later after other things are done. 19:01:41 <nirik> ok. Thanks for coming everyone. Do continue in #fedora-admin, #fedora-apps and #fedora-noc. ;) 19:01:44 <nirik> #endmeeting