18:00:25 <nirik> #startmeeting Infrastructure (2016-06-09)
18:00:25 <zodbot> Meeting started Thu Jun  9 18:00:25 2016 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:25 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:25 <zodbot> The meeting name has been set to 'infrastructure_(2016-06-09)'
18:00:25 <nirik> #meetingname infrastructure
18:00:25 <zodbot> The meeting name has been set to 'infrastructure'
18:00:25 <nirik> #topic aloha
18:00:25 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore threebean pingou puiterwijk pbrobinson
18:00:25 <zodbot> Current chairs: abadger1999 dgilmore lmacken nirik pbrobinson pingou puiterwijk relrod smooge threebean
18:00:26 <nirik> #topic New folks introductions / Apprentice feedback
18:00:30 <nirik> morning everyone.
18:00:37 <linuxmodder> .fas linuxmodder
18:00:37 <zodbot> linuxmodder: linuxmodder 'Corey W Sheldon' <sheldon.corey@openmailbox.org>
18:00:52 <munjeli> good morning
18:01:07 * lousab is here
18:01:11 * skrzepto is here
18:01:11 <lousab> hi all
18:01:18 <athos> hello :)
18:01:22 <athos> .fas athoscribeiro
18:01:22 <zodbot> athos: ribeiro 'Athos Ribeiro' <athoscribeiro@gmail.com>
18:01:32 <linuxmodder> packing the seats ...good to see
18:01:34 <jflory7> .hello jflory7
18:01:37 <zodbot> jflory7: jflory7 'Justin W. Flory' <me@justinwflory.com>
18:01:55 <nirik> any new folks around today that would like to give a short one-line introduction?
18:02:01 <athos> !
18:02:22 <nirik> athos: feel free to chime in anytime.
18:02:31 <athos> sure:
18:02:48 <linuxmodder> tammy when she gets here too nirik
18:03:30 <tammy_> Hello everyone
18:03:37 <athos> My name is Athos, I'm a brazilian software engineer and CS graduate student. I am getting more involved with the project and would like to join the team :)
18:03:47 <nirik> morning tammy_. Care to give a short introduction of yourself?
18:03:54 <nirik> athos: welcome.
18:03:57 <jflory7> Hi athos, welcome :)
18:04:36 <linuxmodder> athos,  what is your fasid?
18:04:41 <athos> Just became an embassador for LATAM and my 1st package is under review :). I also tried sending PRs for fedora-review and copr once or twice
18:04:42 <linuxmodder> tammy_,  same to you
18:04:45 <athos> ribeiro
18:04:51 <athos> linuxmodder: ^
18:05:10 <nirik> athos: great. Are you more interested in application development type tasks then?
18:06:20 <athos> nirik: I am interested both in app development and in infrastructure maintainance... I am currently learning how ansible works and have some experience with chef
18:06:37 <tammy_> Sure @nirik  I am Tammy and I work as a helpdesk/junior system administrator  I just started the junior sys administration role about six months ago and want to gain more experience in doing so
18:06:54 <nirik> athos: cool. See me after the meeting in #fedora-admin and I can get you setup in the apprentice group...
18:06:59 <linuxmodder> nice ..welcome tammy_  what is your fasid
18:07:07 <athos> thanks! :)
18:07:21 <tammy_> My fasid is tammyb5
18:07:42 <jflory7> Hi, welcome tammy_!
18:07:51 <nirik> tammy_: great! Sorry that many of us have been busy this week... I've seen you asking for assistance, but always at times I am busy or not around. :( But we will get you setup... never fear.
18:07:54 <tammy_> I also signed up for the apprentice group
18:08:05 <linuxmodder> and approved for it :)
18:08:17 <tammy_> No worries totally understand
18:08:36 <nirik> it's been spam fighting this last few weeks. ;( Pesky spammers
18:08:40 <linuxmodder> tammy_,  on that note do oyu have a ssh key in fas setup yet  (noticced no gpg keys mentioned in fas)
18:08:42 <nirik> ok, any other new folks?
18:09:04 <linuxmodder> seemed like no other new nicks to me
18:09:25 <tammy_> Wow
18:09:27 <nirik> ok, on to info and announcements...
18:09:35 <nirik> #topic announcements and information
18:09:35 <nirik> #info We are still in Fedora 24 Final freeze - everyone
18:09:35 <nirik> #info spam attacks ongoing. Many thanks to Patrick for cleaning them up via basset
18:09:35 <nirik> #info spam attackers have opened over 10,000 accounts which have been cleaned by basset
18:09:35 <nirik> #info Added ru lang to ask.fedoraproject.org - kevin
18:09:36 <nirik> #info faf and retrace services added in stg via proxy01.stg - kevin
18:09:38 <nirik> #info Notification bounces now going to /dev/null - kevin
18:09:42 <nirik> #info alt.stg.fedoraproject.org setup for websites - kevin
18:09:43 <linuxmodder> tammy_,  worst case find me after meeting we can fix that pesky  batcaveo01 issue
18:09:54 <nirik> anything else anyone would like to note or discuss from those?
18:10:04 <nirik> tammy_: also happy to help in #fedora-admin after the meeting.
18:10:52 <linuxmodder> when is alt  goign  strictly  webisites ?
18:11:12 <linuxmodder> alt.fp.o that is not  alt.stg
18:11:16 <nirik> should be at release day...
18:11:18 <munjeli> nirik: is the ongoing spamming long ongoing? Typical?
18:11:28 <nirik> which is not next week sadly (we slipped a week)
18:11:29 <linuxmodder> planned or  actual GA was a n/g
18:11:38 <sayan> .hello sayanchowdhury
18:11:39 <zodbot> sayan: sayanchowdhury 'Sayan Chowdhury' <sayan.chowdhury2012@gmail.com>
18:11:45 <nirik> munjeli: started a few months ago... hasn't really let up much since then. ;(
18:12:10 <nirik> it's really pretty sad. People doing that for a living. ;(
18:12:30 <tammy_> Ok
18:12:59 <linuxmodder> brightside its helping us tighten up fas3
18:13:10 <nirik> we can't really lock things down too much or it would affect legit new contributors. ;( So, we have to mark and clean up...
18:13:18 <nirik> yep. :)
18:13:27 <munjeli> yah, it's a difficult problem.
18:13:28 <jflory7> Hopefully it will relent after a while. I wonder if it will fade off after the GA, if that's the reason for targeting.
18:13:32 * jflory7 shrugs
18:13:44 <linuxmodder> puiterwijk,  you around later have a few ideas for basset
18:13:56 <nirik> jflory7: I don't think it's anything really other than we are somewhat open so they can spew their spam and hope search engines will pick it up.
18:14:06 <jflory7> True.
18:14:08 <nirik> they are also hitting bugzilla some.
18:14:10 * pingou late
18:14:15 <jflory7> pingou: o/
18:14:16 <linuxmodder> jflory7,  doubtful the attention is  not seemingly  GA centric from what I've seen
18:14:18 <patricku> linuxmodder: you can also file tickets. But let me know if you have ideas
18:14:18 <nirik> and I heard they are hitting kernel.org bugzilla too now. ;(
18:14:41 <nirik> anyhow.
18:14:49 <linuxmodder> and mozilla is getting  traction from them too
18:15:04 <nirik> I didn't see any discussion items on the gobby. Did anyone have any? or shall we move on to apprentice open office hours...
18:15:13 <patricku> Again: if people know people in those communities under attack, feel free to send them my way
18:15:25 <linuxmodder> patricku, noted
18:15:54 <patricku> I'd be glad to help more people with setting up Basset or any other cross distro anti spam project
18:16:08 <nirik> sure. :)
18:16:30 <nirik> #topic Apprentice office hours
18:16:59 <munjeli> lousab and I are looking at the infrastructure mapping ticket and have questions
18:17:00 <nirik> any apprentices have questions or issues? or need things to work on ? perhaps we should hit the easyfix tickets and see if there's some that could use new folks working on them
18:17:09 <nirik> excellent. :) ask away.
18:17:10 <linuxmodder> munjeli,  shoot
18:17:49 <munjeli> I would like to make a map that exposes useful information interactively, does this mean service dependencies, networking?
18:17:56 <munjeli> What is useful to the team?
18:17:58 <linuxmodder> my connection is getting  unstable fyi  advance  apoligies if i d/c
18:18:19 <nirik> munjeli: well, IMHO I would start with something pretty basic at first. you can always add to it after.
18:18:20 <chakal> hi, i start few weeks ago and i need some guidance about what have to do
18:18:22 <linuxmodder> munjeli,  useful information being what in your workflow
18:18:49 <nirik> ie, just datacenters, what machines are in each...
18:18:52 <linuxmodder> chakal,  you looked at and done stuff on  the Join page yet?
18:18:55 <linuxmodder> welcome btw
18:19:10 <chakal> yes
18:19:29 <munjeli> linuxmodder: I did this last time as an html page with d3 that showed routes, acls, and showed error conditions like black holes on AWS
18:19:29 <nirik> chakal: take a look at https://fedoraproject.org/easyfix/ and the fedora-infrastructure section...
18:19:35 <lousab> nirik: what do you think if we start mapping machines using the code of https://apps.fedoraproject.org/
18:19:35 <chakal> i have logged on ssh server and already installed ansible
18:19:38 <linuxmodder> munjeli,  I am busy next week but  email still works too if you need a hand ( I was the one who originally  handed that off to you)
18:19:38 <nirik> see if any of those tickets look like something interesting to work on
18:20:04 <chakal> yes y have fas accound and registered on infrastructure list
18:20:44 <nirik> lousab: you mean make the map use that same interface?
18:20:49 <lousab> yes
18:20:53 <linuxmodder> chakal,  ssh key setup and / or  easyfix  tickets look good for you ?
18:21:01 <nirik> thats not a bad thought... :)
18:21:26 <linuxmodder> might get a littel busy but I like the idea
18:21:32 <nirik> it's simple, (json) and might work nicely for a map
18:21:41 <tammy_> Are there other things am apprentice can do
18:22:08 <linuxmodder> tammy_,  what you have in mind?
18:22:09 <munjeli> Yes, I don't like the layout of the graphing there - I used a sankey diagram last time to show flow, but the graph is not authentic to most infrastructure diagramming.
18:22:15 <linuxmodder> there is plenty
18:22:25 <lousab> nirik: exactly and it should be quite simple to start from there
18:22:34 <linuxmodder> sankey diagram?
18:23:03 <lousab> munjeli: what if we change a little bit the graphic?
18:23:23 <munjeli> Yes, a sankey diagram is a dataviz that describes flow. But it was also too busy. I think authentically, we use things closer to circuit diagrams to map datacenters
18:23:28 <nirik> tammy_: well, the main thing is easyfix tickets (proposing solutions/patches), but also if you can hang out in irc or on the list there will often be discussions about apps or things and you can chime in that you would like to help out with those. Also we always need docs cleanup/help, and there's a few things listed on the main apprentice page as "quests" that anyone can work on.
18:23:58 <nirik> https://fedoraproject.org/wiki/Infrastructure_Apprentice#Longer_term_quests
18:24:04 <chakal> linuxmodder, I dont know what i have to do on easeyfix
18:24:06 <nirik> updating CSI variables and docs work
18:24:06 <munjeli> lousab: sure, a good first pass would be a graph diagram
18:24:47 <linuxmodder> munjeli,  seen the infra_hosts file on batcave01?
18:24:51 <nirik> chakal: it depends on the ticket. Usually its something you can provide a patch for against the ansible repo...
18:24:52 <munjeli> linuxmodder: where is the best place to understand network and service dependencies? the data?
18:25:03 <munjeli> Yah, I have the list of hosts..
18:25:18 <nirik> munjeli: nagios has some of that, and ansible has host/vhost dependencies and datacenter info
18:25:25 <linuxmodder> service dependencies ?  as in what each needs or  is used for?
18:25:30 <munjeli> Yup, nagios is what I want.
18:25:53 <linuxmodder> nagios01.phx2
18:26:00 <munjeli> Yeah, understanding what will fail and how. spoffiness, etc
18:26:01 <linuxmodder> should help ^
18:26:36 <nirik> https://admin.fedoraproject.org/nagios/
18:26:39 <munjeli> What ports are open, how the infrastructure is scoped (private and public subnets)
18:26:42 <nirik> and https://admin.fedoraproject.org/nagios-external
18:26:56 <linuxmodder> munjeli,  take read at infrastructure.fedoraproject.org/infra/docs for the SOPs on those
18:26:58 <nirik> anyone should be able to login there. ;)
18:27:29 <munjeli> Even if we build for just seeing the basics, we should be open to extending for monitoring interactively, sec team loves stuff like this.
18:27:38 <nirik> BTW, nagios also has a map, but not all hosts are in nagios... and the map is... not so great
18:27:44 <munjeli> Gtk
18:27:49 <linuxmodder> munjeli,  indeed we sec folks do :)
18:28:08 <lousab> well let's start :)
18:28:09 <munjeli> :)
18:28:17 <linuxmodder> nirik,  does that nagios map have a undate script like ambassador map?
18:28:36 <munjeli> where is information on how to repo and code review?
18:28:37 <nirik> it generates it live from the data it has
18:29:16 <linuxmodder> looks like something out of the matrix :)
18:29:27 <linuxmodder> munjeli,  repo what ?
18:29:37 <linuxmodder> the nagios and ansible stuff?
18:29:40 <nirik> munjeli: well, if this is goign to be like apps, you can look at the ansible roles/apps
18:29:41 <munjeli> The code for this map..
18:29:52 <munjeli> where should we put it?
18:30:03 <linuxmodder> batcave01 ?
18:30:17 <linuxmodder> or somewhere on stg
18:30:18 <munjeli> but where do you git? is github?
18:30:25 <nirik> I'd say a patch against ansible in the ticket... call it roles/netmap or something?
18:30:33 <linuxmodder> munjeli,  some things you could
18:30:38 <tammy_> I will work on the docs and questions
18:30:40 <linuxmodder> but we have pagure :)
18:30:44 <nirik> munjeli: ssh to batcave01, git clone ansible
18:30:50 <munjeli> sweet.
18:31:03 <linuxmodder> nirik,  you know off hand if any SOPs need love like the guides did?
18:31:18 <nirik> ansible repo is public, but the base location for it is batcave. It's not on pagure or github or anything.
18:31:28 <nirik> linuxmodder: I am sure they do.
18:31:32 <nirik> tammy_: sounds good.
18:31:35 <linuxmodder> munjeli,  ssh to MOST if not all hosts  drops you in /home/fedora/<username>
18:32:04 <linuxmodder> nirik,  any reason we couldn't throw the public bits for ansible on pagure?
18:32:05 <tammy_> How does Nagios work interested in that as well
18:32:11 <munjeli> nirik: gtk.
18:32:26 <linuxmodder> tammy_,  its a monitorign appliance kind of like a lookout
18:32:50 <munjeli> I'm pretty committed to an agnostic interface as I want this tooling for my own job and my open source project.
18:32:56 <linuxmodder> iirc  its what status.fedoraproject.org is polled from
18:33:04 <tammy_> OK
18:33:12 <nirik> linuxmodder: yes, we need to not depend on a remote repo. So, until we have things setup so we can merge changes from both pagure and batcave into the batcave repo, we cannot have it at pagure.
18:33:24 <linuxmodder> munjeli,  pagure is  agnostic you can  do your own local instance even
18:33:40 <nirik> status is actually manually updated.
18:33:57 <linuxmodder> that is still an issue  ah didn't realie the  sync thing was still issue
18:34:02 <nirik> there is a pending easyfix ticket to add a host to nagios if anyone wants to take that on. :)
18:34:18 <linuxmodder> might look at that
18:34:39 <nirik> well, if our ansible repo was on pagure, and pagure was down, we could redeploy it with ansible... execept we couldn't because there would be no ansible repo to use. ;) chicken and egg.
18:34:47 <lousab> nirik: i could start from there also
18:35:00 <lousab> wich number
18:35:06 <pingou> nirik: there is a RFE to allow mirroring in pagure :)
18:35:21 <pingou> something which is slowly getting higher in my list :)
18:35:22 <nirik> pingou: i know. ;) That will be nice when it arrives.
18:35:28 <nirik> lousab: let me look...
18:35:30 <pingou> fingers crossed :)
18:35:52 <nirik> https://fedorahosted.org/fedora-infrastructure/ticket/5337
18:36:03 <linuxmodder> pingou,  that needs a secure db  return tho no?
18:36:04 <lousab> ok
18:36:12 <linuxmodder> for the rsync'ing part
18:36:26 <pingou> linuxmodder: I don't follow you, what do you mean?
18:36:30 <nirik> pingou: once we have that it would be nice to move to a PR model for a lot of the infra stuff too... although not in all cases.
18:36:46 <pingou> nirik: def
18:36:52 <linuxmodder> pingou,  somewhere or  some host that the backend trusts coming back thru  bastion
18:37:03 <nirik> would be nice for freezes and for new folks to submit things.
18:37:10 <linuxmodder> ^^
18:37:22 <pingou> linuxmodder: it would be using ssh just like everyone
18:37:35 <pingou> pagure will provide you with a public ssh key to put on the receiving side
18:37:36 <linuxmodder> pingou,  ok
18:37:38 <pingou> and a git hook
18:37:40 <pingou> et voila :)
18:37:51 <nirik> chakal: did you see anything in this discussion you wanted to work on? did you look at the easyfix tickets?
18:38:06 <linuxmodder> se3parate ssh key then the one on pagure already to git clone git@pagure?
18:38:20 <nirik> chakal: or perhaps docs and CSI changes would be a good way to start?
18:38:22 <linuxmodder> pingou, ^
18:39:12 <pingou> linuxmodder: I don't think the system ssh keys can be used to ssh into anything, am I wrong?
18:39:29 <athos> /5/5
18:39:37 <linuxmodder> pingou,  the one you setup for pagure for git over ssh can't be used?
18:39:43 <linuxmodder> if so why not ?
18:40:24 <pingou> linuxmodder: pagure knows your public key, how could it push to something w/ a public key?
18:40:31 <chakal> nirik, i am a litle beat lost
18:40:34 <pingou> and no pagure does not want to know your private key
18:41:21 <linuxmodder> no I was thinking setting it pagure pubkey + fas pass or  something
18:41:32 <linuxmodder> using a OAUTH style ssh auth
18:41:44 <nirik> chakal: BTW, I am kevin... you just emailed me. ;) So, perhaps best would be working on docs/CSI variables?
18:41:50 <nirik> take a look at: https://fedoraproject.org/wiki/Infrastructure_Apprentice#Longer_term_quests
18:42:06 <chakal> nirik, ok for me perfect
18:42:09 <pingou> linuxmodder: I don't think pagure wants to do anything w/ your fas password
18:42:28 <linuxmodder> well then maybe I'm thinking something weird
18:42:37 <pingou> and I don't see how this would work to mirror projects outside Fedora (for example on github)
18:42:43 <linuxmodder> how does it  handle git clone git@pagure.io atm then
18:42:48 <nirik> chakal: Cool. There's usually someone around in #fedora-admin that can answer questions too, so ask if you get stuck or need more info...
18:42:56 <linuxmodder> and couldn't we do same or similar for the mirroring
18:43:05 <pingou> linuxmodder: it has your public key, the rest is ssh/gitolite
18:43:36 <linuxmodder> right but couldn't the mirroring over ssh be done like the tunneling thru bastion is
18:44:07 <pingou> I'm not sure how that would work
18:44:11 <linuxmodder> auth to pagure and then to the external repo and  vice versa on retrun
18:44:31 <athos> I think CSI variables would be a good introduction point for me as well :) if that's ok
18:44:34 <linuxmodder> use pagure similar to how  bastion is for (only with mirroring)
18:44:36 <pingou> linuxmodder: tunneling through bastion is done on your side, there is nothing to do on bastion itself
18:44:52 <nirik> anyhow, we could close out now, or if folks would like I could pontificate about some application or setup in infrastructure... I think I was going to talk about mailman3 a few weeks back. Does anyone have interest in that?
18:44:53 <linuxmodder> that i get
18:44:56 <pingou> so if that were to work, it wouldn't be something to be done in pagure
18:45:10 <nirik> athos: great. We do have a lot of hosts to update still I am pretty sure.
18:45:15 <linuxmodder> ! i do
18:45:53 <linuxmodder> mm3 has been kicking my ass lately :(
18:46:01 <nirik> athos / tammy_ / chakal: also, if you see any problems or improvements to the CSI doc that apprentice page points to, that would be great to submit as well. ;)
18:46:36 <nirik> #topic Learn about - mailman3
18:46:44 <athos> ack
18:46:45 <nirik> ok, I will talk just a bit about mailman3.
18:46:50 <linuxmodder> i.e broken links or outdated info ( login with fas creds on most  wiki pages to edit them(
18:47:31 <nirik> abompard is our mailman3 guru. He wrote large parts of hyperkitty (the archiver) and works upstream on the admin and core bits as well. :) Hopefully if I say something incorrect he will correct me.
18:48:05 <nirik> We are now 100% on mailman3 for our lists. All of them run on mailman01.phx2.fedoraproject.org internally. Thats both fedoraproject.org lists and fedorahosted.org lists.
18:48:41 <linuxmodder> so mailman02 and 03  are  decom'd?
18:48:48 <nirik> mailman3 has a number of parts that fit together. There's a core list handler setup, hyperkitty (the archiver and web poster) and postorius (the admin interface)
18:49:17 <nirik> we do have a mailman02, but it's not doing anything. It's not easy/possible to load balance mailman3.
18:49:37 <nirik> emails come in via our smtp-mm hosts (we have 3 of them spread around the world)
18:50:04 <nirik> from there they go to mailman01. If mailman01 is down, the smtp-mm hosts just queue the incoming emails until they can be processed.
18:50:34 <linuxmodder> is that what was happening a few weeks back when  things were delaye liek crap
18:50:37 <nirik> the web part of things goes via our proxy network into mailman01.
18:51:35 <tammy_> Great nirik will do
18:51:40 <nirik> Not sure which thing you mean, but that was likely due to someone on the perl sig changing acls on all their packages, which generated an email for each branch of each package and each acl and was going to the scm-commits list.
18:51:58 <nirik> That generated a pretty vast backlog of emails and notifications.
18:52:11 <linuxmodder> likely
18:52:15 <pingou> which has pretty much nothing to do with mailman :)
18:52:52 <nirik> yeah. but hopefully that will be fixed in FMN soon. ;)
18:53:13 <nirik> anyhow, lets see. mailman3 allows you to login via persona, fedora openid or yahoo currently.
18:53:13 * pingou ducks
18:53:41 <linuxmodder> ho0w many yahoo id  folsk do we still have?
18:53:44 <nirik> if you login via fedora openid, you are username@fedoraproject.org... which may not be the address you subscribe to lists with
18:53:56 <linuxmodder> seems so '90s to me
18:54:04 <nirik> but you can add additional emails and manage them from your fedoraproject.org account once you login
18:54:08 <tammy_> I have yahoo
18:54:21 <nirik> no idea. abompard kept it because yahoo supports openid. ;)
18:54:23 <pingou> linuxmodder: people using yahoo are probably grateful to your kind words
18:54:47 <nirik> persona is also going away toward the end of the year...
18:55:41 <nirik> ok, thats a high level overview. Anyone have any more specific questions on mailman3 (if they are too specific, may defer to abompard :)
18:56:16 <linuxmodder> mozilla is  still supporting openid  so why is persona going away?
18:56:17 <pingou> nirik: any idea how easy/hard it is to add new domains?
18:56:29 <munjeli> nirik: is there a client running on the host that sends stuff out by email?
18:56:30 <pingou> linuxmodder: lack of resources
18:56:30 <nirik> linuxmodder: mozilla is dropping persona.
18:56:39 <puiterwijk> linuxmodder: because persona is totally unrelated to openid.
18:56:41 <nirik> pingou: I think it's pretty easy.
18:56:53 <linuxmodder> nvm was thinking oneandedone.mozilla which is openid
18:56:54 <nirik> munjeli: yeah, thats part of core mailman3.
18:57:01 <pingou> nirik: cool, at one point we may want to see if we want @lists.pagure.io lists or so :)
18:57:28 <nirik> pingou: +1
18:57:41 <linuxmodder> or jsut hooks for existing  lists ?
18:59:08 <nirik> pingou: or... we could setup a new server with that near pagure and tie them more closely... ie, allow it to add/remove them, perhaps some kind of patchwork thing to watch for PRs on list?
18:59:14 <nirik> but thats all down the road. ;)
18:59:18 <nirik> #topic Open Floor
18:59:43 <nirik> anyone have any items for open floor? anything we didn't cover? questions, comments, favorate kind of rice?
18:59:56 <pingou> yeah, down the road :)
19:00:19 <linuxmodder> down the road?
19:00:24 <jflory7> Nothing from me!
19:00:31 <jflory7> nirik++
19:00:45 <jflory7> Thanks for chairing.
19:01:00 <nirik> linuxmodder: something to be determined later after other things are done.
19:01:41 <nirik> ok. Thanks for coming everyone. Do continue in #fedora-admin, #fedora-apps and #fedora-noc. ;)
19:01:44 <nirik> #endmeeting