18:00:02 <nirik> #startmeeting Infrastructure (2017-03-16) 18:00:02 <zodbot> Meeting started Thu Mar 16 18:00:02 2017 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:02 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:02 <zodbot> The meeting name has been set to 'infrastructure_(2017-03-16)' 18:00:02 <nirik> #meetingname infrastructure 18:00:02 <zodbot> The meeting name has been set to 'infrastructure' 18:00:02 <nirik> #topic aloha 18:00:02 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore threebean pingou puiterwijk pbrobinson 18:00:02 <zodbot> Current chairs: abadger1999 dgilmore lmacken nirik pbrobinson pingou puiterwijk relrod smooge threebean 18:00:02 <nirik> #topic New folks introductions 18:00:08 <nirik> morning everyone. 18:00:09 <marc84> hi everyone 18:00:14 <smooge> hello 18:00:15 <athos> hello :) 18:00:22 <puiterwijk> Hello 18:00:30 <ForrestaGrim> hi 18:01:08 <clime> Hey 18:01:20 <nirik> any new folks like to give a short introduction of themselves? 18:01:20 <threenodedev> hi 18:01:26 <mboddu> .hello mohanboddu 18:01:27 <zodbot> mboddu: mohanboddu 'Mohan Boddu' <mboddu@bhujji.com> 18:01:30 <nirik> I know we had a few new folks on the list this week. 18:01:43 <Aurelie_> hi 18:01:52 <capitanocrunch> hello 18:02:05 <tflink> hello 18:02:16 <ForrestaGrim> Ben Hart here, I emailed the list a couple times this week. Looking to start learning and helping the group in any way I can. 18:02:24 <Savitoj> Hello 18:02:38 <Aurelie_> I can start if there are no others who want to introduce theirselves... :) 18:02:47 <nirik> ForrestaGrim: welcome. :) 18:02:56 <nirik> Aurelie_: go ahead, and welcome to you as well. 18:03:14 <Aurelie_> Hi everyone, I'm Aurelie from London. I've been working on Linux (RedHat 5,6,7/CentOS/SuSe) for the last 12 years and I've been using it for 20 (I started with Slackware when I was 17... :D). 18:03:23 <Aurelie_> I work for a Dutch Bank. My team and I are responsible for about 1k systems (VMware VM's/HP Blades), several RH5/7 Clusters and two IBM GPFS Linux Storage clusters + clients. 18:03:32 <Aurelie_> We use Chef, IPA, Satellite 6 (Satellite 5 before that). Currently learning Ansible, Python and RPM (Fedora) packaging. I'd like to practice with Ansible/Python but I'm open to any SysAdmin task you might want me to tackle really... :) 18:04:05 <Aurelie_> and I prepared this short introduction beforehand ... Lol :) That's it from me. 18:04:11 <nirik> very nice. ;) Note that we only have about 600 instances, so you all are bigger than we are. ;) 18:04:27 <Savitoj> Welcome Aurelie_ :) 18:04:41 <Aurelie_> thanks nirik and Savitoj :) 18:04:47 <bowlofeggs> .hello bowlofeggs 18:04:48 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <randy@electronsweatshop.com> 18:04:57 <Savitoj> I would go next :) 18:05:00 * doteast here and late :( 18:05:08 <nirik> Savitoj: go for it. 18:05:11 <Savitoj> My name is Savitoj Singh, I'm working in Red Hat (Pune) since 3.7 years as a Senior Systems Administrator and overall 5 years experience. We as a team takes care of 1300+ servers over a globe and manages 4 major Red Hat CEE Labs. 18:05:11 <Savitoj> I'm comfortable with Python and shell scripting, and I have a work experience on RH OpenStack, OpenShift, Docker, Vagrant, Jenkins, GitLab, CI/CD, RHV, CFME, Ansible, Puppet etc. My interests are more inclined towards DevOps. 18:05:11 <Savitoj> I would really like to learn Emacs and grind python skills. 18:05:41 <Savitoj> Certifications: https://www.redhat.com/rhtapps/certification/verify/?certId=110-473-592 18:05:42 <Savitoj> LinkedIn: https://www.linkedin.com/in/savitojs 18:05:43 <nirik> excellent. Wow... lots of pretty experenced folks this week (not that you have to be by any means) 18:05:51 <Savitoj> That's it from my side :) 18:06:02 * jcline is here, but late 18:06:14 <Savitoj> Thank you nirik :) 18:06:19 * x3mboy here, late too 18:06:19 <nirik> welcome all. Do see me after the meeting in #fedora-admin and we can get anyone interested added to the apprentice group to look around and such 18:06:21 <dgilmore> hey 18:06:40 <capitanocrunch> Ciao, i'm fabrizio from Italy, i consider myself a Junior SysAdmin, playing with Bash Apache Postfix etc. 18:07:20 <nirik> x3mboy: I saw you added to the gobby doc that you wanted to be added to the apprentice group... do see me after the meeting in #fedora-admin and we can get you added. 18:07:23 <nirik> welcome capitanocrunch 18:07:37 <ForrestaGrim> I'll go, except correctly this time 18:07:42 <x3mboy> nirik, roger that! 18:08:03 <ForrestaGrim> Sorry for earlier folks, Anyway Im Ben, from Montana., I am the least experienced out of this group it seems. 18:08:37 <nirik> we all started at the beginning someday. ;) 18:08:53 <ForrestaGrim> Been using Linux on and off for years now. But finally landed a job where it's our primary OS. We support over 300 web services the the state of Montana and along with another tech we support around 50 virtual servers on Vmware. 90% of those are Centos 5+ 18:09:08 <ForrestaGrim> Im green, but loving it. 18:09:45 <nirik> ForrestaGrim: ah. You do know that el5 is heading to extended life only at the end of the month? 18:09:46 * Aurelie_ well done ForrestaGrim, sounds like a good start! :) 18:10:06 <ForrestaGrim> I do nirik.. luckily the 5's are in tst only. 18:10:14 <ForrestaGrim> thank you Aurelie 18:10:45 <nirik> Great to have a bunch of new folks. ;) Please do feel free to ask questions anytime here in the meeting or in #fedora-admin or on the list. No question is a dumb question! 18:10:55 <puiterwijk> I think NL just doubled their presence in this meeting! :) 18:11:00 <nirik> did we get all the new folks now? 18:11:06 <puiterwijk> welcome all! 18:11:26 * Aurelie_ thank puiterwijk! :) 18:11:30 <capitanocrunch> tnx! 18:11:38 <ForrestaGrim> thanks! 18:11:48 <Savitoj> thanks everyone :) 18:11:53 <nirik> ok, lets move along to status / info dump then... 18:12:00 <nirik> #topic announcements and information 18:12:00 <nirik> #info Fedora 26 Alpha freeze is on now! - everyone 18:12:00 <nirik> #info Various hosted projects redirects and data so they could migrate - kevin 18:12:00 <nirik> #info A few new jenkins projects - kevin 18:12:00 <nirik> #info Some new tickets marked easyfix for apprentices to look into - kevin 18:12:01 <nirik> #info Some RDU systems were moved to new cage. Some RDU systems will be re-ipd in April 18:12:03 <nirik> #info building bvirthost06/virthost04 is ongoing 18:12:16 <nirik> any other announcements or info anyone wants to add? or discussion from the above 18:12:16 <nirik> ? 18:13:04 <Aurelie_> I'll take a look at the new easyfix tickets and see if I can take ownership of any... 18:13:05 <nirik> Things are pretty quiet since we are in freeze. 18:13:25 <puiterwijk> I've been doing some performance testing and tweaking for Pagure and libgit2. 18:13:31 <nirik> Some of those tickets I marked I am not sure how easy they will be... but they should be interesting and at least we can move them forward some 18:13:33 <puiterwijk> Hoping to have some concrete PRs very soon 18:13:38 <nirik> Oh, also: 18:13:55 <nirik> #info Fedora 26 alpha was no go at the meeting today, so we have another week of freeze 18:14:16 <nirik> puiterwijk: great. 18:14:44 <nirik> #topic meeting time/meeting moderator - kevin 18:14:54 <nirik> So, daylight saving time is moving us around again... 18:15:03 <nirik> and will change in .eu in a few weeks again. 18:15:06 <capitanocrunch> i started to look at the SOP_StandardOperatingProcedures, so i'll search for that kind of easyfix ticket 18:15:24 * cverna is late 18:15:42 <Aurelie_> what's the impact on the infrastructure? Any past issue? 18:15:54 <nirik> capitanocrunch: https://fedoraproject.org/easyfix/ is the list of all of them over all out apps 18:16:07 <puiterwijk> Aurelie_: the infra doesn't really impact from DST, as that all runs on UTC 18:16:09 * Aurelie_ nice one! Thanks 18:16:15 <nirik> yeah, we use UTC for everything. 18:16:27 <puiterwijk> It's really just the meeting that is somewhat confusing at times :) 18:16:29 * Aurelie_ understood 18:16:29 <nirik> I was just going to ask if we should look at adjusting the meeting any, or this time was still good for everyone. 18:16:36 <nirik> or if we should run a whenisgood again or something. 18:16:43 <nb> .hello nb 18:16:44 <zodbot> nb: nb 'Nick Bebout' <nb@nb.zone> 18:16:52 <ForrestaGrim> in my time zone it's lunch, works out perfect ;) 18:17:13 <Aurelie_> it should be fine with me 18:17:15 <nirik> Also, I was thinking we could pass around who runs the meeting... I''m happy to do it, but I think it might be fun for others to do and then it gets less boring... :) 18:17:17 <nb> fine with me 18:17:51 <puiterwijk> nirik: how about the first person to email on Monday after 15UTC will lead the meeting that week? 18:17:58 <puiterwijk> (email to the infra list) 18:18:19 <nirik> ha. I was thinking at the end of each meeting we pick the person for next time. 18:18:49 <nirik> so sounds like this time is mostly ok still... 18:19:09 <puiterwijk> Yeah, I think this is one of the better times. 18:19:15 <puiterwijk> For EMEA/NA, that is 18:19:37 <nirik> ok. 18:19:39 <Savitoj> nirik: In India, it's 11:50 PM atm :) it's quite late but okay. 18:19:55 <nirik> Savitoj: yeah, it's hard to do global meetings. ;( 18:20:12 <nirik> some teams also alternate... 18:20:57 <Savitoj> nirik: okay for me :) I sleep late 18:21:05 <nirik> ok, moving along then... 18:21:22 <nirik> #topic Learn about: updates syncing scripts - kevin 18:21:45 <nirik> There were some questions about this eariler in the week, so I thought I would talk about how things are setup now... 18:21:53 <x3mboy> For me CLT is 15.20 (work time) so it's ok 18:22:40 <nirik> We have a master mirror volume (an NFS share on a netapp) that we put all our images/rpms/etc content on and that is in turn exposed on our download servers. 18:23:01 <nirik> There's currently 4 machines that write content to that same volume 18:24:38 <nirik> 1. 'alt' is a machine where some few people have write access to write things to /pub/alt/ This includes 18:25:01 <nirik> all the stuff under https:////dl.fedoraproject.org/pub/alt/ 18:25:14 <nirik> and can be updated at any time by those folks. 18:25:57 * smooge takes his antacid 18:25:58 <nirik> 2. rawhide-composer - composes rawhide (our devel release) and copies it to /pub/fedora/linux/development/rawhide and also /pub/alt/fedora-secondary/development/rawhide 18:26:14 <nirik> 3. branched composer does the same for fedora 26 currently 18:26:26 <nirik> (except 26 dirs instead of rawhide) 18:27:07 <nirik> 4. Finally bodhi-backend01 does all the updates for fedora. for f24/f25/epel567 it writes to /pub/fedora/linux/updates and updates-testing 18:27:28 <nirik> for f26 it does that and also /pub/alt/fedora-secondary/updates and updates-testing 18:27:58 <nirik> Finally whenever any of them write content we need to update a file called 'fullfiletimelist' with information about the content 18:28:22 <nirik> however, thats only at the top of trees... 18:28:40 <nirik> http://dl.fedoraproject.org/pub/fedora/fullfiletimelist-fedora 18:28:50 <nirik> and /pub/epel and /pub/fedora-secondary/ 18:29:04 <nirik> so, you may relize that there's a problem here. :) 18:29:30 <ForrestaGrim> that list will never stop growing.. will it? 18:29:44 <nirik> if say rawhide and branched are both syncing, branched finishes first and updates the list, it will be wrong because rawhide will keep updating and then also update the list (and it will be right again) 18:30:09 <nirik> well, it's basically a list of all the files and their update times/etc... so yeah, it will grow over time. 18:30:52 <nirik> we do have locking on making the files... 18:31:27 <nirik> anyhow, thats where we are now. :) In order to make it better I think we would have to re-arrange things so each machine is the only thing that updates under one tree. 18:32:03 <nirik> BTW, the lists are for our quick-mirror-fedora script. 18:32:19 <nirik> https://pagure.io/quick-fedora-mirror 18:32:25 <nirik> sorry, quick-fedora-mirror. 18:32:30 <Aurelie_> but not when it comes to write them...? 18:32:38 <nirik> its pretty cool if you have a chance to look at how it works. 18:33:00 * Aurelie_ got disconnected.... :( 18:33:25 <nirik> well, we could also fix this by locking around any content updating... ie, process 1 has to finish syncing and updating before process 2 can run... 18:33:32 <nirik> but that would make things a lot slower. 18:33:52 <nirik> the problem is that multiple processes update files under the same trees 18:34:39 <nirik> This also came up in the context of getting rid of cron jobs. ;) 18:35:02 <nirik> Currently the alt and bodhi-backend01 updates to that file are in crons. 18:35:14 <nirik> But we have a message bus for bodhi at least that will tell it when to sync. 18:35:51 <nirik> anyhow, any questions on that, or shall we move on to apprentice hours... 18:36:27 <nirik> #topic Apprentice Open office hours 18:36:29 <Aurelie_> nope, I lost part of the conversation, I'll read it offline later 18:36:47 <nirik> Any apprentices with questions, comments, needing things to do, etc? 18:37:15 <ForrestaGrim> Can I ask.. 18:37:36 <nirik> sure, feel free to jump in. 18:38:20 <ForrestaGrim> I parused the apprentice page and I just want to make sure.. you guys are looking for folks who are active first and foremast. But as an apprentice our main goal is to become familiar with the systems/configs/How everyhitng fits together. Inspect tickets, offer advice and help out wherever we can. 18:38:24 <ForrestaGrim> Basic gist yes? 18:38:33 <ForrestaGrim> *most 18:38:47 <nirik> sure, and work on easyfix tickets, propose fixes, etc. 18:38:57 <ForrestaGrim> yup ypu.. ok good just clarifying. 18:39:09 <nirik> yep. 18:39:14 <Aurelie_> I'm keen to get operative asap but I know it'll take time. I'm keeping an eye on Nagios and ssh-browsing the servers I'm allowed to login to (with focus on Ansible), etc... and Pagure and its tickets too... :) 18:39:35 <nirik> pagure is very active these days (development wise :) 18:39:54 <Aurelie_> nice! (well, it depends... :-D) 18:39:56 <puiterwijk> Well, also from users wise since fhosted shut down... :) 18:40:09 <nirik> true... it's hopping. 18:40:27 <nirik> #topic Next weeks moderator 18:40:41 <nirik> Who would like to run the meeting next week? :) 18:41:13 <puiterwijk> nirik: I still vote for "first to email to list after 15:00UTC on monday" :-) 18:41:23 <puiterwijk> But I'm fine with moderating next week 18:41:24 <nirik> puiterwijk: ok, we could try that too... 18:41:31 <ForrestaGrim> lol 18:41:50 <nirik> ok, lets sign you up and if someone else mails first you can hand off to them. :) 18:41:58 <puiterwijk> Yep, sounds good 18:42:09 <nirik> #info puiterwijk to moderate next week. 18:42:13 <nirik> #topic Open Floor 18:42:18 <nirik> Anyone have anything for open floor? 18:42:57 <Savitoj> nirik: I have a quick question, Just in case we're not able to join the meeting for any reason.. How can we look at meeting notes ? 18:43:02 <nirik> questions, comments, favorite pie from pi day? 18:43:20 <Savitoj> nirik: I mean IRC logs of meeting 18:43:23 <nirik> Savitoj: they go to meetbot.fedoraproject.org also to the 'meeting-minutes' mailing list. :) 18:43:26 <puiterwijk> Savitoj: you can view the logs on https://meetbot.fedoraproject.org/ 18:43:52 <Savitoj> Good to know that.. Thank you nirik puiterwijk :) 18:43:57 <nirik> The mailing list is kind of cool because you can see all the various groups and what the meet on... but it's a lot of content too 18:44:02 <Aurelie_> yes, found the meetbot this morning, really useful... 18:44:26 <ForrestaGrim> So is there a list anywhere, or some sort of notations on the current members of the group and maybe primary skills or the like? 18:45:10 <nirik> ForrestaGrim: there's a 'officers' page... but that doesn't have everyone I don't think... 18:45:12 * nirik looks 18:45:28 <nirik> https://fedoraproject.org/wiki/Infrastructure/Officers 18:45:46 <nirik> But yeah, thats not too useful. 18:45:47 <ForrestaGrim> More out of curiosity than anything really. Questions should be aimed towards the group rather than a specific person(s) 18:46:05 <nirik> yep. :) 18:46:11 <puiterwijk> That's a very limited set of the infra people. I think a big list of everyone is kinda hard due to turnover of volunteers, 18:46:22 <ForrestaGrim> ^ ahh makes sense 18:46:29 <nirik> Basically myself, puiterwijk and smooge are full time sysadmin folks... relrod is part time... nb and other folks as their time permits. 18:46:47 <ForrestaGrim> and you primary three work for Fed/RH? 18:47:09 <nirik> pingou and bowlofeggs and jcline are full time devel folks, with lots of other contibutors... 18:47:35 * jcline waves 18:48:16 <nirik> ForrestaGrim: yep... but there's no requirement to do so. Anyone can contibute and gain access based on their work 18:48:42 <nirik> we try and make things as open as we can. 18:48:56 <ForrestaGrim> working for RH is a goal of mine, eventually. So yeah like I said just curious. 18:49:01 <ForrestaGrim> ;) 18:49:37 * bowlofeggs waves erratically 18:49:39 <nirik> many folks who now do started contibuting in their spare time. :) 18:50:12 <nirik> ok, will close out the meeting in minute % rand (30) 18:50:23 <x3mboy> I need to run in 20 minutes 18:50:27 <nirik> and my favorite pie is lemon chiffon. :) 18:50:28 <Aurelie_> nice to meet you all Thanks 18:50:42 <ForrestaGrim> *googles chiffon* 18:50:43 <Savitoj> Nice meeting you all :) Thanks much 18:50:55 <x3mboy> s/20/110/g 18:51:04 <nirik> and new folks, I'll be in #fedora-admin to add people to apprentice. ;) 18:51:09 <nirik> Thanks for coming everyone! 18:51:10 <smooge> thank you all 18:51:14 <nirik> #endmeeting