18:00:05 <nirik> #startmeeting Infrastructure (2015-11-05)
18:00:05 <zodbot> Meeting started Thu Nov  5 18:00:05 2015 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:05 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:05 <nirik> #meetingname infrastructure
18:00:05 <nirik> #topic aloha
18:00:05 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk pbrobinson
18:00:05 <zodbot> The meeting name has been set to 'infrastructure'
18:00:05 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pbrobinson pingou puiterwijk relrod smooge threebean
18:00:06 <nirik> #topic New folks introductions / Apprentice feedback
18:00:17 * mirek-hm is here
18:00:28 * dotEast2015_ here
18:00:32 <puiterwijk> I'm here, but handling copr and dell
18:00:34 <odin2016> here
18:00:38 <mattdm> n.b. Council Meeting over in #fedora-meeting-1
18:00:45 * jsmith lurks
18:00:48 <nirik> any new folks like to give a short one line introduction? or apprentices with questions or comments?
18:01:02 * sayan is here
18:01:06 <odin2016> Yes.
18:01:58 <odin2016> I joined with  in the last week, I currently work as an intro level sysadmin here in Michigan.
18:02:00 * aikidouke has a question after intros
18:02:07 <odin2016> I think that's about one line.
18:02:08 <nirik> welcome odin2016. ;)
18:02:10 * bitlord present (just listening)
18:02:11 <smdeep> .hello smdeep
18:02:12 <zodbot> smdeep: smdeep 'Sudeep Mukherjee' <smdeep@gmail.com>
18:02:14 * mpduty is here
18:02:20 <smdeep> I am having net issues
18:02:25 <nirik> odin2016: you are more interested in sysadmin than application development?
18:02:35 <nirik> smdeep: happens to us all. :)
18:03:07 <odin2016> right now, yes, but I've not an issue learning some application development.
18:03:35 <athos> Hello, my name is Athos. Trying to learn a little more about the team. Just listening here :) Current working with packaging and using Copr at work =D
18:03:41 <odin2016> I feel like you need to at least have an idea what an app is doing if you are going to support any part of it.
18:04:10 <nirik> odin2016: see me after the meeting in #fedora-admin and I can point you in the right direction to get started. ;)
18:04:21 <nirik> athos: welcome. :) feel free to chime in with quesions anytime.
18:04:33 <odin2016> nirik: will do.
18:04:36 <nirik> aikidouke: you had some questions to bring up?
18:04:54 <aikidouke> question: when F23 released, mirrorlist was reporting some mirrors that said they had the iso's, but were generating 404s, etc. Should we be checking if the mirror site actually has the iso file?
18:05:12 <nirik> well, it's not so easy I'm afraid. ;)
18:05:22 <athos> ty
18:05:30 <sayan> kushal is having network issues so he might not able to join in
18:05:30 <nirik> yes, that would be nice to know, but we cannot check all files on all mirrors... there's just way too many of them
18:05:44 <nirik> we could perhaps do better with isos specifically...
18:06:02 <aikidouke> I had an idea along that line, just checking isos..
18:06:16 <aikidouke> maybe i'll put something together and share it
18:06:18 <nirik> https://github.com/fedora-infra/mirrormanager2 is the code and adrianr and pingou would be the ones to talk to about it. ;)
18:06:24 <aikidouke> thanks
18:06:57 <nirik> cool. anything else? or shall we move on to status/info?
18:07:32 <nirik> #topic announcements and information
18:07:32 <nirik> #info 🌭  🌭 Fedora 23 is released! 🌭 🌭  - everyone
18:07:32 <nirik> #info REMINDER: apprentice work day 2015-11-18. Join us - everyone
18:07:32 <nirik> #info REMINDER: nov apprentice status email went out, please reply - all apprentices
18:07:32 <nirik> #info Nasty power outage last saturday at phx2, many thanks to Patrick for handling it.
18:07:33 <nirik> #info mailman3 migrations possibly resuming nov 16th - abompard
18:07:35 <nirik> #info [release] pkgdb2: 1.32.2 - pingou - http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/EBE5VXJFBPSN3UG7BHCTRYX5FCLTOVK7/
18:07:38 <nirik> #info First release of statscache deployed.  There will be kinks to work out. - threebean / nyazdani / rtnpro - https://apps.fedoraproject.org/statscache
18:07:41 <nirik> #info New release of tunir for Autocloud backends - kushal - http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/MXXRQHXX6BIRXULY5L5JVUF2X7DJEU2F/
18:07:46 <nirik> any other announcements or info to share?
18:08:05 <nirik> we are now out of freeze, so we can catch up on changes that were pending in the run up to f23 release
18:09:13 <nirik> ok, on to discussion items
18:09:23 <nirik> #topic meeting time - kevin
18:09:36 <nirik> due to DST time for this meeting has shifted in NA...
18:09:47 <nirik> is everyone ok with this time moving forward? or should we shift it back?
18:10:16 <mirek-hm> well I just find via the hard way that if you want to use F23 cloud images you need to put at the top "raw: dnf install -y python2 python-dnf libselinux-python libsemanage-python python-firewall"  https://fedorahosted.org/cloud/ticket/126
18:10:16 <odin2016> I like it. It actually works better since I work at night for my full time job.
18:10:28 <threebean> works for me :)
18:10:37 <dotEast2015_> same here
18:10:38 <aikidouke> no pref either
18:10:52 <mirek-hm> so if you want to use ansible with F23 bear that in mind
18:11:35 <nirik> mirek-hm: yeah, this was discussed on the devel list not long before release. ;(
18:11:55 <nirik> but yeah we should add that to our f23 cloud setup
18:12:09 <nirik> #info people seem ok with sticking with 18UTC
18:12:24 <nirik> #topic fas3 plans - pingou/xavier/kevin
18:12:37 <nirik> just wanted to highlight the email pingou sent to the list about fas3.
18:12:52 <nirik> There's a number of things left to do, but we wanted to try and roll it out before too long.
18:13:06 <nirik> so if anyone is looking for things to do, helping with those items would be great.
18:14:51 * nirik is looking up the link to the post
18:15:04 <nirik> https://lists.fedoraproject.org/archives/list/infrastructure%40lists.fedoraproject.org/thread/YZIAYVJDQF427A6FGPKFX5BU6AWTS7QG/
18:15:17 <nirik> #topic mirror lists migration from @redhat.com - kevin/smooge
18:15:58 <nirik> so we have 2 lists still hosted at redhat.com that were for mirror admins coordinating releases.
18:16:08 <nirik> we are going to be moving those over to lists.fedoraproject.org soon.
18:16:24 <nirik> This is mostly just a headsup/notification of that
18:16:44 <nirik> tflink: you around? still game to talk about buildbot today?
18:17:00 <tflink> crap, i completely forgot about that
18:17:22 <nirik> sorry, I should have reminded you.
18:17:29 <nirik> want me to talk about squid instead?
18:17:32 <tflink> it's on my calendar and everything
18:17:42 <tflink> if you're prepared enough for it, sure
18:17:51 <tflink> would buildbot be next week, then?
18:18:06 <nirik> well, not really prepared, but I am fine just winging it. ;)
18:18:12 <nirik> then it would be dec 5th
18:18:18 <nirik> sorry, dec 3rd
18:18:56 <nirik> is dec 3rd ok for you?
18:19:01 <tflink> yeah, that'll work
18:19:07 <nirik> cool.
18:19:08 * tflink sets up more/better reminders this time
18:19:40 <nirik> :)
18:19:43 <nirik> #topic Learn about: squid - nirik
18:20:06 <nirik> so, not really prepped much on this, but I know how it's setup, so hopefully I can provide a nice overview to everyone.
18:20:22 <nirik> We run one squid caching proxy server: kojipkgs01.phx2.fedoraproject.org
18:20:27 * aikidouke grabs popcorn
18:20:53 <nirik> it runs squid and behind that a local apache that serves packages from a nfs mount.
18:21:09 <nirik> these are all the packages you download from koji... rpm builds.
18:21:39 <nirik> this includes the koji builders, that download lots of rpms to build mock roots that build other packages in.
18:22:16 <nirik> so, most hits it gets are from the local koji builders, but it does get some hits from the outside net of people downloading specific packages, etc.
18:23:05 <nirik> it's setup with 64GB memory and 150GB or so disk
18:23:14 <nirik> we try and have it cache to memory as much as possible.
18:23:35 <nirik> this makes things faster for builders/downloads and takes some strain off our netapp nfs server.
18:24:03 <nirik> Currently this box is a bit of a single point of failure, as when it's down no builds can happen, etc.
18:24:22 <nirik> We have thought some about spinning up another one and having a pair for HA, but haven't gotten to it.
18:24:45 <nirik> anyone have any questions on the setup?
18:25:06 <aikidouke> seems pretty straightforward, thanks nirik
18:25:12 <nirik> In theory we could just remove it and use koji directly, but it would be slower and more strain on the setup. ;)
18:25:49 <nirik> Theres an outstanding bug on it I think if someone wants to dive into trying to fix it:
18:26:01 <janslow_> this might be a stupid question or not nirik(?), and maybe one about our infrastructure as a whole; but what OS does it run? Fedora/RHEL?
18:26:33 <nirik> https://fedorahosted.org/fedora-infrastructure/ticket/4507
18:26:40 <nirik> janslow: not a stupid question at all.
18:26:46 <nirik> it's running rhel7
18:26:55 <janslow> do we run any Fedora servers in prod?
18:27:02 <tflink> most of the qa stuff is fedora
18:27:05 <nirik> yep. Lots of them.
18:27:14 <janslow> alright, so there's a mix- good to know
18:27:17 <nirik> all the builders are currently f21 (going to move them to f23 soon)
18:27:41 <dotEast2015_> nirik, so squid as you mentioned is SPOF, but what about the backend netapp filer?
18:27:56 <nirik> we try and prefer rhel when possible, as it's much less updates churn... but we use Fedora where it makes sense to do so
18:28:18 <puiterwijk> dotEast2015_: that's another SPOF for about everything in the build infra. But that's normally reasonably stable. (unless when it isn't)
18:28:23 <nirik> dotEast2015_: it's a pair of filers in a HA config (at least my understanding, we don't directly have access to the bare hw there)
18:29:54 <nirik> It would be fun sometime to write up all the SPOF type things we have left and see if we could come up with ways to mitigate/fix them.
18:30:13 <dotEast2015_> hmm. I see. and does infra has access to other set of filers, other than those in phyx?
18:30:32 <nirik> or multiple points of failure, like our dns servers when I push out a change that breaks them all. ;(
18:30:45 <nirik> nope, our main datacenter phx2 is the only one with netapps.
18:31:19 <nirik> everywhere else we use local disk
18:31:25 <dotEast2015_> would glusterfs be of use in this situation?
18:31:32 <puiterwijk> nirik: the dns stuff needs a bind checkconfig before restart
18:31:57 <nirik> dotEast2015_: well, possibly, but it's a ton of disk. Our koji mount is 41TB.
18:32:02 <nirik> puiterwijk: yes.
18:32:36 <nirik> anyhow, anything else on squid?
18:33:11 <nirik> #topic Open Floor
18:33:18 <nirik> anyone have anything for open floor?
18:33:18 <dotEast2015_> yes, is the HA setup already planned or is it on the todo list?
18:33:36 <nirik> dotEast2015_: it's on a "we should get to that someday" list. ;)
18:33:41 <bitlord> I have a question about apprentice work day
18:33:46 <dotEast2015_> :) I see
18:33:54 <nirik> it's hard to prioritize that kind of thing when the single one works fine
18:34:01 <nirik> bitlord: fire away.
18:34:21 <bitlord> because I don't know when I'll be available, how long will it last? whole day (on all timezones?)
18:34:50 <bitlord> I would like to try to do something, also I need to ask to be added to fi-apprentice again, and try to explore our infrastructure
18:35:08 <nirik> bitlord: yeah, I am sure I (and I hope others) will sleep sometime, but will try and be around all day as much as I can
18:35:25 <nirik> I can re-add you anytime. ;)
18:35:46 <bitlord> ok, thanks ;-)
18:35:53 <linuxmodder> nirik,  can you confirm my reply  to kevin's email went from yesterday  got a  weierd bounce back
18:36:09 <nirik> linuxmodder: yes, I got your email... twice I think even. ;)
18:36:37 <linuxmodder> sorry fro the noise it must have gone to ml but not  kevin then  my apol.
18:36:45 <nirik> no worries.
18:37:02 <nirik> anything else for open floor? questions, comments, ideas, favorate cookies?
18:37:04 <linuxmodder> the way the bounce back was writtent couldn't tell
18:37:33 <linuxmodder> I may have some for post mtg  ( $dayjob had me  miss the mtg re-reading  scrollback
18:38:09 <nirik> sure, anytime. ;)
18:38:27 <nirik> ok, lets continue over in #fedora-admin, #fedora-apps and #fedora-noc. ;)
18:38:31 <nirik> thanks for coming everyone!
18:38:36 <nirik> #endmeeting