18:00:05 #startmeeting Infrastructure (2015-11-05) 18:00:05 Meeting started Thu Nov 5 18:00:05 2015 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:05 Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:05 #meetingname infrastructure 18:00:05 #topic aloha 18:00:05 #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk pbrobinson 18:00:05 The meeting name has been set to 'infrastructure' 18:00:05 Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pbrobinson pingou puiterwijk relrod smooge threebean 18:00:06 #topic New folks introductions / Apprentice feedback 18:00:17 * mirek-hm is here 18:00:28 * dotEast2015_ here 18:00:32 I'm here, but handling copr and dell 18:00:34 here 18:00:38 n.b. Council Meeting over in #fedora-meeting-1 18:00:45 * jsmith lurks 18:00:48 any new folks like to give a short one line introduction? or apprentices with questions or comments? 18:01:02 * sayan is here 18:01:06 Yes. 18:01:58 I joined with in the last week, I currently work as an intro level sysadmin here in Michigan. 18:02:00 * aikidouke has a question after intros 18:02:07 I think that's about one line. 18:02:08 welcome odin2016. ;) 18:02:10 * bitlord present (just listening) 18:02:11 .hello smdeep 18:02:12 smdeep: smdeep 'Sudeep Mukherjee' 18:02:14 * mpduty is here 18:02:20 I am having net issues 18:02:25 odin2016: you are more interested in sysadmin than application development? 18:02:35 smdeep: happens to us all. :) 18:03:07 right now, yes, but I've not an issue learning some application development. 18:03:35 Hello, my name is Athos. Trying to learn a little more about the team. Just listening here :) Current working with packaging and using Copr at work =D 18:03:41 I feel like you need to at least have an idea what an app is doing if you are going to support any part of it. 18:04:10 odin2016: see me after the meeting in #fedora-admin and I can point you in the right direction to get started. ;) 18:04:21 athos: welcome. :) feel free to chime in with quesions anytime. 18:04:33 nirik: will do. 18:04:36 aikidouke: you had some questions to bring up? 18:04:54 question: when F23 released, mirrorlist was reporting some mirrors that said they had the iso's, but were generating 404s, etc. Should we be checking if the mirror site actually has the iso file? 18:05:12 well, it's not so easy I'm afraid. ;) 18:05:22 ty 18:05:30 kushal is having network issues so he might not able to join in 18:05:30 yes, that would be nice to know, but we cannot check all files on all mirrors... there's just way too many of them 18:05:44 we could perhaps do better with isos specifically... 18:06:02 I had an idea along that line, just checking isos.. 18:06:16 maybe i'll put something together and share it 18:06:18 https://github.com/fedora-infra/mirrormanager2 is the code and adrianr and pingou would be the ones to talk to about it. ;) 18:06:24 thanks 18:06:57 cool. anything else? or shall we move on to status/info? 18:07:32 #topic announcements and information 18:07:32 #info 🌭 🌭 Fedora 23 is released! 🌭 🌭 - everyone 18:07:32 #info REMINDER: apprentice work day 2015-11-18. Join us - everyone 18:07:32 #info REMINDER: nov apprentice status email went out, please reply - all apprentices 18:07:32 #info Nasty power outage last saturday at phx2, many thanks to Patrick for handling it. 18:07:33 #info mailman3 migrations possibly resuming nov 16th - abompard 18:07:35 #info [release] pkgdb2: 1.32.2 - pingou - http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/EBE5VXJFBPSN3UG7BHCTRYX5FCLTOVK7/ 18:07:38 #info First release of statscache deployed. There will be kinks to work out. - threebean / nyazdani / rtnpro - https://apps.fedoraproject.org/statscache 18:07:41 #info New release of tunir for Autocloud backends - kushal - http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/MXXRQHXX6BIRXULY5L5JVUF2X7DJEU2F/ 18:07:46 any other announcements or info to share? 18:08:05 we are now out of freeze, so we can catch up on changes that were pending in the run up to f23 release 18:09:13 ok, on to discussion items 18:09:23 #topic meeting time - kevin 18:09:36 due to DST time for this meeting has shifted in NA... 18:09:47 is everyone ok with this time moving forward? or should we shift it back? 18:10:16 well I just find via the hard way that if you want to use F23 cloud images you need to put at the top "raw: dnf install -y python2 python-dnf libselinux-python libsemanage-python python-firewall" https://fedorahosted.org/cloud/ticket/126 18:10:16 I like it. It actually works better since I work at night for my full time job. 18:10:28 works for me :) 18:10:37 same here 18:10:38 no pref either 18:10:52 so if you want to use ansible with F23 bear that in mind 18:11:35 mirek-hm: yeah, this was discussed on the devel list not long before release. ;( 18:11:55 but yeah we should add that to our f23 cloud setup 18:12:09 #info people seem ok with sticking with 18UTC 18:12:24 #topic fas3 plans - pingou/xavier/kevin 18:12:37 just wanted to highlight the email pingou sent to the list about fas3. 18:12:52 There's a number of things left to do, but we wanted to try and roll it out before too long. 18:13:06 so if anyone is looking for things to do, helping with those items would be great. 18:14:51 * nirik is looking up the link to the post 18:15:04 https://lists.fedoraproject.org/archives/list/infrastructure%40lists.fedoraproject.org/thread/YZIAYVJDQF427A6FGPKFX5BU6AWTS7QG/ 18:15:17 #topic mirror lists migration from @redhat.com - kevin/smooge 18:15:58 so we have 2 lists still hosted at redhat.com that were for mirror admins coordinating releases. 18:16:08 we are going to be moving those over to lists.fedoraproject.org soon. 18:16:24 This is mostly just a headsup/notification of that 18:16:44 tflink: you around? still game to talk about buildbot today? 18:17:00 crap, i completely forgot about that 18:17:22 sorry, I should have reminded you. 18:17:29 want me to talk about squid instead? 18:17:32 it's on my calendar and everything 18:17:42 if you're prepared enough for it, sure 18:17:51 would buildbot be next week, then? 18:18:06 well, not really prepared, but I am fine just winging it. ;) 18:18:12 then it would be dec 5th 18:18:18 sorry, dec 3rd 18:18:56 is dec 3rd ok for you? 18:19:01 yeah, that'll work 18:19:07 cool. 18:19:08 * tflink sets up more/better reminders this time 18:19:40 :) 18:19:43 #topic Learn about: squid - nirik 18:20:06 so, not really prepped much on this, but I know how it's setup, so hopefully I can provide a nice overview to everyone. 18:20:22 We run one squid caching proxy server: kojipkgs01.phx2.fedoraproject.org 18:20:27 * aikidouke grabs popcorn 18:20:53 it runs squid and behind that a local apache that serves packages from a nfs mount. 18:21:09 these are all the packages you download from koji... rpm builds. 18:21:39 this includes the koji builders, that download lots of rpms to build mock roots that build other packages in. 18:22:16 so, most hits it gets are from the local koji builders, but it does get some hits from the outside net of people downloading specific packages, etc. 18:23:05 it's setup with 64GB memory and 150GB or so disk 18:23:14 we try and have it cache to memory as much as possible. 18:23:35 this makes things faster for builders/downloads and takes some strain off our netapp nfs server. 18:24:03 Currently this box is a bit of a single point of failure, as when it's down no builds can happen, etc. 18:24:22 We have thought some about spinning up another one and having a pair for HA, but haven't gotten to it. 18:24:45 anyone have any questions on the setup? 18:25:06 seems pretty straightforward, thanks nirik 18:25:12 In theory we could just remove it and use koji directly, but it would be slower and more strain on the setup. ;) 18:25:49 Theres an outstanding bug on it I think if someone wants to dive into trying to fix it: 18:26:01 this might be a stupid question or not nirik(?), and maybe one about our infrastructure as a whole; but what OS does it run? Fedora/RHEL? 18:26:33 https://fedorahosted.org/fedora-infrastructure/ticket/4507 18:26:40 janslow: not a stupid question at all. 18:26:46 it's running rhel7 18:26:55 do we run any Fedora servers in prod? 18:27:02 most of the qa stuff is fedora 18:27:05 yep. Lots of them. 18:27:14 alright, so there's a mix- good to know 18:27:17 all the builders are currently f21 (going to move them to f23 soon) 18:27:41 nirik, so squid as you mentioned is SPOF, but what about the backend netapp filer? 18:27:56 we try and prefer rhel when possible, as it's much less updates churn... but we use Fedora where it makes sense to do so 18:28:18 dotEast2015_: that's another SPOF for about everything in the build infra. But that's normally reasonably stable. (unless when it isn't) 18:28:23 dotEast2015_: it's a pair of filers in a HA config (at least my understanding, we don't directly have access to the bare hw there) 18:29:54 It would be fun sometime to write up all the SPOF type things we have left and see if we could come up with ways to mitigate/fix them. 18:30:13 hmm. I see. and does infra has access to other set of filers, other than those in phyx? 18:30:32 or multiple points of failure, like our dns servers when I push out a change that breaks them all. ;( 18:30:45 nope, our main datacenter phx2 is the only one with netapps. 18:31:19 everywhere else we use local disk 18:31:25 would glusterfs be of use in this situation? 18:31:32 nirik: the dns stuff needs a bind checkconfig before restart 18:31:57 dotEast2015_: well, possibly, but it's a ton of disk. Our koji mount is 41TB. 18:32:02 puiterwijk: yes. 18:32:36 anyhow, anything else on squid? 18:33:11 #topic Open Floor 18:33:18 anyone have anything for open floor? 18:33:18 yes, is the HA setup already planned or is it on the todo list? 18:33:36 dotEast2015_: it's on a "we should get to that someday" list. ;) 18:33:41 I have a question about apprentice work day 18:33:46 :) I see 18:33:54 it's hard to prioritize that kind of thing when the single one works fine 18:34:01 bitlord: fire away. 18:34:21 because I don't know when I'll be available, how long will it last? whole day (on all timezones?) 18:34:50 I would like to try to do something, also I need to ask to be added to fi-apprentice again, and try to explore our infrastructure 18:35:08 bitlord: yeah, I am sure I (and I hope others) will sleep sometime, but will try and be around all day as much as I can 18:35:25 I can re-add you anytime. ;) 18:35:46 ok, thanks ;-) 18:35:53 nirik, can you confirm my reply to kevin's email went from yesterday got a weierd bounce back 18:36:09 linuxmodder: yes, I got your email... twice I think even. ;) 18:36:37 sorry fro the noise it must have gone to ml but not kevin then my apol. 18:36:45 no worries. 18:37:02 anything else for open floor? questions, comments, ideas, favorate cookies? 18:37:04 the way the bounce back was writtent couldn't tell 18:37:33 I may have some for post mtg ( $dayjob had me miss the mtg re-reading scrollback 18:38:09 sure, anytime. ;) 18:38:27 ok, lets continue over in #fedora-admin, #fedora-apps and #fedora-noc. ;) 18:38:31 thanks for coming everyone! 18:38:36 #endmeeting