18:00:01 #startmeeting Infrastructure (2016-04-21) 18:00:01 Meeting started Thu Apr 21 18:00:01 2016 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:01 Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:01 The meeting name has been set to 'infrastructure_(2016-04-21)' 18:00:02 #meetingname infrastructure 18:00:02 #topic aloha 18:00:02 #chair smooge relrod nirik abadger1999 lmacken dgilmore threebean pingou puiterwijk pbrobinson 18:00:02 The meeting name has been set to 'infrastructure' 18:00:02 Current chairs: abadger1999 dgilmore lmacken nirik pbrobinson pingou puiterwijk relrod smooge threebean 18:00:02 #topic New folks introductions / Apprentice feedback 18:00:45 any new folks want to give a short one line introduction of themselves? 18:00:50 or apprentices with questions or comments? 18:01:03 * pingou here 18:01:07 Good evening (finally on time) 18:01:14 * puiterwijk here 18:01:23 * misc is here too 18:01:31 good evening 18:01:38 Sure! My name is Miguel. Software developer / sysadmin by trade. Looking to help where ever I can. 18:01:46 /me is here 18:01:52 hello every one! \o/ 18:01:52 * sayan is here 18:02:21 hello, myself Suraj Narwade, devops engineer willing to contribute my knowledge as well as enhance it 18:02:33 * tflink is here 18:02:51 welcome radioact1ve, snarwade. :) 18:04:05 nirik, I am working on infra-docs, some ansible links are also needs to be updated, 18:04:31 great. Thanks for working on it 18:04:58 any other intros or questions? or shall we move on to status/info? 18:05:30 #topic announcements and information 18:05:30 #info Freed up space on log01, should be good for a bit longer now - kevin 18:05:30 #info db-koji01 moved to faster virthost and local storage - kevin 18:05:30 #info daily emails from ask on the way to being fixed - kevin 18:05:30 #info We are now in Fedora 24 Beta freeze! - everyone 18:05:31 #info rkhunter template updated to use inventory_hostname - kevin 18:05:33 #info pkgs02 /srv backups hopefully on track - kevin 18:05:37 #info composer.stg nfs mounts fixed up - kevin 18:05:39 #info New Basset release with Trac support live - patrick 18:05:41 anything folks would like to add or expand on there? 18:06:12 I've officially taken over from Ralph on leading hubs 18:06:21 cool. 18:06:31 Oh, thats one thing I wanted to mention... 18:06:37 likely going to nicely fill up my plate :) 18:06:55 * misc wanted to speak about fedora cloud wg and openshift instance 18:07:08 in case people didn't know, threebean has moved on to a different place in Red Hat (working on releng tools, etc) He will be very missed. 18:07:23 however, there's now a developer opening on the Fedora Engineering team... 18:07:29 s/will be/is/ 18:07:41 http://paul.frields.org/2016/04/08/fedora-engineering-team-opening-april-2016/ 18:07:48 Wishing him best of luck with that endeavor :) 18:08:05 so, if you or anyone you know meets that description above, do have them apply. ;) 18:08:32 misc: want to make that a discussion item? we can do it after the shortener discussion... 18:09:42 nirik: sure, a #info ? (cause I wasn't able to install gobby on RHEL 7) 18:09:55 #topic url shorteners - kevin 18:10:07 misc: lets do it after this, should be short... 18:10:17 so, I posted a thing about url shorteners to the list. 18:10:27 Feedback on it is welcome there or here. ;) 18:10:42 I don't want to decide anything today, but would be nice to hear feedback on what people think... 18:10:56 I'd prefer to keep using a FOSS solution 18:11:04 even if we do not host it ourself 18:11:09 yeah. 18:11:25 ur1.ca seems pretty fast again now, but I don't know anything about their resources. 18:11:44 maybe we should contact them first see what they think 18:12:45 I could do so, sure. 18:13:04 cydrobolt's offer is also interesting 18:13:12 I was just about to say that ^ 18:13:15 http://2tu.us/ is another free one 18:13:20 and in this regards, may also be easy to contact 18:13:46 pingou: well, but it's a massive bunch of PHP. I'm not sure we want to run more php if we can avoid it 18:13:54 * doteast is late 18:14:15 well, the offer was the software right? we would still have to run it? 18:14:25 puiterwijk: I'm not currently thinking about hosting anything 18:14:34 * nirik would like to avoid more work too 18:15:24 pingou: well, the website that's running on his software that the repo points to was very slow in my tests. So I'm pretty sure that if we start putting load on it, it's also going to be too slow for fpaste 18:15:42 puiterwijk: I haven't tested either 18:15:49 puiterwijk: did you retest ur1.ca recently? or ? 18:15:58 nirik: no 18:16:25 Can see if it works again, sometime 18:16:44 But maybe asking them first would be best 18:17:01 2tu.us seems more active... 18:17:10 and ur1.ca even points to them. 18:17:58 2tu.us is closed it seems.. :) 18:18:00 "Warning: TightURL is not currently accepting new URLs. This is probably temporary." 18:18:24 ah, didn't see that. 18:18:40 You only see that after submitting a URL 18:18:58 fun 18:19:09 ok, so perhaps I will ping ur1.ca and see what they say. 18:20:19 #info will gather more feedback over the next few weeks. 18:20:31 #info nirik will contact ur1.ca and see what their status is 18:20:41 anything else on this now? 18:21:33 #topic Openshift in fedora private cloud status - misc 18:21:40 ok so 18:21:42 misc take it away. ;) 18:22:11 for people who are on the fedora cloud Wg, I have been volunterred to help on installing a openshift instance for fedora 18:22:19 nirik: did you mean, bring it on? :) 18:22:20 https://lists.fedoraproject.org/archives/list/cloud@lists.fedoraproject.org/thread/OTYETAFXAYSHYV3NSFSJOJRTLGPFYVEG/ 18:23:11 #info https://lists.fedoraproject.org/archives/list/cloud@lists.fedoraproject.org/thread/OTYETAFXAYSHYV3NSFSJOJRTLGPFYVEG/ 18:23:20 cool. 18:23:27 so I know various people have been contacted around for discussin gif this was doable, etc, but I wanted frst to make it offficial 18:23:43 patrick was also voluntereed with me, and already did create a tenant on the cloud 18:23:49 so the plan is: 18:24:07 - first, make a test installation on fedora openstack cloud, so we can validate the ansible playbook, etc 18:24:30 - then, make it run on new servers running in a RH DC, once they will have network and also be acquired and shipped 18:24:41 (so far, we have just looked at the type of server) 18:24:54 misc: were those new servers approved? or ? 18:25:10 nirik: jzb said that we have the budget 18:25:15 cool. 18:25:23 but he is in Boston for meeting this week and I was in holiday last week 18:25:30 the goal of the installation is not to run any prod workload, more testing how it work, etc 18:25:51 ie, how it work as a user perspective 18:25:59 It would also note be supported in any way, and data can be gone at any moment. 18:26:01 as a 3rd option if it turns out to work really well in the cloud, we could just add the new machines as more compute power? 18:26:17 just an idle thought 18:26:19 there is several advantage for fedora, such as "seeing how a consumer of docker image would have to deal with image", "help with contairisation of infra", etc 18:26:38 puiterwijk: yeah, you can even said that i said we should kill random stuff using cron :) 18:26:43 (people didn't like the suggestion) 18:26:55 nirik: why not 18:26:56 I'm +1 to it. Chaos Monkey in the cloud! 18:27:03 :) 18:27:09 I think one of the thing that was proposed was to set it on baremetal with atomic 18:27:30 (cause using atomic is also on the list of thing to do, at least for dogfooding and feedback) 18:27:44 sure, ok 18:27:47 but for now, what to do with server is a bit distant 18:27:52 yepp 18:27:57 so I announced that on fedora-cloud 18:28:03 said during their meeting 18:28:10 but the discussion will be on fedora-infra 18:28:19 Also, I think that in the current state we would only open it up to the cloud WG, right? 18:28:35 sounds good. Do keep us posted or let us know if there is anything we can do to help out... 18:28:47 (to limit the number of people that have access to a number that me and misc can personally support, until it's a bit more stable) 18:28:48 nirik: same as any sysadmin a cloning machine would help :) 18:28:52 a bug reppellent too :p 18:29:25 and a magic 'don't need to sleep' pill. ;) 18:29:34 (oh, and before I start a flamewar: "more stable" pointed at our setup, not the software. I don't know about the stability of the software) 18:29:49 puiterwijk: so far, it didn't crash :p 18:29:52 And maybe unlimited budget for hardware to experiment with? :) 18:30:04 anyhow... anything else on this? 18:30:11 misc: then you've been lucky! My setup crashes every 5 minutes :) 18:30:15 nope, nothing on my side 18:30:24 Nothing from me either 18:30:29 the big problem is of course having enough free time for me and puiterwijk at the same time 18:30:55 indeed. 18:31:06 Yeah, and all three components separate are hard enough.. Now they also need to be combined! 18:32:04 #topic Apprentice Open topics 18:32:28 so, we decided last week to try and devote some of the meeting to just talking with apprentices, answering questions or pointing out things to work on... 18:32:51 anyone have anything specific they wanted apprentices to work on? or apprentices with questions about anything? 18:33:09 * nirik pulls up https://fedoraproject.org/easyfix/ 18:33:42 ^ was about to bring that up. I'm scanning it looking for things to tackle 18:34:04 oh, we could talk about: https://fedorahosted.org/fedora-infrastructure/ticket/3639 because I added it to easyfix, but it needs some more discussion... 18:34:24 basically I would like to have our smtp connections use tls/encryption. 18:34:56 but the question is... which certs should we use? new ones? wildcard? self signed? 18:35:05 I was just looking at your latest comment on that ticket and I personally support the idea to use letsencrypt cert for it 18:35:11 nirik, as per last week's discussion for my doubt, i started reading SOPs in fedora infra . 18:35:19 i would like to start on this : https://fedorahosted.org/fedora-infrastructure/ticket/4485 18:35:48 well, in order to use letsencrypt we need it to be automated. I don't want to have to get a new cert every 2 weeks by doing some manual thing. 18:36:02 pingou seems to be busy! 18:36:15 gkadam: cool. :) yeah, we are all busy... 18:36:21 nirik, I am already onto this :) https://fedorahosted.org/fedora-infrastructure/ticket/5128 18:36:41 snarwade: do you have the info you need for now for it? 18:36:55 gkadam: the script needs to be adjusted to collect the info for EPEL as well as Fedora 18:37:01 I can try and look into a way to automate it, haven't used letsencrypt for smpt yet but I can give it a try 18:37:41 nirik, no thanks :) 18:37:57 winterchillz: well, the smtp part doesn't matter, it's ssl certs like anything else that uses certs. ;) 18:38:33 I'll try to come up with a script and give it as much testing as possible then, if that's okay :) 18:39:39 sure, or just come up with a way that might work and report that to the list? 18:39:55 * nirik hasn't used letsencrypt, so not sure how it's setup with our clients 18:40:28 * misc did use letsencrypt 18:40:53 for smtp, it might be a bit harder, since you either need some dns interaction, or http interaction 18:41:29 pingou, so i need to understand the script at https://github.com/pypingou/fedora-owner-change/blob/master/fedora-owner-change.py 18:41:49 * nirik notes they are working on revamping all the way smtp does encryption, but it will be a while until that gets out into any software. 18:41:56 gkadam: that will help :) 18:42:02 pingou, where can i get epel related information ? 18:42:24 gkadam: in the same place as the script gets the Fedora one, in datagrepper :) 18:43:32 gkadam: ping me on IRC if you need more precise help w/ the script, it may be a little hairy :) 18:43:38 so, instead of letsencrypt we could also just use our existing wildcard cert... or get a new digicert one thats got alternates for the hosts it would be used on. 18:44:09 * nirik guesses puiterwijk would be against putting the wildcard cert on bastion* and smtp-mm* servers. 18:44:10 nirik, does it need to be seen 'assigned' status on link ? 18:44:11 pingou, thanks! i will ping you if i am stucked! 18:44:19 I think using the wildcard cert would not be the way to go, since that means we need to trust more servers with it 18:44:24 nirik: hah. How did you figure? 18:45:02 snarwade: you can assign it to yourself if you like, thats fine. 18:45:37 puiterwijk: how many alternates can we specify? can we do one cert with... 5 or so alternates? 18:45:46 nirik: yep, we can 18:45:51 nirik, thanks :) 18:46:09 I think there's no max, or the max is at least >10 18:46:26 oh nice, one of our smtp-mm hosts is not in dns. ;) oops 18:47:44 I would advice against adding it during the meeting... :) 18:47:57 anyhow, I think that might be the way to go unless letsencrypt turns out to be super easy (which I don't think it will be) 18:51:11 I can update that ticket with that info. 18:53:11 misc, can we have a bit of chat regarding letsencrypt at some point, I just have a couple of questions and would be happy to hear your opinion on them if you have the time 18:53:12 any other questions or folks still looking for things to work on? 18:53:40 winterchillz: sure, we can discuss after the meeting on #fedora-admin 18:53:47 cheers 18:55:27 #topic Open Floor 18:55:35 anyone have anything for open floor? questions comments? 18:56:42 I have a question regarding some of the hosts open to the fi-apprentice group 18:56:55 sure, fire away 18:57:57 I've been trying my access around and I just noticed that smtp-mm-ib01 denies my key while smtp-mm-tummy01 is not reachable 18:58:24 I wonder if the list is not completely up-to-date or I'm pressing the wrong buttons around :) 18:58:52 so, looking at the info, they should all be accessable for apprentices... 18:59:20 https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/inventory/group_vars/smtp-mm 18:59:27 fas_client_groups: sysadmin-noc,sysadmin-tools,fi-apprentice 18:59:34 Yes, I grabbed the list as suggested from the hosts_with_var_set script 18:59:38 thats the groups that should be able to access the hosts in that group 19:00:02 so, it might be somehow the way you are getting to them... 19:00:15 are you just doing ssh smtp-mm-ib01.fedoraproject.org ? or ? 19:00:29 Yep, straight from my machine 19:00:37 * winterchillz braces for failure on my side 19:00:52 you may need to use bastion as a jump host 19:01:39 well, that should work, but we can debug it in #fedora-admin after the meeting? 19:01:52 Of course 19:02:05 cool. ;) we are over time... 19:02:10 thanks for coming everyone! 19:02:13 #endmeeting