18:00:02 <nirik> #startmeeting Infrastructure (2016-02-18) 18:00:02 <zodbot> Meeting started Thu Feb 18 18:00:02 2016 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:02 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:02 <zodbot> The meeting name has been set to 'infrastructure_(2016-02-18)' 18:00:02 <nirik> #meetingname infrastructure 18:00:02 <zodbot> The meeting name has been set to 'infrastructure' 18:00:03 <nirik> #topic aloha 18:00:03 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk pbrobinson 18:00:03 <nirik> #topic New folks introductions / Apprentice feedback 18:00:03 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pbrobinson pingou puiterwijk relrod smooge threebean 18:00:09 * threebean waves 18:00:19 <nirik> morning everyone 18:00:28 <aldapetr> hi everyone 18:00:33 <smdeep> .hello smdeep 18:00:34 <nirik> any new folks like to give a short one line introduction? or any apprentices with questions or comments? 18:00:34 <zodbot> smdeep: smdeep 'Sudeep Mukherjee' <smdeep@gmail.com> 18:01:07 * sayan is here 18:01:08 <tflink> .hello tflink 18:01:09 <zodbot> tflink: tflink 'Tim Flink' <tflink@redhat.com> 18:01:21 <smdeep> nirik, I just realised that my reply to the monthly email has not gone out 18:01:25 <smdeep> I will email tomorrow 18:01:35 * relrod here, but has to jump out and run to a meeting on campus 18:01:39 <tflink> some day I'll memorize the various introduction macros in zodbot :) 18:02:02 <nirik> smdeep: no problem 18:02:14 <smdeep> nirik, thanks 18:02:27 <nirik> ok, on to announcements then... 18:02:40 <nirik> #topic announcements and information 18:02:41 <nirik> #info more spring cleaning - kevin 18:02:41 <nirik> #info mass update/reboot cycle yesterday for glibc and other updates - kevin 18:02:41 <nirik> #info New FMN release for taskotron - ralph 18:02:51 <nirik> any other announcements or status things? 18:02:52 <threebean> that one's still in progress, fwiw. 18:03:20 <nirik> threebean: is there any way to adjust these updates to say 'admin updated foo's filters' or something? 18:03:32 <nirik> everytime it happens someone asks why it showed they updated their filters when they didn't. :) 18:04:01 <nirik> not a great biggie tho 18:04:15 <threebean> heh, agreed. 18:04:19 * threebean files a ticket on the FMN queue for it. 18:04:43 <nirik> or I guess it could supress them entirely? but then it's kinda behind your back... 18:05:09 * threebean nods 18:05:16 <nirik> #topic Spring cleaning progress - kevin 18:05:23 <threebean> some people expressed that while it was confusing, they like being notified that something changed on them. 18:05:45 <nirik> so I have been doing reinstalls, making some progress. I think next week we should have most of the vmhosts done and people can redo vm's... 18:05:56 <nirik> all of staging should be ok to reinstall. 18:06:24 <nirik> hopefully we can be pretty much done by the end of the month aside some stragglers. 18:06:33 <smooge> after that will be doing a password change on things? 18:06:52 <nirik> which passwords did you have in mind? 18:07:34 <nirik> I'd be fine with changing db passwords, etc... 18:07:50 <nirik> I do not want to force users to change without some better reasons. 18:08:25 <nirik> I'm also going to try and play with db replication and see if we can migrate db's with limited downtime, we will see how that goes. 18:08:36 <threebean> heh, that one's long on the list :) 18:09:08 <nirik> yeah, but current rhel7 postgres might make it possible. 18:09:16 <nirik> without a bunch of 3rd party junk 18:10:01 <threebean> cool, cool. 18:10:14 * nirik has a thought, checks something 18:11:31 <nirik> aside the db hosts and a few corner cases like value01.stg... could we just mass destroy *stg* and rebuild it all at once? 18:11:55 <smooge> I was thinking about the root password and the db passwords. Users can do what they want. 18:12:09 <nirik> smooge: yeah, I am fine with changing them. 18:12:48 <nirik> I'll think more on the stg thing... if I can figure a way, I might just mass destroy/create it in the next few days so it's all done. 18:13:01 * jflory7 has a question to ask when it gets to Open Floor 18:13:03 <nirik> anything else on spring cleaning? 18:13:39 <threebean> mass destroy/create of stg will likely let some moths out. 18:13:59 <nirik> it might... yeah. 18:14:28 <threebean> I'll try to be around to help debug cranky systems. 18:14:29 <nirik> #topic easyfix tickets ideas - kevin 18:14:36 <nirik> threebean: thanks. ;) 18:14:53 <jflory7> My question might be of mutual interest to easyfix tickets too 18:15:11 <nirik> so, I was talking to stickster the other day, and we talked a bit about onboarding new people and such, and our easyfix story kinda blows. ;( 18:15:25 <nirik> right now we have a number of easyfix tickets, but they are all really old 18:15:36 <nirik> and some of them it's unclear are really easyfixes. 18:15:53 * dotEast2015 checks in late 18:16:13 <nirik> I know it's hard to file them. It's hard to think of things you want done, but someone without much access can do 18:16:25 <nirik> and sometimes it's so much easier to just do it and it's done. 18:17:03 <nirik> so, I think we should try and be more proactive on our easyfixes. If they haven't been solved in a while (a month? 6 months/release? something) we just do them or unmark them easyfix 18:17:30 * threebean nods 18:17:33 <nirik> and we should try and brainstorm some more to file and kinds of easyfixes that are good... 18:17:53 * linuxmodder stumbles in late 18:17:54 <linuxmodder> the story ? 18:18:06 <nirik> linuxmodder: read back. ;) 18:18:10 <threebean> linuxmodder: trying to figure out better onboarding/easyfix strategy. 18:18:34 <linuxmodder> I am 18:18:48 <linuxmodder> was not getting the easyfix story sucks part the rest made sense 18:18:48 <nirik> so one thing we could do is go over them in meetings again... thats kinda a drag sometimes tho... 18:19:12 <nirik> oldest one currently 18:19:23 <nirik> .ticket 2888 18:19:25 <zodbot> nirik: #2888 (Implement HTTP Strict Transport Security) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/2888 18:19:47 <nirik> thats mostly waiting on changes at a pretty high level, there's not much for any apprentice to do? or am I missing something? 18:20:01 <threebean> no, sounds right. 18:20:28 * nirik unmarks easyfix 18:20:49 <nirik> .ticket 3294 18:20:51 <zodbot> nirik: #3294 (Enable varnish caching for applications) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/3294 18:21:04 <nirik> this has a patch pending puiterwijk looking at it, but he's been traveling so much he's not had time. 18:21:12 <threebean> like.. there are deep changes, and broad changes. hsts is "easy" in that it's just "turn on some headers", right? but it requires thinking about side effects on all our apps and that can go on and on. 18:21:46 <nirik> threebean: yeah, and there's about 50 sites to look into, etc... so I wouldn't call it easy 18:21:51 <threebean> ideally, easyfixes are neither broad nor deep. 18:22:01 <nirik> .ticket 3748 18:22:04 <zodbot> nirik: #3748 (Automate review stats) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/3748 18:22:45 <nirik> dbruno was working on this one, but it's unclear if he has time to 18:22:45 <linuxmodder> the varnish bit is what exactly? 18:23:17 <nirik> linuxmodder: varnish is a cache... so cache commonly accessed pages. 18:23:40 <linuxmodder> so like a cloudfare style front edge cache ? 18:23:42 <nirik> this one has some scripting required. Basically it's making a report of who did package reviews and sponsoring over the last week. 18:24:17 <linuxmodder> nirik, re:3748 can't datgrepper pull some the leg work of that? 18:24:21 <nirik> this one might be easier after bugzilla fedmsgs or if we scrap bugzilla for reviews and move it to it's own app 18:24:27 * threebean nods 18:24:34 <threebean> nirik: both true there. 18:24:40 <nirik> linuxmodder: yeah, I guess, I don't know what cloudflare uses. ;) 18:24:49 <threebean> nirik: in the meantime, we might ask the commops crew if any of them are interested in taking on that scripting work. 18:24:50 <nirik> linuxmodder: we don't have fedmsgs from bugzilla (yet) 18:25:12 <nirik> threebean: ok, how best to do that? 18:25:22 <linuxmodder> threebean, noted -- will add to etherpad for tonight power session for commops 18:25:35 <nirik> .ticket 4485 18:25:36 <threebean> nirik: ^^ boom. linuxmodder, thank you :) 18:25:37 <zodbot> nirik: #4485 (run POC-Change script for EPEL) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/4485 18:25:42 <nirik> linuxmodder: cool. thanks. 18:25:46 <nirik> this one is somewhat similar... 18:26:10 <nirik> this is just running an existing script we have for fedora also for epel... so nothing much new needs writing I don't think. 18:26:32 <nirik> so, this one seems like a good easyfix to me. :) any apprentices up for it? 18:26:39 <pcreech> o/ 18:26:45 <threebean> yay! 18:26:59 <nirik> .ticket 4507 18:27:01 <zodbot> nirik: #4507 (kojipkgs redirects from https to http for directories with missing trailing slashes) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/4507 18:27:12 <linuxmodder> nirik, that was 3748 right? 18:27:15 <nirik> pcreech: you want it? I'm happy to help point you the right way... 18:27:24 <nirik> linuxmodder: yep. 18:27:30 <pcreech> nirik: Yeah, i'll take it. Doesn't seem to be too dificult 18:27:46 <nirik> I am not sure 4507 is an easyfix. It's a weird case between squid and apache and no one seems to be able to track it down. 18:28:05 <pcreech> oh, sorry, content switched there and i didn't notice 18:28:16 <pcreech> one sec... looking.. 18:28:26 <linuxmodder> noted on tonight's etherpad nirik threebean 18:28:28 <nirik> I am going to unmark 4507 then too. 18:28:55 <pcreech> nirik: eh, sure, i can take that one as well. 18:29:08 <nirik> pcreech: which one? 18:29:12 <pcreech> 4507 18:29:17 <linuxmodder> sounds like rsync foo/ bar kinda thing i can take a stab over the weekend (4507 18:29:26 <nirik> pcreech: it's not simple sadly. 18:29:44 <linuxmodder> nirik, pcreech and I can tag team it if needed :) 18:29:44 <nirik> it's squid and apache and https mixxed up. It's not at all a clear case. ;) 18:29:45 <pcreech> nirik: That's what peaked my interest :) 18:29:55 <nirik> ok, if you folks want to, go for it. ;) 18:30:18 <nirik> .ticket 4941 18:30:20 <zodbot> nirik: #4941 (Tarball of all current specfiles) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/4941 18:30:26 <threebean> maybe best to start by making a diagram for that one, pcreech, linuxmodder 18:30:36 <threebean> to lay out how the pieces interact. 18:30:48 <nirik> dotEast2015 already added a patch here, but we havent tested it. ;( 18:30:50 <linuxmodder> threebean, who needs graphs and diagrams ? :) 18:31:03 <linuxmodder> brb 18:31:09 <pcreech> threebean: good suggestion! 18:31:21 <nirik> I guess I can try... or threebean / pingou: you have any cycles to test this in stg? 18:31:27 * threebean clicks 18:31:37 <dotEast2015> Yeah, once tested I can push it 18:31:39 <nirik> .ticket 4973 18:31:40 <zodbot> nirik: #4973 (add nagios check for mailman01's REST interface.) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/4973 18:31:50 <threebean> nirik: yeah, I got this one. 18:31:59 <nirik> threebean: super! 18:32:16 <nirik> abompard: where did we leave ticket 4973? I think we needed some kind of auth? or ? 18:32:21 * nirik can update that ticket asking 18:32:50 <dotEast2015> Threebean. So 4941 is with u? 18:33:11 <nirik> .ticket 5096 18:33:12 <zodbot> nirik: #5096 (Add "Connection Info" back into CommBlog) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/5096 18:33:23 <threebean> dotEast2015: yup 18:33:24 <nirik> puiterwijk was going to do this one, will just remove easyfix. 18:33:51 <dotEast2015> Cool 18:34:12 <nirik> ok, I think thats all of them. 18:34:30 <threebean> phew. 18:34:48 <threebean> nirik: going over those every week could indeed be a bummer.. if we don't have new news or anything like that. 18:34:51 * linuxmodder back 18:35:00 <threebean> maybe once a month? but then there's extra overhead involved in trying to remember to add it to the agenda. 18:35:23 <nirik> yeah, I could try and add it to my apprentice nag process I guess... 18:35:26 <nirik> thats the first of the month. 18:35:30 <linuxmodder> not touching the rest one -- still too new to that 18:35:53 <nirik> Will try that and see if we get anywhere. :) 18:35:57 <threebean> :) 18:36:21 <threebean> if the easyfix queue gets more slim, then making we take another focused crack and trying to brainstorm things to fill it back up. 18:36:22 <nirik> Also on the ticket topic. I plan latter this year sometime to move to pagure.io 18:36:28 <linuxmodder> nirik, or add to the monthly infra-ap mailing 18:36:47 <nirik> threebean: yeah 18:37:07 <nirik> when we move to pagure.io I am going to be doing some pretty ruthless pruning of old tickets. ;) 18:37:20 <nirik> just FYI 18:37:24 <aikidouke_2> yay! 18:37:27 <nirik> ok, anything more on this? or moving along... 18:37:38 <nirik> #topic Finding a new fedbadges admin - ralph 18:37:43 <nirik> threebean: you're up 18:37:50 <threebean> hey, so, badges.fedoraproject.org is one of the things I've been responsible for. 18:38:12 <threebean> I've been focusing more on other stuff recently and I figured it was a good candidate to try and hand off to one or more people to manage. 18:38:30 <nirik> this seems like something to me a group/several people would be good to work on. 18:38:32 <jflory7> What all goes into the duties of a fedbadges admin? 18:38:35 * threebean nods 18:38:46 <threebean> there are three different slices of work, last I thought about it: 18:38:59 <threebean> - sysadmin work to keep the server up. it is usually stable so this is light-duty. 18:39:25 <threebean> - ticket-triage, people-facing work. this involves handling the new-badge-idea queue of tickets and politely working with people to help them with their ideas, connect them with designers, etc.. 18:39:50 <threebean> - and lastly, a development role that keeps needing attention. some new badge ideas are just not possible out of the box.. so they require new scripting to make happen. 18:40:34 <threebean> i'm most concerned about the 2nd role - the ticket triage stuff. people seem to really like the badges system, and they come to the ticket queue with their best ideas. lately I haven't felt I'm able to respond with adequate attention to match their .. passion. 18:40:38 <jflory7> I think this might be something of interest to CommOps too, if decause is around 18:40:42 * threebean nods 18:41:02 <jflory7> I could volunteer myself for the Trac-fu sort of stuff... I enjoy doing that kind of stuff anyways. 18:41:18 <threebean> this could be a good fit :) 18:41:42 <nirik> there's also some bugs... like some of the longer age badges don't seem to be getting awarded... but I never seem to get time to look into it. ;) 18:41:44 <jflory7> I probably wouldn't be of much use for the first and third roles, but I could definitely give a hand for the second 18:41:50 * threebean nods 18:41:54 <threebean> nirik: yup. 18:41:55 <jflory7> But I would definitely get this on CommOps radar too 18:42:22 <threebean> jflory7: thanks, that's much appreciated. 18:42:40 * aikidouke_2 will take a look at the scripting 18:42:49 <nirik> https://fedorahosted.org/fedora-badges/ticket/392 18:42:59 <aikidouke_2> doesn't sound like you need much help keeping the server from falling over tho? 18:43:04 <linuxmodder> sorry got sidetracked in #fedora 18:43:05 <jflory7> For the current moment, though, I think I can accept some increased responsibility for the Badges Trac triaging and ticket organization. 18:43:11 <nirik> the server has been pretty trouble free overall 18:43:15 * linuxmodder reading back 18:43:52 <threebean> aikidouke_2: for that ticket #392, the cronjob that awards it lives here https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/badges/backend/files/cron -- those are the "lifecycle" badges 18:44:13 <jflory7> threebean: Could you (or another chair) do a #help message with a one-liner description of the fedbadges admin role? This will auto-fire a notification in #fedora-commops :) 18:44:16 <threebean> so.. it's a matter of looking for the logs of that thing and poking through to figure out what's getting stuck. 18:44:22 <threebean> jflory7: sure, one sec. 18:44:47 <aikidouke_2> logs! sure, I will take a look at that 18:45:03 <threebean> #help infra team is looking for people to step up and fill roles for badges.fedoraproject.org: 1) sysadmin, 2) ticket triage, and 3) dev. 18:45:15 <linuxmodder> jflory7, can bring it up later in our powwow 18:45:23 * pcreech can help some with #3 18:45:53 <nb> threebean, /me is willing to help 18:46:04 <nb> i can help more with #1 and #2, not sure how much good i would be with dev 18:46:07 <nb> but i can sysadmin and ticket 18:46:10 <dotEast2015> I can help with #1, just need more details 18:46:13 <jflory7> threebean++ linuxmodder++ 18:46:25 <threebean> awesome. let's nominally put "commops" in charge (if they're willing). then the dev and sysadmin work kind of flows from there. 18:46:46 <threebean> if people with badges ideas want X or Y to happen, that kinda sets the direction for what to script up. 18:46:54 <nirik> cool. ;) 18:46:59 <threebean> often people will come to the ticket queue with an impossible idea.. 18:47:16 <threebean> so, it would be kind of up to commops/jflory7 to alert the dev crew that input is needed on a new idea. 18:47:29 <threebean> you gotta somehow break it to the person that the idea either needs to be reformed, or reconsidered. 18:47:32 * corey84 for jflory7's cookie pushing spree :) 18:47:52 <nirik> excellent. Anything more on this? 18:48:10 <corey84> threebean, define impossible 18:48:18 <threebean> corey84: heh, that's up to you I guess. 18:48:20 <jflory7> corey84: You'll have to wait until F24 for more cookies from me. ;) 18:48:29 <jflory7> threebean: Acknowledged. 18:48:44 <threebean> nirik: nothing more from me. I'll plug dotEast2015 and any other sysadmin-focused folks into the SOPs and groups we have. 18:49:04 <nirik> alrighty. ;) 18:49:08 <jflory7> threebean: For getting up to speed about the Badges Trac, is there a way I can get the rundown of current practices or find information to pick up a larger role with triaging and corralling? 18:49:28 <threebean> jflory7: no, unfortunately. I've just been doing it since we launched the thing. 18:49:38 <threebean> jflory7: but, I can braindump about it after the meeting. 18:49:43 <nirik> jflory7: there's a pretty detailed workflow on the trac... and in the tickets. adamw set it up 18:49:50 <threebean> oh - that's right. forgive me. 18:50:03 <nirik> but not sure how documented it is actually 18:50:04 <threebean> adamw programmed me. 18:50:15 <jflory7> threebean++ If you're around anywhere from 6pm - 9pm our time, decause, CommOps, and I will be doing a Hangout session later on 18:50:25 <jflory7> nirik: Ahh, yeah -- I'll make sure to check there too. 18:50:28 <jflory7> nirik++ 18:51:18 <dotEast2015> Cool.. 18:51:31 <nirik> #topic Open Floor 18:51:35 * jflory7 has something 18:51:42 <nirik> ok, I didn't have any teach about today, so on to open floor. 18:51:46 <nirik> jflory7: go ahead. 18:52:27 <jflory7> So, this has to do with two things: (1) https://github.com/fedora-infra/ and (2) https://fedorahosted.org/famna/ticket/131 18:52:53 <jflory7> Over in FAmNA, we're looking at attending a hackathon in the next month at a university campus. We're the sponsors of the open source category, and we're able to present projects for hackers to work on or get involved with. I was wondering if anyone here might be interested to help put together some projects that students might be able to hack on in a 48 hour window? 18:53:01 <jflory7> Details will be finalizing over the next week, but I was wondering if there was anyone who might be interested in helping out with letting FAmNA know what projects we could bring to the hackathon for students to work on and submit some code to. 18:53:15 <jflory7> You wouldn't be obligated to help with event logistics of course, but having someone from Infra advise on some projects might be useful for planning purposes. :) 18:53:16 <jflory7> eof 18:53:50 <pingou> something like the easifixes or something else? 18:53:51 <nirik> well, there is existing easyfix tickets, but not sure thats the perfect fit... dunno 18:54:07 <threebean> yeah, I think they want something bigger than easyfix, but not too big. :/ 18:54:23 <pingou> we have a few tickets that might qualifies for this 18:54:35 <pingou> but just setting up the projects often take a little bit of time 18:54:46 <threebean> oo - I don't know how to advise it, but it would be sweet to be able to translate jenkins-job-builder inputs into taskotron inputs automatically. 18:54:51 <jflory7> Yeah, it's shorter notice than I would have liked to have it 18:54:52 <threebean> that could be a 48 hour project. 18:55:08 <threebean> and then we could re-use a bunch of upstream jenkins tests in our taskotron instance. 18:56:14 <nirik> that could be nice 18:56:50 <jflory7> Maybe also useful if issues in GitHub were marked with a special tag, maybe things relating to small features that could be completed as well 18:56:54 <aikidouke_2> nirik - what about some of the things holding up our rhel6 conversion? 18:57:00 * jflory7 isn't quite familiar with your workflow on Infra projects 18:57:12 <jflory7> But there's a lot of potential to be tapped into over that weekend :) 18:57:13 <aikidouke_2> like a FOSS pastebin that we could theme 18:57:20 <threebean> sayan: if you're around, could you share the hackerearth script that gathered easyfix, medium, and hard issues? 18:57:28 <linuxmodder> jflory7, that is for brickhack right? 18:57:32 <jflory7> linuxmodder: Correct. 18:57:35 <nirik> aikidouke_2: well, lets see... not sure. Someone could try and package askbot, but thats a bunch of work... 18:58:03 <jflory7> linuxmodder: Ooh, you should probably check out that FAmNA ticket as well, there's a lot of info packed into there, your feedback as an Ambassador would also be welcome :) 18:58:35 <nirik> threebean: what work is needed to move packages to rhel7? 18:58:56 * linuxmodder looking at 131 18:59:00 <threebean> hm. TG2 and things in its orbit, iirc. 18:59:22 <nirik> or a rewrite, which would be a lot of work 18:59:35 * threebean nods 19:00:13 <threebean> oh my 19:00:21 <threebean> if there are crypto heads.. and if they could fix sigul. 19:00:27 <pingou> oh :) 19:01:17 <linuxmodder> jflory7, about swag and media have you got that squared off with Southern_Gentlem or nb? 19:01:47 <linuxmodder> threebean, crypto nut here can take a look 19:01:54 <jflory7> linuxmodder: I believe nb was in the process of getting that sent to me, and I also wanted to talk at the meeting tonight about getting the banner shipped to me -- want to talk about this in #fedora-ambassadors? 19:02:19 <nb> jflory7, i have f22 media 19:02:21 <nb> we have no f23 19:02:26 <nb> and i have not much sway 19:02:28 <linuxmodder> sure 19:02:29 <nb> swag 19:02:36 <nb> we are trying to get pens ordered 19:02:40 <nirik> well, sigul has somewhat been fixed... at least it's a lot better. 19:02:47 <nb> but we should probably talk about this in -ambassadors not in the infra meeting 19:02:57 <nirik> anyhow, anything left for this meeting? or shall we close out? 19:03:03 <linuxmodder> jflory7, I can help with making some usb media if someone wants to send out the details 19:03:58 <threebean> nirik: oh? 19:04:05 * threebean goes off to look at the sigul commit log 19:04:21 <nirik> threebean: yeah, been some work. ;) 19:04:42 <nirik> ok, thanks for coming everyone. Do continue in #fedora-admin, #fedora-noc and #fedora-apps 19:04:45 <nirik> #endmeeting