15:01:53 <smooge> #startmeeting Infrastructure (2019-03-07)
15:01:53 <zodbot> Meeting started Thu Mar  7 15:01:53 2019 UTC.
15:01:53 <zodbot> This meeting is logged and archived in a public location.
15:01:53 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:53 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:01:53 <zodbot> The meeting name has been set to 'infrastructure_(2019-03-07)'
15:01:53 <smooge> #meetingname infrastructure
15:01:53 <zodbot> The meeting name has been set to 'infrastructure'
15:01:53 <smooge> #topic aloha
15:01:53 <smooge> #chair nirik pingou puiterwijk relrod smooge tflink threebean cverna mizdebsk
15:01:53 <zodbot> Current chairs: cverna mizdebsk nirik pingou puiterwijk relrod smooge tflink threebean
15:02:09 <mizdebsk> .hello2
15:02:12 <zodbot> mizdebsk: mizdebsk 'Mikolaj Izdebski' <mizdebsk@redhat.com>
15:02:19 <nirik> happy snowy morning everyone.
15:02:27 <cverna> hi o/
15:03:44 <pingou> .hello2
15:03:45 <zodbot> pingou: pingou 'Pierre-YvesChibon' <pingou@pingoured.fr>
15:04:41 <abompard> .hello2
15:04:42 <zodbot> abompard: abompard 'Aurelien Bompard' <aurelien@bompard.org>
15:05:43 <smooge> #topic New folks introductions
15:05:43 <smooge> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:05:43 <smooge> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
15:05:57 <frailtyy> Hello
15:06:16 <frailtyy> I just sent a greeting to the mailing list but wanted to see the meeting as well
15:06:34 <pingou> hi and welcome frailtyy
15:06:46 <cverna> frailtyy: welcome :)
15:07:11 <smooge> hello
15:07:19 <smooge> #topic announcements and information
15:07:19 <smooge> #info Beta Freeze Begins 2019-03-05
15:07:19 <smooge> #info Staging Koji sync planned for 2019-03-08 (ticket 7600)
15:07:19 <smooge> #info Started working on fedora-messaging migration, status here https://github.com/orgs/fedora-infra/projects/2
15:07:20 <smooge> #info Reminder: use good commit messages
15:07:36 <smooge> #info smooge has given up bad commit messages for lent
15:08:04 <smooge> #info trying to fix pkg02.stg.fedoraproject.org
15:08:19 <smooge> any other announcements?
15:09:39 <smooge> #topic Oncall
15:09:39 <smooge> #info smooge is on call from 2019-02-28 -> 2019-03-07
15:09:40 <smooge> #info mizdebsk is on call from 2019-03-07 -> 2019-03-14
15:09:40 <smooge> #info bowlofeggs is on call from 2019-03-14 -> 2019-03-21
15:09:40 <smooge> #info ?????? is on call from 2019-03-21 -> 2019-03-28
15:09:41 <smooge> #info Summary of last week: (from smooge )
15:10:05 <smooge> So lots of little irqs last week mostly with trying to unbreak things
15:10:11 <nirik> I can be back on the rotation anytime now...
15:10:17 <mizdebsk> ack, i'll be oncall starting today
15:10:19 <bowlofeggs> .hello2 (sorry i'm late)
15:10:20 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <rbarlow@redhat.com>
15:10:33 <smooge> nirik, I will put you down for the last week
15:10:39 <nirik> ok
15:11:37 <smooge> #topic Monitoring discussion
15:11:38 <smooge> #info https://nagios.fedoraproject.org/nagios
15:11:38 <smooge> #info Go over existing out items and fix
15:11:49 <nirik> this has improved a lot since last week
15:12:25 <nirik> we have some machines that dropped disks.
15:12:31 <smooge> yeah
15:12:36 <nirik> no faf fedmsgs... (no idea why)
15:13:22 <puiterwijk> nirik: it might very well just be broken once more. Last I looked, it was out of disk space and refusing any ssubmissions
15:13:22 <nirik> I guess we can re-enable autosign01... it's back up
15:14:22 <nirik> it has disk space... no idea, can mail contact.
15:15:21 <smooge> #topic Tickets discussion
15:15:22 <smooge> #info https://pagure.io/fedora-infrastructure/report/Meetings%20ticket
15:15:37 <smooge> no issues for this week
15:15:46 <smooge> #topic hotfixes - kevin
15:16:21 <nirik> I just wanted to mention hotfixes... I know we all sometimes just apply changes directly... is there something we could do to make hotfixes less anoying to deal with?
15:17:04 <nirik> perhaps a role you pass the filename(s) you are hotfixing to?
15:17:51 <nirik> The current process is:
15:18:08 <pingou> we would need to pass it the local file(s) and the destination, so would be quite similar to just copying it at the right location directly
15:18:09 <nirik> * add task(s) to cp files, add the orig files, add diff of hotfix to files, run playbook
15:18:39 <smooge> the few I did in the last few weeks I tried to make sure they were also commented clearly in the file
15:19:08 <nirik> there's also a patch ansible module I think... we could move to that perhaps, but might be less clear
15:19:13 * pingou guilty of not having done the two steps in the middle on his last
15:19:17 <cverna> for application openShift solves hotfixing :)
15:19:27 <bowlofeggs> i've been striving not to hotfix as much as possible myself
15:19:39 <bowlofeggs> i'd rather make a real build with the fix, and even more a real release
15:19:57 <nirik> cverna: indeed.
15:20:02 <bowlofeggs> there are times i still do a hotfix when i just have to, but i've found that many times you really can take the time to make a release
15:20:28 <bowlofeggs> and when i do a hotfix i work to make a release for it immdiately afterwards
15:20:34 <bowlofeggs> i don't like the hotfix to be present for very long
15:20:49 <bowlofeggs> it's a recipe for regressions
15:20:51 <nirik> then of course there's AGPL... but not sure how many AGPL things we have left
15:21:02 <pingou> fedora-packages?
15:21:07 <pingou> not sure we have much more
15:21:15 <nirik> yeah, packages, but not sure what else
15:21:53 <nirik> anyhow, I guess perhaps the answer is: try not to, if you do use the process, move your apps to openshift. ;)
15:22:17 <abompard> hyperkitty is AGPL I think
15:22:59 <mizdebsk> it should be easy to run an ad-hoc ansible command to list all AGPL-licensed rpm packages we have installed
15:23:00 <puiterwijk> abompard: I can only see GPLv3
15:23:01 <pingou> nirik: that'd be the current approach anyway :)
15:23:34 * nirik nods. ok, just wanted to discuss it in case anyone had any better ideas than I. ;)
15:24:17 <nirik> abompard: FYI, after this meeting in #epel, folks are working on the great python36 rebuild in epel7... might help out your upgrade stuff
15:24:46 <abompard> nirik: ah cool
15:24:47 <cverna> thanks for bringing it, I did not know that we have this process in place
15:24:47 <abompard> thanks
15:25:21 <smooge> ok next topic?
15:25:53 <nirik> yep. move on
15:26:47 <smooge> #topic Priorities for next week?
15:26:48 <smooge> #info please put tickets needing to be focused on here
15:27:04 <smooge> OK we are in a freeze and we have 120 ish open tickets
15:27:47 <nirik> a full tank of gas and chicago is 700 miles away... hit it!
15:27:53 <cverna> it would be cool if we could try to push a bit for this https://pagure.io/fedora-infrastructure/issue/7625
15:28:04 * nirik makes a bad blues brothers joke
15:28:16 <cverna> it is not critical but that would allow us to move resultsDB to fedora-messaging in stg
15:28:28 <smooge> nirik, I will send you to see the nun
15:28:30 <nirik> we can put in a ticket...
15:30:14 <cverna> thanks
15:30:55 <nirik> just FYI... I got a master run completed before/around freeze time
15:31:07 <nirik> there were some failed hosts tho. If I can find time I will look more at those.
15:31:52 <pingou> so that's what changed back the hook in dist-git :) (I was suspecting as much)
15:31:54 <smooge> ok any other tickets needing a foucs
15:32:34 <smooge> #topic Apprentice Open office minutes
15:32:35 <smooge> #info A time where apprentices may ask for help or look at problems.
15:33:01 <nirik> https://paste.fedoraproject.org/paste/Qf1VIv3CKDGQz9cseL-CTA
15:33:09 <nirik> pingou: yeah... likely so.
15:33:44 <cverna> can we get ride of taiga in infracloud ?
15:33:50 <cverna> or not yet
15:34:04 <mizdebsk> we should ask the owner
15:34:06 <pingou> I think not yet, but we may want to advertise that it will go away
15:34:20 <pingou> on the other side mizdebsk has a good point, it's not ours :)
15:34:21 <nirik> yeah, I was waiting until we had the new one more in use...
15:34:27 <cverna> mizdebsk: I am not sure we have an owner anymore
15:34:34 <mizdebsk> Kellin is/was the owner
15:34:39 <puiterwijk> Well, we could just make backups of all the projects, send those to the project owners and shut it down, worst case.
15:34:39 <nirik> I think abompard took over as owner?
15:34:49 <cverna> yes so we don't have an owner anymore
15:35:27 <abompard> it was just for testing the auth plugin
15:35:33 <nirik> ah, ok.
15:35:38 <abompard> I don't need it anymore
15:35:42 <puiterwijk> abompard: you touched it last, so you own it now! :P
15:35:48 <pingou> ^^
15:35:50 <abompard> puiterwijk: looks like it :-)
15:35:51 <nirik> ha
15:35:53 * puiterwijk thinks he should be careful with that suggestion....
15:36:06 <cverna> abompard it is yoursssss
15:36:10 <abompard> puiterwijk owns everything.
15:36:12 <puiterwijk> There's so much that would then immediately fall to me... oh wait, it does.
15:36:37 <puiterwijk> Anyway, I think we should start setting up a timeline for shutting taiga.fic.o down.
15:36:39 <nirik> anyhow, lets wait until the new one is a bit more established, then look at retiring the old one
15:36:45 <smooge> #topic Open Floor
15:36:46 <puiterwijk> It's easy enough for users to get backups and import to teams.fp.o
15:36:52 <smooge> ok since we move it to that
15:37:09 <nirik> there is also old cloud retirement, could be tied to that too, or related to it.
15:37:23 <smooge> I think we do taiga first and then old-cloud
15:37:41 <smooge> old cloud can have a power off date of July 4th
15:37:47 <vrutkovs> Hi! I'd like to experiment with setting up Matrix server for fedora - where do I start? Apply as an apprentice?
15:38:01 <nirik> we should not set any date on it yet, unitl we have the new setup working
15:38:04 <vrutkovs> discussion at https://pagure.io/fedora-infrastructure/issue/7399
15:38:36 <nirik> vrutkovs: well, I think we are all reluctant to run another service...
15:38:53 <nirik> but could perhaps provide you resources to work on it yourself
15:40:03 <nirik> we could make a cloud instance, but I hate to add things to our old cloud
15:40:23 <smooge> I think this would be a good thing to look at after the new-cloud openshift is in place
15:40:30 <vrutkovs> nirik: right, I think I could manage it on my own, ideally in an openshift namespace
15:40:40 <bowlofeggs> would there be concerns about running a communications system?
15:40:49 <smooge> yes yes there are
15:41:22 <pingou> gdpr my love
15:41:29 <nirik> yeah, if you can wait until we have the community openshift going we could experiment there...
15:41:30 <vrutkovs> GDPR and such? There is a bot which requires consent, but a blessing from legal would be great
15:42:15 <vrutkovs> sure, there's no rush
15:42:36 <vrutkovs> except "all the kool kids like KDE already have their Matrix servers" :P
15:42:36 <bowlofeggs> vrutkovs: well you also have to support the SAR requests and right to be forgotten
15:42:39 <nirik> cool. I really hope to work on it, but then there's weeks like this one... where I don't get to anything on my list. ;(
15:42:51 <bowlofeggs> and if fedora hosts that, we can't rely on community members to handle those requests
15:43:20 <vrutkovs> bowlofeggs: right, its an admin task - there are scripts to kick and erase the user
15:43:43 <mizdebsk> bowlofeggs, if community member doesn't handle it then we can handle it by removing the openshift project
15:44:04 <bowlofeggs> heh true
15:44:34 <bowlofeggs> comms systems are trickier than the other kinds of things we host
15:44:42 <bowlofeggs> including mailing lists, because those are explicitly public
15:45:29 <smooge> ok anything else?
15:45:31 <nirik> well, we would want to make sure we sort that out before we run it as fedoraproject.org... but could run a test setup with fedorainfracloud.org I think.
15:46:36 <cverna> I would like to run a zulip (https://zulipchat.com/) instance :)
15:46:42 <vrutkovs> certainly - for a prod we'd need DNS TXT entries and certificates
15:47:00 * cverna start the chat war
15:47:07 <smooge> I would like people to pay for their own servers in AWS
15:47:15 <smooge> there I have said it..
15:47:18 <cverna> smooge :D
15:47:38 <vrutkovs> bowlofeggs: you mean scaling? We could encourage people to use other instances and federate if the load becomes unbearable
15:47:56 <bowlofeggs> vrutkovs: no, i mean with respect to law
15:47:57 <vrutkovs> however a test instance inside RH can handle ~600 chatrooms on a modest bare install
15:48:03 <smooge> vrutkovs, there are network issues, more legal issues than gpdr
15:48:30 <vrutkovs> bowlofeggs: ah, true, I don't know much about that, I hope Matrix/KDE folks could help us with these
15:48:31 <smooge> it is a quagmire
15:48:32 <bowlofeggs> vrutkovs: if the server is in the US, it is also subject to US warrants, for example
15:48:53 <bowlofeggs> including secret warrants
15:49:18 <bowlofeggs> which doesn't really apply to mailing lists since they are completely public
15:49:33 <smooge> vrutkovs, most of the community orgs usually don't think about dealing with them and say it isn't a problem until they get some sort of major problem
15:49:34 <bowlofeggs> it's because matrix supports "private" chats
15:49:47 <nirik> bowlofeggs: there's private mailing lists too... ;)
15:49:59 <bowlofeggs> nirik: ah interesting,i didn't realize we hosted those
15:50:01 <vrutkovs> smooge: we could enforce using plain-stored chats, no e2e (although its possible)
15:50:09 <nirik> not many, but a few
15:50:13 <bowlofeggs> nirik: but, i would think in that case teh e-mail provider would be served a warrant and not us
15:50:18 <bowlofeggs> for matrix, we would be served
15:50:25 <bowlofeggs> ianal of course ☺
15:50:38 <vrutkovs> in the end its all just entries in postgresq, I imagine it would be easy to provide matching entries
15:51:05 * nirik doesn't think we are going to solve this here... :)
15:51:12 <smooge> yep
15:51:26 <smooge> I think this is not something we can solve or debate to a solution
15:51:58 <vrutkovs> agree, lets comment in https://pagure.io/fedora-infrastructure/issue/7399 - I'll try reach out to Matrix folks about legal problems
15:52:02 <smooge> so let us end this meeting and move on to problems we can deal with..
15:52:22 <cverna> thanks all
15:52:35 <cverna> thanks for chairing smooge
15:52:55 <abompard> thanks
15:53:13 <mizdebsk> thanks
15:53:14 <frailtyy> thanks!
15:53:15 <pingou> thanks for chairing smooge :)
15:55:27 <smooge> #endmeeting