14:02:41 <smooge> #startmeeting Infrastructure (2018-10-04)
14:02:41 <zodbot> Meeting started Thu Oct  4 14:02:41 2018 UTC.
14:02:41 <zodbot> This meeting is logged and archived in a public location.
14:02:41 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:02:41 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:02:41 <zodbot> The meeting name has been set to 'infrastructure_(2018-10-04)'
14:02:41 <smooge> #meetingname infrastructure
14:02:41 <smooge> #topic aloha
14:02:41 <zodbot> The meeting name has been set to 'infrastructure'
14:02:41 <smooge> #chair nirik pingou puiterwijk relrod smooge tflink threebean cverna
14:02:41 <zodbot> Current chairs: cverna nirik pingou puiterwijk relrod smooge tflink threebean
14:02:48 <relrod> morning
14:02:52 <aikidouke> good morning!
14:02:54 <bowlofeggs> .hello2
14:02:54 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <rbarlow@redhat.com>
14:02:59 <cverna> morning :)
14:03:00 <bcotton> .hello2
14:03:13 <smooge> sorry for the 2 minute delay... that 30 second email took a lot longer than I realized
14:03:18 <zodbot> bcotton: bcotton 'Ben Cotton' <bcotton@redhat.com>
14:03:40 <smooge> #topic New folks introductions
14:03:40 <smooge> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
14:03:43 <mkonecny> hi
14:03:50 <pingou> ó/
14:03:53 <amrzaki_> Hello all
14:04:17 <odin2016> hel
14:04:18 <odin2016> lo
14:04:21 <odin2016> *hello
14:04:46 <chris787> hello
14:04:54 <chris787> Hello !
14:04:55 <chris787> My name is Christophe, I'm from France, Grenoble.
14:04:55 <chris787> I work as a sysadmin for 3 years now and i'd like to contribute/help the fedora project.
14:04:55 <chris787> I have experience of administering CentOS servers, bash and actually working on a SaaS infra project that includes Ansible, Gitlab ans Openshift (Docker/Kubernetes).
14:04:55 <chris787> I can contribute 3/4 hours per week. My time zone is Europe/Paris.
14:04:55 <chris787> IRC handle : chris787
14:04:55 <chris787> FAS login : chris7878
14:04:56 <chris787> Thanks.
14:05:17 <clime> hello
14:05:40 <cverna> welcome chris787
14:05:43 <abompard> .hello2
14:05:44 <zodbot> abompard: abompard 'Aurelien Bompard' <aurelien@bompard.org>
14:05:46 <chris787> thks
14:05:51 <creaked> .hello2
14:05:52 <zodbot> creaked: creaked 'Will Chellman' <creaked@gmail.com>
14:05:54 <pingou> Bonjour chris787
14:06:11 <odin2016> .hello2
14:06:13 <zodbot> odin2016: odin2016 'None' <davidjshier@gmail.com>
14:06:16 <amrzaki_> Welcome chris787
14:06:23 <chris787> slt
14:06:36 <mkonecny> welcome chris787
14:06:43 <chankuma> Hi, i'm Chandra, i know Python, Linux (RHCSA), Flask, Django.
14:07:27 <mkonecny> welcome chankuma
14:07:47 <cverna> welcome chankuma :)
14:07:53 <smooge> hello chankuma
14:07:57 <smooge> hello chris787
14:08:04 <chris787> hello everybody
14:08:17 <chankuma> If someone can point me to a suitable project (Python, Ansible, Jenkins), that would be hhelpful.
14:08:19 <chris787> where do we start ? :-D
14:08:28 <smooge> #topic announcements and information
14:08:28 <smooge> #info tflink is on extended PTO
14:08:28 <smooge> #info smooge is on PTO 2018-10-06->2018-10-09
14:08:28 <smooge> #info nirik is on PTO 2018-10-24->2018-10-28
14:08:28 <smooge> #info Final Freeze will start 2018-10-09
14:08:30 <smooge> #info Updates/reboots done except for QA/builders
14:08:31 <smooge> #info fedora-tagger/statscache retired - cverna/nirik
14:09:06 <smooge> chris787, https://fedoraproject.org/wiki/Infrastructure/GettingStarted
14:09:54 <mkonecny> this is nice crossroad for newcomers ^^
14:10:08 <smooge> Are there any other announcements for the work done this week or coming up next week?
14:10:49 <pingou> I'm going to try to get pagure 5.1 before the freeze starts
14:10:53 <pingou> in both pagure.io and src.fp.o
14:11:15 <smooge> #info pagure 5.1 may roll out next week
14:11:34 <smooge> pingou, so that is next tuesday
14:11:39 <pingou> yup
14:11:43 <pingou> hoping to do this on Monday
14:11:45 <puiterwijk> #info changes to src.fp.o inbound. look at various lists for more info
14:12:21 <smooge> ok any other announcements?
14:12:34 <smooge> #topic Oncall
14:12:34 <smooge> #info Puiterwijk is on call from 2018-09-27->2018-10-04
14:12:34 <smooge> #info ??? is on call from 2018-10-04 -> 2018-10-11
14:12:34 <smooge> #info smooge is on call from 2018-10-11 -> 2018-10-18
14:12:34 <smooge> #info ??? is on call from 2018-10-18 -> 2018-10-25
14:12:37 <smooge> #info Summary of last week: (from Puiterwijk)
14:13:01 <puiterwijk> Not many pings, bunch of outages today due to reboot
14:13:22 <smooge> I think you said you would take the next week also if no one else would take it?
14:13:37 <puiterwijk> Yeah. Though I believe bowlofeggs said he could take the week except for weekend?
14:13:57 <puiterwijk> But yeah, if not, I can.
14:14:18 <bowlofeggs> i actually can't do that now :/
14:14:41 <puiterwijk> okay. Then I will continue on...
14:14:43 <smooge> I am going to ask nirik if he can take oncall to the 23rd and then I will take from the 23rd to the 1st
14:14:45 <relrod> I will take 10-18 through 10-25
14:14:54 <smooge> oh cool thanks relrod
14:14:54 * nirik arrives late, reads up
14:15:41 <nirik> I can
14:17:08 <smooge> hi nirik
14:17:43 <smooge> #topic torrents and their troubles - kevin
14:17:43 <smooge> #info used more than I realized
14:17:53 <smooge> this is just a point from last weeks meeting.
14:18:07 <nirik> yeah, I didn't have time to update the meeting gobby. ;(
14:18:10 <smooge> I did the analysis that people asked for and found that torrents are used more than I realized
14:18:19 <smooge> oh I kept this on for that reason
14:18:49 <smooge> #info https://smooge.fedorapeople.org/stats-torrents/
14:20:14 <nirik> nice stats smooge. Thank you for making them.
14:20:16 <smooge> it is hard to see how many download requests are really per day. They way the logs go is that if my torrent requests are over multiple hours, I get counted by the tracker as seperate events
14:20:41 <nirik> For now, BTW, I have just kept the rtorrent seeder...
14:21:10 <smooge> So the best average is actually the number of users versus clients per hour which is still reasonable
14:21:39 <smooge> so I have changed from saying 'who will rid me of this troublesome software' to 'well its a pain but what are you going to do?'
14:21:50 <smooge> anyway that was all I had on this
14:22:11 <smooge> next topic
14:22:13 <smooge> #topic Postgrest for fpdc evaluation: https://paste.fedoraproject.org/paste/9CtScjC8be3vmCv1CryFcA - clime
14:22:14 <bowlofeggs> nirik: i misunderstood the dates smooge was asking about - i can do oncall for this week, except for the weekend
14:22:45 <clime> I did some investigation of PostgREST...
14:22:52 <clime> and it looks good.
14:23:03 <clime> it's mainly really easy to setup
14:23:20 <clime> some summary is here: https://paste.fedoraproject.org/paste/9CtScjC8be3vmCv1CryFcA
14:24:08 <clime> the generated api is here: http://209.132.184.233:3000/
14:25:06 <clime> you can try out some queries http://209.132.184.233:3000/module_module?name=eq.shared-userspace
14:25:16 <clime> e.g.
14:25:29 <nirik> bowlofeggs: sure, if you want I can take the weekend...
14:26:07 <bowlofeggs> nirik: deal
14:28:02 <clime> the API is self-documented by using OpenAPI standard, which is nice. It enables features like remote joins or specific attribute selection.
14:28:17 <clime> it's quite similar to GraphQL actually.
14:29:03 <smooge> what kind of resources does it need?
14:29:24 <cverna> I need to look at it in more details, but I we were to use PostgREST what would be the development effort ? ie what do we need to do ?
14:29:58 <clime> smooge: i think not much, there was some mention about it in testimonials but i didn't do benchmarks
14:30:26 <clime> cverna: it would nice to have a python client for the api
14:30:52 <clime> i think there is no generic client in python for it atm
14:31:27 <cverna> clime: we still have to define how we store the data in postgres since we don't really want to do 100% what PDC was doing
14:31:41 <clime> cverna: yes, that too
14:31:44 <cverna> that's what is not really clear to me
14:32:12 <clime> right, but we should be able to just tweak/adjust the data model and that's it
14:32:31 <clime> no more coding (unless we want the python client as well)
14:33:20 <puiterwijk> Well, except we'd need to do all the coding in the data layer.
14:33:48 <cverna> ^^ yes because we have or will have some kind of logic
14:33:48 <puiterwijk> All business logic like required fields, mutually exclusive fields, valid values, etc, suddenly needs to be in SQL constraints
14:34:19 <puiterwijk> So there's still coding, just different kind of coding.
14:34:38 <clime> puiterwijk: ye, but definining those constraints should is just kind-of declarative coding that you when you create a db
14:35:02 <cverna> yes for example we currently have calculated fields like that https://github.com/fedora-infra/fpdc/blob/master/fpdc/releases/models.py#L18
14:35:15 <clime> the question is then if there is some functionality that would be not covered by the rest api and the contrainst themselves
14:35:32 <cverna> I am not sure how that would look in sql and how easy it would be to maintain
14:36:12 <puiterwijk> cverna: probably a custom view of the table, or a function, would be needed for that
14:36:16 <clime> cverna: well, such query could be done just over the rest api or you could make a store PostgREST procedure for it called 'status'
14:36:28 <bowlofeggs> we can just make a twisted nest of database triggers and stored procedures!
14:36:43 <clime> yes if you do a stored procedure, you can invoke it through /rpc/<proc_name>
14:36:59 <clime> bowlofeggs: well, i assume most of the logic can be done client-side
14:37:08 <clime> on fpdc you can have mostly just data
14:37:18 <clime> that's why the python client would be useful
14:37:28 <puiterwijk> clime: except that you want the server to validate that the data is correct...
14:37:35 <puiterwijk> client-side validity checking is incredibly weak
14:37:36 <bowlofeggs> i was just reliving horror days from early in my career when people liked triggers and stored procedures
14:37:43 <puiterwijk> bowlofeggs++
14:37:52 <clime> puiterwijk: well, yes, but those can be done just by the db constraints you mentioned
14:37:56 <puiterwijk> I believe that we shouldn't use triggers and stored procedures too extensively
14:38:03 <clime> we don't need to
14:38:04 <puiterwijk> clime: yes, but that requires SQL "coding".
14:38:07 <puiterwijk> Which is what I meant
14:38:24 <bowlofeggs> at my last job a disgruntled former employee had "encrypted" a stored procedure in our MSSQL database that nobody knew the key to (weird that you can do that?)
14:38:25 <clime> well those are just table/constraint definitions that you can find in any alembic migration
14:38:29 <puiterwijk> So it's still coding, you still need to tell the computer the same business logic, just in the database in SQL rather than python
14:38:42 <clime> right, but declarative coding mostly
14:38:47 <cverna> I don't feel really good with that part too, it will be more challenging for people to contribute too, where Django is pretty much well known
14:38:48 <bowlofeggs> puiterwijk: i believe we shouldn't use them at all ☺
14:39:02 <puiterwijk> bowlofeggs: we do use one trigger in all of Fedora Infra! :)
14:39:15 <puiterwijk> A trigger in FAS to automatically delete expired sessions, because the FAS code for it crashes :D
14:39:37 <bowlofeggs> haha
14:39:49 <bowlofeggs> well i suppose that's a fun reason to do it ☺
14:40:10 <puiterwijk> Actually, it might've been askbot because it didn't have any code for cleanups
14:40:14 <smooge> ok anyway.. clime can you write up a proposal on what you are wanting and how this might work. I think that we need more info that IRC can produce
14:40:43 <clime> smooge: i am not sure right now what more i can write.
14:41:00 <clime> I would need to investigate how fpdc is currently used
14:41:09 <clime> which i can do if needed
14:41:14 <cverna> it is not :P
14:41:24 <clime> alright, that was quick :D
14:41:35 <clime> so how it could be used
14:41:44 <clime> and that i don't know very well
14:41:46 <cverna> we currently onlyhave this https://fpdc.stg.fedoraproject.org/api/v1/
14:42:16 <clime> right
14:43:40 <nirik> well, it's up to the folks doing the work what tools to use... so they can mull this over and decide...
14:44:27 <smooge> ok clime thank you for the information then. Is there anything else on this topic people would like to talk about?
14:45:06 <smooge> #topic Tickets discussion
14:45:06 <smooge> #info https://pagure.io/fedora-infrastructure/report/Meetings%20ticket
14:45:24 <smooge> .ticket https://pagure.io/fedora-infrastructure/issue/7273
14:45:24 <zodbot> smooge: Error: 'https://pagure.io/fedora-infrastructure/issue/7273' is not a valid integer.
14:45:41 <smooge> .ticket 7273
14:45:43 <zodbot> smooge: Issue #7273: Dist-repo failed to generate due to missing signed RPMs - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/7273
14:46:17 <nirik> so there was some question from mizdebsk about how we should handle failures.
14:46:23 <nirik> anyone have opinions?
14:47:05 <mizdebsk> ideally it would be best to write missing signed copies before dist-repo is generated
14:47:31 <mizdebsk> but i'm not sure if the effort to write is worth fixing this corner case
14:47:35 <puiterwijk> Sure. Feel free to add it to tag2distrepo?
14:47:40 <puiterwijk> Should be simple enough
14:47:54 <puiterwijk> The infra repos are small enough that we could do this I think
14:48:20 <nirik> we have a lot of stuff in our tags... but I guess we can't know if we need to roll back to some old version
14:50:13 <smooge> my opinion is that the above is ok
14:52:15 <smooge> #topic Open FLoor
14:52:49 <bowlofeggs> i have an item
14:52:59 <smooge> ok your time with the conch
14:53:04 <bowlofeggs> the container registry seemed to get its storage into a weird state
14:53:13 <bowlofeggs> and isn't serving the f29 manifest, but does claim it has one
14:53:18 <bowlofeggs> which is kinda hilarious
14:53:26 <bowlofeggs> i don't have the expertise to fix it
14:53:29 <bowlofeggs> but i am willing to help
14:53:53 <bowlofeggs> is anyone available/interested in helping me with that? kevin suggested switching it from gluster to nfs
14:54:06 <bowlofeggs> which i'm on board with but don't know how to do
14:54:46 <nirik> I think puiterwijk was going to move it, but I don't know if he had the chance
14:54:52 <smooge> basically what needs to be done is a 'share' needs to be created on the netapp. That then needs to be mounted on the two systems.
14:54:53 <cverna> yes the prod registry is in a weird state, the pagination is not working too, maybe this is side effect of the storage issue
14:55:09 <bowlofeggs> it's basically a "mini outage" because it seems to serve the other containers, but not f29 (which affects bodhi CI and maybe other CI and also anyone who wants to test the f29 beta container)
14:55:17 <smooge> The storage then needs to be copied over to the share and the configs pointed to it.. The gluster then needs to be turned off
14:55:18 <nirik> cverna: it could be. Basically the two had different ideas of the storage
14:55:23 <puiterwijk> nirik: I was having trouble with finding my email with the netapp info
14:55:36 <puiterwijk> nirik: could you resend that?
14:55:43 <bowlofeggs> i'm not even sure we should copy existing data over to nfs
14:55:43 <nirik> I can make the netapp side... how big? or yeah, I could resend
14:55:48 <bowlofeggs> the existing data might be corrupt?
14:56:00 <bowlofeggs> might be better to make a fresh registry and republish to it?
14:56:02 <nirik> bowlofeggs: we were thinking of merging the two...
14:56:13 <bowlofeggs> oh, is there an easy way to do that?
14:56:18 <nirik> do we have all the data to populate it with?
14:56:24 <puiterwijk> nirik: if you can fix that, great.
14:56:29 <bowlofeggs> i presume we do, in koji maybe?
14:56:45 <bowlofeggs> i don't know how we'd merge the data, unless that's something gluster can do (without corrupting?)
14:56:47 <puiterwijk> bowlofeggs, nirik: we have the data in koji. But also, we should have all the objects and refs
14:57:02 <puiterwijk> bowlofeggs: well, the data has different names for different objects...
14:57:07 <bowlofeggs> oh ok
14:57:14 <puiterwijk> so just copy everything from node01, and then copy everything from node02 without overwrite
14:57:17 <bowlofeggs> so as long as they don't conflict that might work
14:57:20 <bowlofeggs> cool
14:57:22 <bowlofeggs> yeah worth a try
14:57:37 <puiterwijk> and if it doesn't work, we can always get the data back from koji... somehow :)
14:57:43 <bowlofeggs> yeah
14:57:44 <puiterwijk> Or just sync from staging registry
14:57:48 <puiterwijk> errr, candidate
14:57:59 <bowlofeggs> i mean, the publish script that releng uses might just put what is supposed to be there in place?
14:57:59 <puiterwijk> the not-quite-production-but-still-production registry:P
14:58:05 <bowlofeggs> haha yeah
14:58:24 <bowlofeggs> well is there any way i can be of assistance?
14:58:30 <bowlofeggs> or would i just be in the way?
14:58:52 <cverna> it would be good to get ride of the f25 and f26 containers too
14:58:53 <puiterwijk> I think you might have access being in releng
14:59:03 <cverna> we don't really want people to use these
14:59:19 <puiterwijk> cverna: or just redirect them to f27 and f28 :D
14:59:32 * puiterwijk is all for redirecting EOL containers to latest-stable :D
14:59:33 <smooge> bowlofeggs, does that answer your questions?
15:00:09 <bowlofeggs> puiterwijk: yeah i do have access to the registry, i just don't know much about how to get an NFS share made
15:00:17 <smooge> I would like to thank everyone for coming to the meeting today. I apologize for the mixup time in the email
15:00:19 <bowlofeggs> i think i could figure out how to mount one if it existed
15:00:30 <puiterwijk> bowlofeggs: ah. That me or Kevin would need to do. But after that, I bet you can help merging
15:00:30 <bowlofeggs> smooge: yeah i think we can move on
15:00:32 <cverna> puiterwijk: we could, but they will still be shown on html page ?
15:00:42 <bowlofeggs> puiterwijk: yeah i think i could try to figure out the merge
15:00:44 <puiterwijk> cverna: no, just a silent redirect of the manifest request!
15:01:05 <puiterwijk> "docker pull fedora:26" "Hey, I have fedora 28?"
15:01:09 <smooge> ok are we done?
15:01:14 <puiterwijk> smooge: yep, sorry
15:01:20 <smooge> #endmeeting