14:02:41 #startmeeting Infrastructure (2018-10-04) 14:02:41 Meeting started Thu Oct 4 14:02:41 2018 UTC. 14:02:41 This meeting is logged and archived in a public location. 14:02:41 The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:02:41 Useful Commands: #action #agreed #halp #info #idea #link #topic. 14:02:41 The meeting name has been set to 'infrastructure_(2018-10-04)' 14:02:41 #meetingname infrastructure 14:02:41 #topic aloha 14:02:41 The meeting name has been set to 'infrastructure' 14:02:41 #chair nirik pingou puiterwijk relrod smooge tflink threebean cverna 14:02:41 Current chairs: cverna nirik pingou puiterwijk relrod smooge tflink threebean 14:02:48 morning 14:02:52 good morning! 14:02:54 .hello2 14:02:54 bowlofeggs: bowlofeggs 'Randy Barlow' 14:02:59 morning :) 14:03:00 .hello2 14:03:13 sorry for the 2 minute delay... that 30 second email took a lot longer than I realized 14:03:18 bcotton: bcotton 'Ben Cotton' 14:03:40 #topic New folks introductions 14:03:40 #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves 14:03:43 hi 14:03:50 ó/ 14:03:53 Hello all 14:04:17 hel 14:04:18 lo 14:04:21 *hello 14:04:46 hello 14:04:54 Hello ! 14:04:55 My name is Christophe, I'm from France, Grenoble. 14:04:55 I work as a sysadmin for 3 years now and i'd like to contribute/help the fedora project. 14:04:55 I have experience of administering CentOS servers, bash and actually working on a SaaS infra project that includes Ansible, Gitlab ans Openshift (Docker/Kubernetes). 14:04:55 I can contribute 3/4 hours per week. My time zone is Europe/Paris. 14:04:55 IRC handle : chris787 14:04:55 FAS login : chris7878 14:04:56 Thanks. 14:05:17 hello 14:05:40 welcome chris787 14:05:43 .hello2 14:05:44 abompard: abompard 'Aurelien Bompard' 14:05:46 thks 14:05:51 .hello2 14:05:52 creaked: creaked 'Will Chellman' 14:05:54 Bonjour chris787 14:06:11 .hello2 14:06:13 odin2016: odin2016 'None' 14:06:16 Welcome chris787 14:06:23 slt 14:06:36 welcome chris787 14:06:43 Hi, i'm Chandra, i know Python, Linux (RHCSA), Flask, Django. 14:07:27 welcome chankuma 14:07:47 welcome chankuma :) 14:07:53 hello chankuma 14:07:57 hello chris787 14:08:04 hello everybody 14:08:17 If someone can point me to a suitable project (Python, Ansible, Jenkins), that would be hhelpful. 14:08:19 where do we start ? :-D 14:08:28 #topic announcements and information 14:08:28 #info tflink is on extended PTO 14:08:28 #info smooge is on PTO 2018-10-06->2018-10-09 14:08:28 #info nirik is on PTO 2018-10-24->2018-10-28 14:08:28 #info Final Freeze will start 2018-10-09 14:08:30 #info Updates/reboots done except for QA/builders 14:08:31 #info fedora-tagger/statscache retired - cverna/nirik 14:09:06 chris787, https://fedoraproject.org/wiki/Infrastructure/GettingStarted 14:09:54 this is nice crossroad for newcomers ^^ 14:10:08 Are there any other announcements for the work done this week or coming up next week? 14:10:49 I'm going to try to get pagure 5.1 before the freeze starts 14:10:53 in both pagure.io and src.fp.o 14:11:15 #info pagure 5.1 may roll out next week 14:11:34 pingou, so that is next tuesday 14:11:39 yup 14:11:43 hoping to do this on Monday 14:11:45 #info changes to src.fp.o inbound. look at various lists for more info 14:12:21 ok any other announcements? 14:12:34 #topic Oncall 14:12:34 #info Puiterwijk is on call from 2018-09-27->2018-10-04 14:12:34 #info ??? is on call from 2018-10-04 -> 2018-10-11 14:12:34 #info smooge is on call from 2018-10-11 -> 2018-10-18 14:12:34 #info ??? is on call from 2018-10-18 -> 2018-10-25 14:12:37 #info Summary of last week: (from Puiterwijk) 14:13:01 Not many pings, bunch of outages today due to reboot 14:13:22 I think you said you would take the next week also if no one else would take it? 14:13:37 Yeah. Though I believe bowlofeggs said he could take the week except for weekend? 14:13:57 But yeah, if not, I can. 14:14:18 i actually can't do that now :/ 14:14:41 okay. Then I will continue on... 14:14:43 I am going to ask nirik if he can take oncall to the 23rd and then I will take from the 23rd to the 1st 14:14:45 I will take 10-18 through 10-25 14:14:54 oh cool thanks relrod 14:14:54 * nirik arrives late, reads up 14:15:41 I can 14:17:08 hi nirik 14:17:43 #topic torrents and their troubles - kevin 14:17:43 #info used more than I realized 14:17:53 this is just a point from last weeks meeting. 14:18:07 yeah, I didn't have time to update the meeting gobby. ;( 14:18:10 I did the analysis that people asked for and found that torrents are used more than I realized 14:18:19 oh I kept this on for that reason 14:18:49 #info https://smooge.fedorapeople.org/stats-torrents/ 14:20:14 nice stats smooge. Thank you for making them. 14:20:16 it is hard to see how many download requests are really per day. They way the logs go is that if my torrent requests are over multiple hours, I get counted by the tracker as seperate events 14:20:41 For now, BTW, I have just kept the rtorrent seeder... 14:21:10 So the best average is actually the number of users versus clients per hour which is still reasonable 14:21:39 so I have changed from saying 'who will rid me of this troublesome software' to 'well its a pain but what are you going to do?' 14:21:50 anyway that was all I had on this 14:22:11 next topic 14:22:13 #topic Postgrest for fpdc evaluation: https://paste.fedoraproject.org/paste/9CtScjC8be3vmCv1CryFcA - clime 14:22:14 nirik: i misunderstood the dates smooge was asking about - i can do oncall for this week, except for the weekend 14:22:45 I did some investigation of PostgREST... 14:22:52 and it looks good. 14:23:03 it's mainly really easy to setup 14:23:20 some summary is here: https://paste.fedoraproject.org/paste/9CtScjC8be3vmCv1CryFcA 14:24:08 the generated api is here: http://209.132.184.233:3000/ 14:25:06 you can try out some queries http://209.132.184.233:3000/module_module?name=eq.shared-userspace 14:25:16 e.g. 14:25:29 bowlofeggs: sure, if you want I can take the weekend... 14:26:07 nirik: deal 14:28:02 the API is self-documented by using OpenAPI standard, which is nice. It enables features like remote joins or specific attribute selection. 14:28:17 it's quite similar to GraphQL actually. 14:29:03 what kind of resources does it need? 14:29:24 I need to look at it in more details, but I we were to use PostgREST what would be the development effort ? ie what do we need to do ? 14:29:58 smooge: i think not much, there was some mention about it in testimonials but i didn't do benchmarks 14:30:26 cverna: it would nice to have a python client for the api 14:30:52 i think there is no generic client in python for it atm 14:31:27 clime: we still have to define how we store the data in postgres since we don't really want to do 100% what PDC was doing 14:31:41 cverna: yes, that too 14:31:44 that's what is not really clear to me 14:32:12 right, but we should be able to just tweak/adjust the data model and that's it 14:32:31 no more coding (unless we want the python client as well) 14:33:20 Well, except we'd need to do all the coding in the data layer. 14:33:48 ^^ yes because we have or will have some kind of logic 14:33:48 All business logic like required fields, mutually exclusive fields, valid values, etc, suddenly needs to be in SQL constraints 14:34:19 So there's still coding, just different kind of coding. 14:34:38 puiterwijk: ye, but definining those constraints should is just kind-of declarative coding that you when you create a db 14:35:02 yes for example we currently have calculated fields like that https://github.com/fedora-infra/fpdc/blob/master/fpdc/releases/models.py#L18 14:35:15 the question is then if there is some functionality that would be not covered by the rest api and the contrainst themselves 14:35:32 I am not sure how that would look in sql and how easy it would be to maintain 14:36:12 cverna: probably a custom view of the table, or a function, would be needed for that 14:36:16 cverna: well, such query could be done just over the rest api or you could make a store PostgREST procedure for it called 'status' 14:36:28 we can just make a twisted nest of database triggers and stored procedures! 14:36:43 yes if you do a stored procedure, you can invoke it through /rpc/ 14:36:59 bowlofeggs: well, i assume most of the logic can be done client-side 14:37:08 on fpdc you can have mostly just data 14:37:18 that's why the python client would be useful 14:37:28 clime: except that you want the server to validate that the data is correct... 14:37:35 client-side validity checking is incredibly weak 14:37:36 i was just reliving horror days from early in my career when people liked triggers and stored procedures 14:37:43 bowlofeggs++ 14:37:52 puiterwijk: well, yes, but those can be done just by the db constraints you mentioned 14:37:56 I believe that we shouldn't use triggers and stored procedures too extensively 14:38:03 we don't need to 14:38:04 clime: yes, but that requires SQL "coding". 14:38:07 Which is what I meant 14:38:24 at my last job a disgruntled former employee had "encrypted" a stored procedure in our MSSQL database that nobody knew the key to (weird that you can do that?) 14:38:25 well those are just table/constraint definitions that you can find in any alembic migration 14:38:29 So it's still coding, you still need to tell the computer the same business logic, just in the database in SQL rather than python 14:38:42 right, but declarative coding mostly 14:38:47 I don't feel really good with that part too, it will be more challenging for people to contribute too, where Django is pretty much well known 14:38:48 puiterwijk: i believe we shouldn't use them at all ☺ 14:39:02 bowlofeggs: we do use one trigger in all of Fedora Infra! :) 14:39:15 A trigger in FAS to automatically delete expired sessions, because the FAS code for it crashes :D 14:39:37 haha 14:39:49 well i suppose that's a fun reason to do it ☺ 14:40:10 Actually, it might've been askbot because it didn't have any code for cleanups 14:40:14 ok anyway.. clime can you write up a proposal on what you are wanting and how this might work. I think that we need more info that IRC can produce 14:40:43 smooge: i am not sure right now what more i can write. 14:41:00 I would need to investigate how fpdc is currently used 14:41:09 which i can do if needed 14:41:14 it is not :P 14:41:24 alright, that was quick :D 14:41:35 so how it could be used 14:41:44 and that i don't know very well 14:41:46 we currently onlyhave this https://fpdc.stg.fedoraproject.org/api/v1/ 14:42:16 right 14:43:40 well, it's up to the folks doing the work what tools to use... so they can mull this over and decide... 14:44:27 ok clime thank you for the information then. Is there anything else on this topic people would like to talk about? 14:45:06 #topic Tickets discussion 14:45:06 #info https://pagure.io/fedora-infrastructure/report/Meetings%20ticket 14:45:24 .ticket https://pagure.io/fedora-infrastructure/issue/7273 14:45:24 smooge: Error: 'https://pagure.io/fedora-infrastructure/issue/7273' is not a valid integer. 14:45:41 .ticket 7273 14:45:43 smooge: Issue #7273: Dist-repo failed to generate due to missing signed RPMs - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/7273 14:46:17 so there was some question from mizdebsk about how we should handle failures. 14:46:23 anyone have opinions? 14:47:05 ideally it would be best to write missing signed copies before dist-repo is generated 14:47:31 but i'm not sure if the effort to write is worth fixing this corner case 14:47:35 Sure. Feel free to add it to tag2distrepo? 14:47:40 Should be simple enough 14:47:54 The infra repos are small enough that we could do this I think 14:48:20 we have a lot of stuff in our tags... but I guess we can't know if we need to roll back to some old version 14:50:13 my opinion is that the above is ok 14:52:15 #topic Open FLoor 14:52:49 i have an item 14:52:59 ok your time with the conch 14:53:04 the container registry seemed to get its storage into a weird state 14:53:13 and isn't serving the f29 manifest, but does claim it has one 14:53:18 which is kinda hilarious 14:53:26 i don't have the expertise to fix it 14:53:29 but i am willing to help 14:53:53 is anyone available/interested in helping me with that? kevin suggested switching it from gluster to nfs 14:54:06 which i'm on board with but don't know how to do 14:54:46 I think puiterwijk was going to move it, but I don't know if he had the chance 14:54:52 basically what needs to be done is a 'share' needs to be created on the netapp. That then needs to be mounted on the two systems. 14:54:53 yes the prod registry is in a weird state, the pagination is not working too, maybe this is side effect of the storage issue 14:55:09 it's basically a "mini outage" because it seems to serve the other containers, but not f29 (which affects bodhi CI and maybe other CI and also anyone who wants to test the f29 beta container) 14:55:17 The storage then needs to be copied over to the share and the configs pointed to it.. The gluster then needs to be turned off 14:55:18 cverna: it could be. Basically the two had different ideas of the storage 14:55:23 nirik: I was having trouble with finding my email with the netapp info 14:55:36 nirik: could you resend that? 14:55:43 i'm not even sure we should copy existing data over to nfs 14:55:43 I can make the netapp side... how big? or yeah, I could resend 14:55:48 the existing data might be corrupt? 14:56:00 might be better to make a fresh registry and republish to it? 14:56:02 bowlofeggs: we were thinking of merging the two... 14:56:13 oh, is there an easy way to do that? 14:56:18 do we have all the data to populate it with? 14:56:24 nirik: if you can fix that, great. 14:56:29 i presume we do, in koji maybe? 14:56:45 i don't know how we'd merge the data, unless that's something gluster can do (without corrupting?) 14:56:47 bowlofeggs, nirik: we have the data in koji. But also, we should have all the objects and refs 14:57:02 bowlofeggs: well, the data has different names for different objects... 14:57:07 oh ok 14:57:14 so just copy everything from node01, and then copy everything from node02 without overwrite 14:57:17 so as long as they don't conflict that might work 14:57:20 cool 14:57:22 yeah worth a try 14:57:37 and if it doesn't work, we can always get the data back from koji... somehow :) 14:57:43 yeah 14:57:44 Or just sync from staging registry 14:57:48 errr, candidate 14:57:59 i mean, the publish script that releng uses might just put what is supposed to be there in place? 14:57:59 the not-quite-production-but-still-production registry:P 14:58:05 haha yeah 14:58:24 well is there any way i can be of assistance? 14:58:30 or would i just be in the way? 14:58:52 it would be good to get ride of the f25 and f26 containers too 14:58:53 I think you might have access being in releng 14:59:03 we don't really want people to use these 14:59:19 cverna: or just redirect them to f27 and f28 :D 14:59:32 * puiterwijk is all for redirecting EOL containers to latest-stable :D 14:59:33 bowlofeggs, does that answer your questions? 15:00:09 puiterwijk: yeah i do have access to the registry, i just don't know much about how to get an NFS share made 15:00:17 I would like to thank everyone for coming to the meeting today. I apologize for the mixup time in the email 15:00:19 i think i could figure out how to mount one if it existed 15:00:30 bowlofeggs: ah. That me or Kevin would need to do. But after that, I bet you can help merging 15:00:30 smooge: yeah i think we can move on 15:00:32 puiterwijk: we could, but they will still be shown on html page ? 15:00:42 puiterwijk: yeah i think i could try to figure out the merge 15:00:44 cverna: no, just a silent redirect of the manifest request! 15:01:05 "docker pull fedora:26" "Hey, I have fedora 28?" 15:01:09 ok are we done? 15:01:14 smooge: yep, sorry 15:01:20 #endmeeting