18:00:19 #startmeeting Infrastructure (2015-09-24) 18:00:19 Meeting started Thu Sep 24 18:00:19 2015 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:19 Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:19 #meetingname infrastructure 18:00:19 #topic aloha 18:00:19 #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk pbrobinson 18:00:19 The meeting name has been set to 'infrastructure' 18:00:19 Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pbrobinson pingou puiterwijk relrod smooge threebean 18:00:20 #topic New folks introductions / Apprentice feedback 18:02:15 any new folks like to give a short one line introduction? or apprentices with questions or comments? 18:02:46 * threebean is here 18:02:52 * aikidouke is here 18:02:55 hi 18:03:09 * danofsatx is here, but no longer in the program :( 18:03:14 * pcreech|work is here 18:04:24 ok, if no intros or apprentice questions, lets move on to status / info: 18:04:27 #topic announcements and information 18:04:27 #info Fedora 23 Beta is out the door smoothly - everyone 18:04:27 #info Next freeze (final) coming up 2015-10-13 - kevin 18:04:27 #info new release of the-new-hotness went out this morning (with some hiccups) - ralph 18:04:28 #link https://lists.fedoraproject.org/archives/list/infrastructure%40lists.fedoraproject.org/thread/TXYVETKPKSD7DZER6DLJYIGGJKWFIVPF/ 18:04:29 #info our infrastructure git repos (ansible, puppet, dns, etc..) now publish fedmsg messages about commits - relrod/ralph 18:04:32 #link https://apps.fedoraproject.org/datagrepper/raw?category=infragit&delta=127800 18:05:02 I'd like to add a item: I have a outage scheduled for tomorrow at 16:00UTC to migrate lockbox01 to batcave01. 18:05:26 I'll be swapping IP addresses, so ssh will tell you it's changed and you will need to update your known hosts. 18:05:53 and I am sure I will have missed some things, because lockbox01 in puppet was many layers of stuff (Althought I tried to get it all) 18:06:06 * roshi is here as well 18:06:07 but we can work through them after the migration. 18:06:16 Note that /home will be synced. 18:06:44 any other announcements or status? 18:06:59 * pingou late 18:07:02 ok, moving along. 18:07:05 #topic Grant admin permission in staging Koji to mizdebsk 18:07:05 .ticket 4900 18:07:07 nirik: #4900 (Grant admin permission in staging Koji to mizdebsk) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/4900 18:07:20 koschei testing in staging has been quite painful, mostly due to staging koji not working (which is fixed now, thanks to nirik and threebean) and due to lack admin permission to simulate corner cases 18:07:23 mizdebsk: you here? thanks for listing things clearly there, that helps. 18:07:26 i asked for admin permission on releng trac about half year ago, but there hasn't been any progress, so i'm bringing this issue here in hope you'll help with resolving it 18:07:35 are there any reasons why i wouldn't be granted admin permission in *staging* koji only? 18:07:39 * lnxslck is here 18:07:57 well, it wasn't too clear what you needed it for, I think last time we talked you said you needed to import src.rpms, which we don't want to do... 18:08:06 but I think this ticket is more clear 18:08:23 this has been fixed now - i can build from staging dist-git, so no need to import srpms any longer 18:08:33 I personally don't have any objections. dgilmore: can you look at the ticket and let us know what you think? 18:08:36 yeah 18:09:15 stg koji is much more usable than it used to be 18:09:33 mostly due to threebean''s work on the prod->stg script 18:09:55 yes, definitely 18:10:15 I''ve not yet done image builds on it, but we should make sure that works too 18:10:48 i will revert every change i make for purposes of testing koschei, but we have the playbook to resync stagig koji from prod in case something would go terribly wrong 18:10:58 my two cents: I think its reasonable to give out staging koji admin rights more broadly 18:11:23 mizdebsk: idle thought: could koschei be extended to images? ie, if you know all packages that go into an image or are needed to make it, you could rebuild it anytime those things change? 18:11:26 it doesn't apply in this case, but it would be nice to be a ground where we could develop new koji plugins (I know maxamillion was looking at this, and imcleod wanted to try stuff out too) 18:11:42 I guess thats a sidetrack sorry 18:12:06 nirik: i don't know, haven't thought about images 18:12:40 threebean: yeah. 18:12:54 I want to try and test new mock in stg before we roll to prod. 18:13:03 and rpm and dnf, etc 18:13:50 test critpath in stg before we roll to prod? :) 18:14:16 sure. Mostly for things like changes to the builders. 18:14:23 +1 18:14:25 when we move them to f23 we should test that in stg too 18:15:13 I would rather we dont give admin to stg koji in order to sidstep doing things that are supposed to be doen elsewhere 18:15:24 dgilmore: as in? 18:15:27 done where? 18:15:40 so for koschei we should let mizdebsk be able to approve packages etc in pkgdb 18:15:57 that way we test the full suite of interrelated apps 18:16:08 sure, but he may need admin for the other items listed in the ticket. 18:16:24 like canceling stuck jobs or changing something to see how it breaks kschei 18:16:34 what about blocking? am i supposed to wait half a day to test a code path? blocking directly is much simpler 18:17:14 additionally, any changes made in stg will be completely wiped out when the next sync happens. 18:18:10 being able to run repo-regen is also important 18:19:03 you do not need admin to cancel stuck jobs 18:19:28 well, admin or the koschei cert. 18:19:49 i can simulate regen-repo with untag/tag on any random build, but then kojira generates both f24-build and rawhide, which is slow - only one host in createrepo channel 18:20:12 anyway, if there is valid reasons outside of adding and removing packages then we can look at it 18:20:15 yeah, I can add more, thats due to them not havig rw /mnt/koji 18:20:31 I would just rathe we try to follow production process 18:21:07 if we can't test things in stg what is it really good for? Do we need a thrid environment, test ? 18:21:09 sure, thats why we have been fixing up stg to behave like prod 18:21:39 pingou: I am not sure I follow you there 18:21:42 i'll be following prod process whenever possible, but in many cases it is very impractical or impossible and would mean not testing certain features or code paths at all 18:22:03 dgilmore: I don't quite see the harm in letting people do changes in stg to test other application and giving them the rights to do so 18:22:32 pingou: still not following you 18:22:33 we're already having troubles to replicate prod in stg and if we can't make stg such that it can be use for integration testing 18:23:00 then I wonder what we need to do to have an environment we can test in 18:23:52 pingou: if we can make mizdebsk have admin rights in pkgdb he can add and remove packages as needed 18:24:02 in stg that's an easyfix 18:24:14 it hads the benefit of testing the full suite of interconnected apps 18:24:24 if it's all what mizdebsk needs it'll be done in a few minutes :) 18:24:29 rathe than just adding the package to koji and building 18:24:45 but the problem with treating stg as just another happily functioning prod is that you then ignore the corner cases that could hit prod 18:24:49 thats the kind of thing I am trying to avoid 18:24:50 pingou: no, that definitely doesn't satisfy my needs for thesting koschei 18:25:11 mizdebsk: that was also my impression :) 18:25:17 so IMHO it's valid and valuable to be able to introduce errors in stg and confirm dependent apps still handle things 18:25:27 my impression was that is all thats needed 18:25:36 so apparently I have missed something 18:25:36 especially when you can just wipe and sync to prod anytime to remove whatever you did 18:26:34 https://fedorahosted.org/fedora-infrastructure/ticket/4900 has a list 18:27:27 I will have to read the ticket later 18:27:34 but I have not yet seen it before now 18:28:20 well, it would be nice to approve this, since he's been waiting a long time... 18:28:24 dgilmore: out of curiosity, what makes you relunctant in giving access to koji in stg? 18:28:40 IMHO we should also be free to grant it to the people working on plugins. 18:28:51 is there a security concern or an aspect we're not thinking about? 18:30:02 pingou: I don't want things to be done in a way where there becomes a reliance on it in prod 18:30:39 I don't see how that would be possible here... 18:30:49 * pingou has an english failure there 18:31:35 since mizdebsk won't be admin in prod no changes would be made there... and if he made changes in stg that koschei depended on it would break when the next prod->stg syc happened or when that version went to prod 18:31:38 we rely on something to be working in a specific way in prod because we adjust stg to behave that way? 18:32:40 and that would be koschei's problem (thus mizdebsk) koji wouldn't be impacted 18:34:04 I am failing to grok what you are trying to say pingou 18:35:07 the 'we rely on something' was me trying to rephrase what you said to see if I understood it correctly :) 18:35:21 the 'and that would be...' was a follow up on nirik's sentence 18:36:02 * nirik is not seeing the big deal here. If it was prod sure, but this is just stg, it's specifically for testing applications before they go to prod 18:36:11 I have two main concerns, 1) is that I do not feel that admin should ever be needed for koschei to do its thing, 2) that we will skip the testing of the other pieces in the packaging puzzle because it is easier to do 18:36:51 for 1) iiuc it's not for koschei to do things it's for mizdebsk to test koschei properly 18:37:09 for 2) I'd say that this is mizdebsk's problem :) 18:37:09 1) agreed in prod, but not in in stg, in stg we want to be able to change things to break koschei so it's properly tested. 18:37:47 ad 1) koschei user in koji will not have admin rights, only me - koschei admin 18:38:18 ad 2) i won't be waiting half a day to test package blocking, so this wouldn't be tested anyways 18:39:12 put another way, mizdebsk needs to be able to break koji. 18:39:42 since there are conflicting opinions, shall we vote on it? 18:40:00 yes, sort of - temporarily to test how koschei reacts to corner cases; and then revert changes asap (un-break it) 18:40:03 I am going to step aside because it seems no one cares what my concerns are 18:40:34 I do... I just don't understand them. ;( 18:40:45 * lmacken doesn't understand dgilmore's #2 18:41:00 lmacken: testing pkgdb, etc 18:41:10 I kinda get that one... 18:41:17 we have rube for testing as many apps in stg as possible, and the pkgdb is in there. 18:41:26 if we block a package in pkgdb.stg it might somehow behave differently from a manual koji admin block 18:41:56 and then if we test via the manual thing it break in prod. 18:42:13 blocking doesn't happen immediately - only after very long delay - at which point i'll be sleeping and not able to observe koschei behaviour 18:42:13 that shouldn't happen 18:42:18 I understand that danger, but I also think it's still worthwhile to test it other ways. 18:43:24 some cases are not possible to simulate with pkgdb, like removing package (not blocking) 18:43:43 I would like for us to do more work on dist-git and make it more useful 18:43:44 mizdebsk: pkgdb can remove packages 18:43:48 also, it should be more on mizdebsk... if he tests in a way that doesn't reflect how it's done in prod and causes a bug to get out hes the one that has to fix it and know to improve the way it's tested. 18:43:51 just have no idea of its effect in koji 18:44:09 pingou: i meant removing package from koji tag listing, not from pkgdb 18:44:24 k 18:44:29 mizdebsk: we remove a package by blocking 18:44:47 that is how it is done 18:45:11 we do not delete a package from existance 18:45:13 usually, but not always; sometimes package gets removed and this happened in the last month 18:45:29 there have been cases due to legal 18:45:44 which afaik has only happened once 18:45:44 this happens eg. when stg koji is sysced with prod; if you added pkg to staging only it would be gone 18:45:58 mizdebsk: we do not do that in prod though 18:46:10 except is legal says it need stobe gone 18:46:18 which we did once years ago 18:46:21 yeah, I can only recall that once.... 18:46:24 and i would like to test that case too 18:46:46 mizdebsk: to remove it like that you have to go into the db 18:46:54 being a koji admin it is not possible 18:47:13 for my purposes it's enough to remove from tag listing, which can be done from xml-rpc 18:47:21 you need to manually crate a delete query 18:49:50 ok, so we have been on this a while, not sure what we want to do. dgilmore: how about we provisionally add him and revisit next week when we can look at koji.stg history? 18:50:17 then if there's things we have more direct concerns about we can bring them up then? 18:52:13 ehh 18:53:09 well, that wasn't a no. ;) 18:53:13 anyhow we are running out of time... 18:53:14 ^^ 18:53:18 #topic Open Floor 18:53:23 any items for open floor? 18:54:02 not from me 18:54:10 dinner's ready :) 18:55:01 * nirik had something, but now cannot recall what it was. 18:55:37 batcave01? 18:56:05 I had a question I was going to ask nirik, but might as well do it here 18:56:18 pingou: already mentioned early in the meeting. ;) 18:56:23 kalev: sue 18:56:24 sure 18:56:31 my question was about https://fedorahosted.org/fedora-infrastructure/ticket/4857 18:56:58 does anyone have ideas how to make private builds happen in koji, where the binaries produced wouldn't be downloadable for general public? 18:57:08 well, lets take this discussion off line 18:57:11 ok 18:57:14 this does not need to be in the infra meeting. 18:57:43 kalev: thats a massive massive undertaking 18:58:11 kalev: you would be best off talking to mikem and mikeb but the answer is it is not simple 18:58:23 but it is off topic for this meeting 18:58:51 sure, I can definitely talk to them 18:58:54 thanks dgilmore 18:59:01 * nirik was going to test a theory about this on stg koji... but hasn't had a chance yet 18:59:09 anyhow, if nothing else will close out in a minute. 19:00:03 thanks for coming everyone! 19:00:06 #endmeeting