15:39:13 #startmeeting RELENG (2014-06-30) 15:39:13 Meeting started Mon Jun 30 15:39:13 2014 UTC. The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:39:13 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:39:18 #meetingname releng 15:39:18 The meeting name has been set to 'releng' 15:39:18 #chair dgilmore nirik tyll sharkcz bochecha masta 15:39:18 Current chairs: bochecha dgilmore masta nirik sharkcz tyll 15:39:19 #topic init process 15:39:25 who is here? 15:39:33 hey. 15:39:39 Hi 15:39:39 * sharkcz waves 15:40:09 #topic #5931 [Proposal] Move new branch and unretire requests to pkgdb2 15:40:15 https://fedorahosted.org/rel-eng/ticket/5931 15:40:41 so apparently tyll_ suggested this to pingu 15:41:10 it will take a massive amount of work and complete changes of many processes to do 15:41:20 hi 15:42:01 I believe it was planned since the beginning of pkgdb1 15:42:19 at least there is a note on the SCM change request page 15:42:41 tyll_: i think it will make some aspects of it much much better 15:42:51 tyll_: and if it was tied into a review app even better 15:43:12 but it will take redoing how we do it all 15:43:22 at the end of, there is an admontion: http://fedoraproject.org/wiki/Package_SCM_admin_requests#Package_Change_Requests_for_existing_packages 15:43:46 tyll_: right, its been there for years 15:43:53 and we have never done anything about it 15:44:05 welcome pingou 15:44:17 is there so much that needs to be changed? I assumed it is only the script that processes branch requests 15:44:22 I think the experience for the users and admins would be much better 15:44:39 and it would give just one place to go see what needs doing 15:45:31 tyll_: well the current scripts are run from your local machine, they use auth to bugzilla, pkgdb, ssh to pkgs01, and some koji auth 15:46:19 tyll_: unblocking requires a releng ticket, adding branches and packages needs a bugzilla request 15:47:03 we would either need to setup some proxy processes in pkgdb2 or get accounts in the different places for it to work 15:47:37 my idea was that pkgdb2 would only replace the tickets 15:47:42 pingou: how would you see this all being glued together in pkgdb2? 15:48:00 pingou: tickets are only used for unblocking unretired packages 15:48:06 so we would keep the current script, they would just start by querying pkgdb2 to get the list of pending actions 15:48:21 dgilmore: there also used for new branch, in bugzilla :) 15:48:25 pingou: there is no script for unblocking 15:48:50 * pingou does not know enough of these processes 15:49:09 pingou: adding branches and new packages requires some flags to be set in bugzilla 15:49:21 note: pkgdb2 already interact with bugzilla 15:49:29 IMHO a good first step would be to only move the bugzilla SCM admind requests to pkgdb 15:49:33 pingou: which the script gets the list of packages from bugzilla when it does a run 15:49:50 tyll_: thats not at all what pingou is looking at doing 15:50:01 ? 15:50:47 pingou: it sounded to me like you just wanted to query bugzilla to get the lis 15:50:51 list 15:50:53 we do that already 15:51:04 actually since rel-eng is processing the SCM admin requests, a second ticket to request package unblocking might not be needed anymore, because one either needs both unretiring and unblocking or nothing 15:51:52 dgilmore: I am not sure I follow you 15:52:09 tyll_: we would have to change the process when unretiring but yeah the unblocking could get thrown in there 15:52:41 pingou: i guess I do not get how you plan to know there is a new cvs admin request 15:53:08 dgilmore: my idea was that to request a new branch, instead of going on bugzilla one would go on pkgdb 15:53:45 pingou: ok and how exactly would that work? 15:53:52 we may before too long have bugzilla fedmsgs. ;) 15:54:23 dgilmore: the user logs in pkgdb, and click on the button "Request a new branch", that gets adding to the list of actions requiring an admin, available at: http://209.132.184.188/api/admin/actions/ 15:54:51 pingou: how does that work for new packages that do not yet exist in pkgdb? 15:54:54 we can also send an email to cvsadmin about this new request 15:55:15 dgilmore: I was thinking new branch of existing packages 15:55:26 pingou: thats really not done that often 15:55:33 ok 15:55:37 pingou: to me is an all or nothing thing 15:55:55 it would be too confusing to use one way for one thing and other for the other 15:56:06 people out of habit would use the old way for the other 15:56:13 but for new pkgs it could be the same, login to pkgdb and request a new package 15:56:27 so bugzilla = review approved -> package automatically created on pkgdb -> user goes there to ask for branches? 15:56:38 tyll_: you would need to provide a bug for the review 15:57:09 pingou: you would need to setup something to ocntantly poll bugzilla 15:57:14 but you could do that 15:57:37 dgilmore: or we wait for bugzilla2fedmsg (hoping it provides us enough info for this kind of action) 15:57:55 pingou: part of the process that the admin does is to verify that the package review is done properly and by someone in the packager group 15:58:29 pingou: so I really would rather that the page just get added to pkgdb 15:58:47 we could check who set the flag to '+' and check it was a packager 15:58:54 but instead sit in a state where it can be reviewed and only added when acked by an admin 15:59:08 pingou: thats not how it works 15:59:19 it might be that only packagers can actually set the review flag 15:59:24 pingou: the review flag has to be + but the cvsadmin flag ? 15:59:37 and thats not enough to tell us the review is done right 15:59:46 that can only be done by examination of the bug 15:59:52 dgilmore: there would be no cvsadmin flag anymore 15:59:54 dgilmore: but doesn't cvsadmin flag disapear if it's moved to pkgdb? 16:00:02 tyll_: bugzilla doesnt know anything about fas groups 16:00:04 (agreed wrt to the review) 16:00:20 pingou: perhapsd 16:00:23 perhaps 16:01:01 what i think would be best, and we can work on something to get to that place is 16:01:17 we have a review app, which i vaguely remeber somoene was working on 16:01:55 that'd be I :] 16:01:56 anyone in 'fedora bugs' can set the flags. 16:01:59 when the package gets marked as reviewed it then calls out and setsup everything for review 16:02:20 dgilmore: I agree bugzilla doesn't know about FAS, but we do, so when seeing (on fedmsg) that someone updated the review flag to '+', we can see who did it (packager?) and create the package setup everything for review 16:02:56 the cvsadmin checks everything off and approves it in review app or pkgdb and then that tool actually goes off and does all the steps to add the package etc 16:03:25 pingou: we have ways to automate that yes 16:03:42 pingou: we should also verify that teh submitter is a approved packager also 16:03:54 if not flag them to get a sponsor 16:04:49 so upon review set has '+' -> automatically set cvsadmin flag to '?' -> cvsadmin member double checks the review and set the cvsadmin flag to '+' 16:05:11 upon cvsadmin flag set to '+' -> pkgdb create the package => user goes to pkgdb to ask for the branches 16:05:39 cvsadmin (or cron job?), creates new branches 16:05:53 well, the downside there is that anyone can set the flag in bz. 16:06:05 what do we need to cvsadmin flag for? 16:06:18 nirik: but we know the email of the user that does the action, so we can check if that person is allowed to do it 16:06:22 probibly should keep state in pkgdb or something. 16:06:42 the reviewer could just set the review flag to + and the SCM admin could post a comment that it is added to pkgdb 16:06:48 review + -> add to scm admin queue -> when admin says 'ok', process it? 16:06:50 pingou: i think it would ba a state in pkgdb or the review app 16:07:00 nirik: that's the idea (iirc) 16:07:12 yeah, we could drop the cvsadmin flag entirely. 16:07:15 s/iirc/iiuc/ 16:07:20 although they need to tell us what branches I guess. 16:07:39 other option 16:07:39 once the package is added to pkgdb, we need to create the git repo, then any additional branches 16:08:10 we alrady have a process that syncs from pkgdb to koji package owners 16:08:25 we could maybe try teach it to deal with unblocking 16:08:29 reivew set to '+', user goes to pkgdb -> request new package w/ its branches, provide link to bugzilla review -> cvsadmin check the package review and create the package/branches 16:08:39 but the sync script is pretty basic 16:08:56 pingou: could be step 1 16:09:29 pingou: in the end the goal of automation was the admin signs off and everything else just happens 16:09:46 we just never got there 16:10:38 dgilmore: if we do that in pkgdb2, package/branch creation will be broadcasted on fedmsg and we could have a listener that just create the git/branches automatically 16:10:50 abadger1999: I liked the other one better :D 16:11:01 * nirik nods. there's lots of places we could make scripts fedmsg aware 16:11:09 :-) 16:11:09 pingou: yes, but we also need to account for the fact that fedmsg is not guaranteed to be reliable 16:11:43 dgilmore: that is true 16:11:45 trust, but verify. ;) 16:13:09 nirik: I guess missing messages are more problematic, since messages are already signed afaik 16:13:35 pingou: we could also rewrite how we sync packages to add them when added in real time rather than the 10 minute cron job 16:13:45 right. I meant we should assume it's reliable, but also have some script/way to check that it got all the messages or that nothing was missed. 16:13:54 periodically check the api endpoint should help cover that tyll_ I think 16:14:49 dgilmore: I don't know the process enough to follow you here 16:14:51 so i guess the summary right now is that we should work towards this, but there is a lot of implementation details to work out 16:15:10 pingou: we can go over it some time 16:15:17 sounds like a summary :) 16:15:22 dgilmore: sure, thanks! :) 16:15:31 #info 8:14 < dgilmore> so i guess the summary right now is that we should work towards this, but there is a lot of implementation details to work out 16:16:04 #info we should work towards more integrated and better automation, there is a lot of details to work out, and it will likely be done in phases 16:16:53 * pingou will add the 'new package request' on pkgdb2 16:17:09 lets hash out how we want it all to work first? 16:17:22 nirik: yeah 16:17:53 pingou: before doing that, lets map out the processes and work out how we would like it all to work 16:18:00 sure 16:18:17 don't want to waste your time and effort 16:18:27 looks like I'll have to subscribe to the releng list :) 16:18:49 pingou: :) it gets less mail than a couple of weeks ago 16:19:01 anything else here? 16:19:16 * pingou subscribed 16:19:23 im wondering if we shouldnt skip the other two tickets, since this took up so much time 16:19:42 and get on with the secondary arch status's 16:19:48 there is not much new there except that autosign01 is now RHEL7 16:20:10 tyll_: that was kinda my thoughts 16:20:35 * nirik nods. 16:20:41 #topic Secondary Architectures updates 16:20:42 #topic Secondary Architectures update - ppc 16:20:47 masta: you're up 16:21:48 gfuess he disapeared 16:22:00 not sure where things were last week 16:22:11 but be and le have merged the hubs 16:22:26 mass rebuild is in progress, 15k build done,. 500+ failures 16:22:30 nightly composes push out ppc64 and ppc64le trees 16:22:47 sharkcz: about where primary is then 16:24:02 ~1200 are broken deps, so not so far 16:24:09 okay 16:24:13 oh... one thing I noticed... 16:24:28 ppc hub is 95% full on / 16:24:40 probably logs 16:24:58 nirik: it's s390 hub, ppc is 82% 1.7T free 16:25:01 sorry, 98% 16:25:05 #action masta to look at teh hub and manage space 16:25:13 94G 87G 2.1G 98% / 16:25:24 ah, the / 16:25:24 sharkcz: / not /mnt/koji 16:25:29 those nagios emails have been going to me and dwa. 16:25:45 should I send them to sysadmin-releng? sysadmin? specific people? 16:25:52 it will be rpms from shadow 16:25:59 nirik: we should s/dwa/masta/ likely there will be a new ppc person very soon 16:26:12 ok, happy to adjust it so it's not just me. ;) 16:26:42 (since I often don't know what to remove) 16:26:53 lets do masta for now 16:27:01 * nirik nods 16:27:11 #topic Secondary Architectures update - s390 16:27:17 sharkcz: floor is yours 16:27:50 s390 is fighting with gcc 4.9 causing build failures in gnutls, texlive, ... 16:28:13 so there is progress, but it is slow 16:28:45 I have people working on reducing the code so bug reports could be filled 16:30:33 okay 16:30:41 anything we can help with? 16:30:58 probably not 16:31:52 nirik: I've freed some space under /tmp on the ppc hub, it's 78% full now 16:31:53 okay thanks 16:32:00 sharkcz: thanks 16:32:06 #topic Secondary Architectures update - arm 16:32:16 nirik: np, shadow would crash soon 16:32:37 so thanks for noticing :-) 16:32:48 statistics: {'older': 358, 'local_only': 0, 'remote_only': 477, 'same': 14599, 'newer': 1, 'total_missing_builds': 551} 16:33:04 very nice numbers 16:33:06 so aarch64 is really close to primary for rawhide 16:34:34 its moving along well 16:34:41 cool. 16:35:00 there is two issues i know of that will need kernel patches to have f21 work on hardware 16:35:50 we should be good to go for alpha 16:36:03 #topic Open Floor 16:36:06 okay 16:36:09 one issue 16:36:23 branching is very very soon 16:37:02 a week from tomorrow 16:37:02 im planning to spend some time this week on trying to get a compose done and see what can be done to optimise things 16:37:20 since we will need to start TC composes next week 16:39:28 anyone have anything else for open floor? 16:39:53 I wanted to propose to get rid of cvs in FAS groups 16:40:02 I mentioned it on the channel once 16:40:25 tyll_: changing groups in fas is a mess and not really possible 16:40:40 we can create new ones and not use the old ones 16:41:01 we can, but would need to dig into all the places that use the cvs groups for things 16:41:32 there is not much stuff in ansible or puppet 16:41:48 I believe only the gitolite script and the pkgdb2 config 16:41:58 if we do go down that route we should use scmadmin sysadmin-scm 16:42:14 tyll_: there is more than that 16:42:20 yes, at least something generic 16:42:34 at the least there is the access to pkgs01 16:42:41 I thought about something like pkgadmin 16:42:51 this is probably in the private git repo I cannot access 16:42:51 tyll_: would need to change pkgdb 16:42:57 sounds like busywork to me... but if you like, propose the exact group changes and such and we can get to it if we get to it? 16:42:58 pkgdb has it in a config file 16:43:10 tyll_: and check bodhi 16:43:22 it might also be strange for newcomers to still see references to CVS 16:43:27 sure 16:44:00 tyll_: propose the changes and we will need to file infra tickets 16:44:15 but I can check bodhi and ansible/puppet and prepare patches, but someone else would need to check the private git repo for ansible/puppet secrets 16:44:27 at least the sudoers files are in there 16:44:31 find / -group cvsadmin as well 16:44:32 yeah 16:44:56 there is likely files that will need ownership changed 16:45:26 ok, I will create a ticket and track tasks there 16:45:50 it's not impossible, just the yummy vs trouble ratio is high... but we can add it on the radar in case we find time. ;) 16:47:59 indeed 16:48:05 okay, anything else? 16:49:14 tyll_: something that could come out of making teh change is a better understanding of what exactly is effected and managed by those groups 16:49:28 * nirik nods. 16:49:50 * masta is back lurking (was pulled away for $dayjob) 16:50:15 dgilmore: yes 16:51:56 nirik: would should be the procedure for the failed PSU in ppc-comm04? are you (infra) going to order the replacement or should we organize it? 16:51:57 okay, if nothing else lets wrap up 16:52:12 nirik: we really should see about replacing dwa with "parasense" so I see those nags 16:52:22 sharkcz: oh yeah, was going to mention that. 16:52:44 smooge mailed ibm about it (it's a loaner machine), so they need to ack the issue and file service stuff on it. 16:52:54 ok 16:53:27 I'll try and keep you posted, feel free to ping me anytime for status 16:53:34 thanks 16:54:03 oh, I did have one more final thing... 16:54:13 nirik: go for it 16:54:42 releng folks might want to look at and weigh in on: https://lists.fedoraproject.org/pipermail/env-and-stacks/2014-June/000428.html 16:54:57 someone is writing a tool to interface with koji and scratch build things. 16:55:05 I'd never heard of it until I saw that post. 16:55:31 thats all 16:55:48 isn't that for auto-rebuild broken deps? 16:55:52 nirik: thanks for bringing it to our attention, ive not heard of it at all 16:55:58 am I to imply a build system based on scratch builds? 16:56:13 pingou: i think it is trying to find broken deps when things change 16:56:23 yeah. not fully clear really. 16:56:41 that is kinda cool, actually. but almost abusive of the build system. 16:57:27 im guessing everytime glibc gets rebuilt they will mass rebuild everything as scratch builds 16:57:36 as it will effect everything 16:57:47 they do say they will limit the builds 16:57:53 scratch builds a privilege, not a right. Once they are abused... 16:58:03 masta: we have no such thing 16:58:17 dgilmore: ? 16:58:18 I believe this idea came up several times on fedora-devel 16:58:34 * pingou has had it as well for a little while 16:58:49 it's a neat idea, I've thought of it myself, but never bothered. 16:58:54 masta: there is no such conecpt of " scratch builds a privilege, not a right. Once they are abused..." and we cant really stop people doing them 16:59:14 also, we have a vastly idle setup of builders. ;) 16:59:29 * nirik is happy to have them do something as long as it's productive. 16:59:49 dgilmore: oh! yeah, I just saying my personal opinion... but really if the build system becomes over subscribed with scratch builds, it might be fair to characterize as I have... as "abuse" 17:00:04 they are a lower priority than real builds. 17:00:11 * pingou was about to ask 17:00:16 dgilmore: sry, I always say things poorly 17:00:49 i suspect that the buildsys might always be full, or they wont be able to send all the builds they want 17:00:53 but I could be wrong 17:01:12 its worth looking at 17:01:26 I think suse does something similar 17:03:21 okay, lets put it on the rader and actively engage the devs 17:03:25 radar 17:04:46 okay if nothing else 17:09:50 dgilmore: What do you think about this? https://fedorahosted.org/rel-eng/ticket/5894 17:10:09 Using unofficial git branches in Fedora pkgs repo for non-Fedora-destined changes? 17:11:18 abadger1999: not really sure. I want to enable people to do things that will eventually get to fedora, or prove its not worth doing 17:11:18 and for things that are not reviewed for Fedora. 17:11:38 dgilmore: We've talked about having a different git repo for things that haven't passed review yet.. 17:11:44 Kinda think that would be better. 17:12:11 We've also thought about having a different git repo for copr. 17:12:21 abadger1999: i think it wouldnt be end of the world for in fedora if we can easily throw the work away 17:12:33 I guess that one we could figure out if we want a separate git repo for building copr packages or reuse fedora pkgs rpeo. 17:12:38 but right now we can not throw anything away 17:12:56 there is a cost to having seperate repos 17:13:03 Yeah. I was considering that (whether it's possible to throw it away) this weekend. 17:13:08 but the benefits might outweigh the costs 17:13:16 abadger1999: right now its not 17:13:19 Yep. 17:13:41 we need a plugin for koji to ensure that a commit used in a build is available via an appropriate branch 17:13:51 we have an open ticket for it 17:16:01 Among other things. 17:16:39 abadger1999: in prinicple im not against allowing people to experiemnet 17:17:02 in practice there is some things we might want to ensure happen 17:18:23 maybe not polute lookaside cache 17:18:26 etc 17:18:54 enable all of the experimental work to be thrown away 17:21:08 seems a bit confusing to have stuff in the fedora pkgs repo thats not shipped in fedora... 17:21:18 nirik: somewhat 17:21:43 nirik: but it could easily enable moving it in 17:22:07 yeah, but what if it has legal problems or something thats not been reviewed yet? 17:22:14 we are almost at 2 hours 17:22:18 I guess if they have commit access to the package anyhow... 17:22:28 nirik: maybe legal can write up something for us 17:23:02 to be able to commit you are saying that it complies 17:24:05 the thing we might need to make really clear is that the experimentantal work meets all fedora requirements 17:25:50 lets get a ticket on this 17:25:56 and sort it out 17:26:00 * nirik nods 17:26:02 https://fedorahosted.org/fesco/ticket/1321 17:26:06 Just filed that. 17:26:15 abadger1999: okay 17:26:21 But there might need to be a releng/infra ticket for the technical details as well as the policy. 17:26:34 (I noted some of the technical details in the last paragraph of that ticket) 17:27:04 those may depend on what fesco decides. 17:27:10 Yeah. 17:27:23 lets see what fesco says 17:27:28 wfm. 17:27:31 for now, we are at 2 hours 17:27:35 an hour over 17:27:39 lets wrap up 17:28:03 #endmeeting