15:39:13 <dgilmore> #startmeeting RELENG (2014-06-30)
15:39:13 <zodbot> Meeting started Mon Jun 30 15:39:13 2014 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:39:13 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:39:18 <dgilmore> #meetingname releng
15:39:18 <zodbot> The meeting name has been set to 'releng'
15:39:18 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta
15:39:18 <zodbot> Current chairs: bochecha dgilmore masta nirik sharkcz tyll
15:39:19 <dgilmore> #topic init process
15:39:25 <dgilmore> who is here?
15:39:33 <nirik> hey.
15:39:39 <tyll_> Hi
15:39:39 * sharkcz waves
15:40:09 <dgilmore> #topic #5931 [Proposal] Move new branch and unretire requests to pkgdb2
15:40:15 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5931
15:40:41 <dgilmore> so apparently tyll_ suggested this to pingu
15:41:10 <dgilmore> it will take a massive amount of work and complete changes of many processes to do
15:41:20 <masta> hi
15:42:01 <tyll_> I believe it was planned since the beginning of pkgdb1
15:42:19 <tyll_> at least there is a note on the SCM change request page
15:42:41 <dgilmore> tyll_: i think it will make some aspects of it much much better
15:42:51 <dgilmore> tyll_: and if it was tied into a review app even better
15:43:12 <dgilmore> but it will take redoing how we do it all
15:43:22 <tyll_> at the end of, there is an admontion:  http://fedoraproject.org/wiki/Package_SCM_admin_requests#Package_Change_Requests_for_existing_packages
15:43:46 <dgilmore> tyll_: right, its been there for years
15:43:53 <dgilmore> and we have never done anything about it
15:44:05 <dgilmore> welcome pingou
15:44:17 <tyll_> is there so much that needs to be changed? I assumed it is only the script that processes branch requests
15:44:22 <dgilmore> I think the experience for the users and admins would be much better
15:44:39 <dgilmore> and it would give just one place to go see what needs doing
15:45:31 <dgilmore> tyll_: well the current scripts are run from your local machine, they use auth to bugzilla, pkgdb, ssh to pkgs01, and some koji auth
15:46:19 <dgilmore> tyll_: unblocking requires a releng ticket, adding branches and packages needs a bugzilla request
15:47:03 <dgilmore> we would either need to setup some proxy processes in pkgdb2 or get accounts in the different places for it to work
15:47:37 <pingou> my idea was that pkgdb2 would only replace the tickets
15:47:42 <dgilmore> pingou: how would you see this all being glued together in pkgdb2?
15:48:00 <dgilmore> pingou: tickets are only used for unblocking unretired packages
15:48:06 <pingou> so we would keep the current script, they would just start by querying pkgdb2 to get the list of pending actions
15:48:21 <pingou> dgilmore: there also used for new branch, in bugzilla :)
15:48:25 <dgilmore> pingou: there is no script for unblocking
15:48:50 * pingou does not know enough of these processes
15:49:09 <dgilmore> pingou: adding branches and new packages requires some flags to be set in bugzilla
15:49:21 <pingou> note: pkgdb2 already interact with bugzilla
15:49:29 <tyll_> IMHO a good first step would be to only move the bugzilla SCM admind requests to pkgdb
15:49:33 <dgilmore> pingou: which the script gets the list of packages from bugzilla when it does a run
15:49:50 <dgilmore> tyll_: thats not at all what pingou is looking at doing
15:50:01 <pingou> ?
15:50:47 <dgilmore> pingou: it sounded to me like you just wanted to query bugzilla to get the lis
15:50:51 <dgilmore> list
15:50:53 <dgilmore> we do that already
15:51:04 <tyll_> actually since rel-eng is processing the SCM admin requests, a second ticket to request package unblocking might not be needed anymore, because one either needs both unretiring and unblocking or nothing
15:51:52 <pingou> dgilmore: I am not sure I follow you
15:52:09 <dgilmore> tyll_: we would have to change the process when unretiring but yeah the unblocking could get thrown in there
15:52:41 <dgilmore> pingou: i guess I do not get how you plan to know there is a new cvs admin request
15:53:08 <pingou> dgilmore: my idea was that to request a new branch, instead of going on bugzilla one would go on pkgdb
15:53:45 <dgilmore> pingou: ok and how exactly would that work?
15:53:52 <nirik> we may before too long have bugzilla fedmsgs. ;)
15:54:23 <pingou> dgilmore: the user logs in pkgdb, and click on the button "Request a new branch", that gets adding to the list of actions requiring an admin, available at: http://209.132.184.188/api/admin/actions/
15:54:51 <dgilmore> pingou: how does that work for new packages that do not yet exist in pkgdb?
15:54:54 <pingou> we can also send an email to cvsadmin about this new request
15:55:15 <pingou> dgilmore: I was thinking new branch of existing packages
15:55:26 <dgilmore> pingou: thats really not done that often
15:55:33 <pingou> ok
15:55:37 <dgilmore> pingou: to me is an all or nothing thing
15:55:55 <dgilmore> it would be too confusing to use one way for one thing and other for the other
15:56:06 <dgilmore> people out of habit would use the old way for the other
15:56:13 <tyll_> but for new pkgs it could be the same, login to pkgdb and request a new package
15:56:27 <pingou> so bugzilla = review approved -> package automatically created on pkgdb -> user goes there to ask for branches?
15:56:38 <dgilmore> tyll_: you would need to provide a bug for the review
15:57:09 <dgilmore> pingou: you would need to setup something to ocntantly poll bugzilla
15:57:14 <dgilmore> but you could do that
15:57:37 <pingou> dgilmore: or we wait for bugzilla2fedmsg (hoping it provides us enough info for this kind of action)
15:57:55 <dgilmore> pingou: part of the process that the admin does is to verify that the package review is done properly and by someone in the packager group
15:58:29 <dgilmore> pingou: so I really would rather that the page just get added to pkgdb
15:58:47 <pingou> we could check who set the flag to '+' and check it was a packager
15:58:54 <dgilmore> but instead sit in a state where it can be reviewed and only added when acked by an admin
15:59:08 <dgilmore> pingou: thats not how it works
15:59:19 <tyll_> it might be that only packagers can actually set the review flag
15:59:24 <dgilmore> pingou: the review flag has to be + but the cvsadmin flag ?
15:59:37 <dgilmore> and thats not enough to tell us the review is done right
15:59:46 <dgilmore> that can only be done by examination of the bug
15:59:52 <tyll_> dgilmore: there would be no cvsadmin flag anymore
15:59:54 <pingou> dgilmore: but doesn't cvsadmin flag disapear if it's moved to pkgdb?
16:00:02 <dgilmore> tyll_: bugzilla doesnt know anything about fas groups
16:00:04 <pingou> (agreed wrt to the review)
16:00:20 <dgilmore> pingou: perhapsd
16:00:23 <dgilmore> perhaps
16:01:01 <dgilmore> what i think would be best, and we can work on something to get to that place is
16:01:17 <dgilmore> we have a review app, which i vaguely remeber somoene was working on
16:01:55 <pingou> that'd be I :]
16:01:56 <nirik> anyone in 'fedora bugs' can set the flags.
16:01:59 <dgilmore> when the package gets marked as reviewed it then calls out and setsup everything for review
16:02:20 <pingou> dgilmore: I agree bugzilla doesn't know about FAS, but we do, so when seeing (on fedmsg) that someone updated the review flag to '+', we can see who did it (packager?) and create the package setup everything for review
16:02:56 <dgilmore> the cvsadmin checks everything off  and approves it in review app or pkgdb and then that tool actually goes off and does all the steps to add the package etc
16:03:25 <dgilmore> pingou: we have ways to automate that yes
16:03:42 <dgilmore> pingou: we should also verify that teh submitter is a approved packager also
16:03:54 <dgilmore> if not flag them to get a sponsor
16:04:49 <pingou> so upon review set has '+' -> automatically set cvsadmin flag to '?' -> cvsadmin member double checks the review and set the cvsadmin flag to '+'
16:05:11 <pingou> upon cvsadmin flag set to '+' -> pkgdb create the package => user goes to pkgdb to ask for the branches
16:05:39 <pingou> cvsadmin (or cron job?), creates new branches
16:05:53 <nirik> well, the downside there is that anyone can set the flag in bz.
16:06:05 <tyll_> what do we need to cvsadmin flag for?
16:06:18 <pingou> nirik: but we know the email of the user that does the action, so we can check if that person is allowed to do it
16:06:22 <nirik> probibly should keep state in pkgdb or something.
16:06:42 <tyll_> the reviewer could just set the review flag to + and the SCM admin could post a comment that it is added to pkgdb
16:06:48 <nirik> review + -> add to scm admin queue -> when admin says 'ok', process it?
16:06:50 <dgilmore> pingou: i think it would ba a state in pkgdb or the review app
16:07:00 <pingou> nirik: that's the idea (iirc)
16:07:12 <nirik> yeah, we could drop the cvsadmin flag entirely.
16:07:15 <pingou> s/iirc/iiuc/
16:07:20 <nirik> although they need to tell us what branches I guess.
16:07:39 <pingou> other option
16:07:39 <dgilmore> once the package is added to pkgdb, we need to create the git repo, then any additional branches
16:08:10 <dgilmore> we alrady have a process that syncs from pkgdb to koji package owners
16:08:25 <dgilmore> we could maybe try teach it to deal with unblocking
16:08:29 <pingou> reivew set to '+', user goes to pkgdb -> request new package w/ its branches, provide link to bugzilla review -> cvsadmin check the package review and create the package/branches
16:08:39 <dgilmore> but the sync script is pretty basic
16:08:56 <dgilmore> pingou: could be step 1
16:09:29 <dgilmore> pingou: in the end the goal of automation was the admin signs off and everything else just happens
16:09:46 <dgilmore> we just never got there
16:10:38 <pingou> dgilmore: if we do that in pkgdb2, package/branch creation will be broadcasted on fedmsg and we could have a listener that just create the git/branches automatically
16:10:50 <pingou> abadger1999: I liked the other one better :D
16:11:01 * nirik nods. there's lots of places we could make scripts fedmsg aware
16:11:09 <abadger1999> :-)
16:11:09 <dgilmore> pingou: yes, but we also need to account for the fact that fedmsg is not guaranteed to be reliable
16:11:43 <pingou> dgilmore: that is true
16:11:45 <nirik> trust, but verify. ;)
16:13:09 <tyll_> nirik: I guess missing messages are more problematic, since messages are already signed afaik
16:13:35 <dgilmore> pingou: we could also rewrite how we sync packages to add them when added in real time rather than the 10 minute cron job
16:13:45 <nirik> right. I meant we should assume it's reliable, but also have some script/way to check that it got all the messages or that nothing was missed.
16:13:54 <pingou> periodically check the api endpoint should help cover that tyll_ I think
16:14:49 <pingou> dgilmore: I don't know the process enough to follow you here
16:14:51 <dgilmore> so i guess the summary right now is that we should work towards this, but there is a lot of implementation details to work out
16:15:10 <dgilmore> pingou: we can go over it some time
16:15:17 <pingou> sounds like a summary :)
16:15:22 <pingou> dgilmore: sure, thanks! :)
16:15:31 <tyll_> #info 8:14 < dgilmore> so i guess the summary right now is that we should work towards this, but there is a lot of implementation details to work out
16:16:04 <dgilmore> #info we should work towards more integrated and better automation, there is a lot of details to work out, and it will likely be done in phases
16:16:53 * pingou will add the 'new package request' on pkgdb2
16:17:09 <nirik> lets hash out how we want it all to work first?
16:17:22 <dgilmore> nirik: yeah
16:17:53 <dgilmore> pingou: before doing that, lets map out the processes and work out how we would like it all to work
16:18:00 <pingou> sure
16:18:17 <dgilmore> don't want to waste your time and effort
16:18:27 <pingou> looks like I'll have to subscribe to the releng list :)
16:18:49 <dgilmore> pingou: :) it gets less mail than a couple of weeks ago
16:19:01 <dgilmore> anything else here?
16:19:16 * pingou subscribed
16:19:23 <dgilmore> im wondering if we shouldnt skip the other two tickets, since this took up so much time
16:19:42 <dgilmore> and get on with the secondary arch status's
16:19:48 <tyll_> there is not much new there except that autosign01 is now RHEL7
16:20:10 <dgilmore> tyll_: that was kinda my thoughts
16:20:35 * nirik nods.
16:20:41 <dgilmore> #topic Secondary Architectures updates
16:20:42 <dgilmore> #topic Secondary Architectures update - ppc
16:20:47 <dgilmore> masta: you're up
16:21:48 <dgilmore> gfuess he disapeared
16:22:00 <dgilmore> not sure where things were last week
16:22:11 <dgilmore> but be and le have merged the hubs
16:22:26 <sharkcz> mass rebuild is in progress, 15k build done,. 500+ failures
16:22:30 <dgilmore> nightly composes push out ppc64 and ppc64le trees
16:22:47 <dgilmore> sharkcz: about where primary is then
16:24:02 <sharkcz> ~1200 are broken deps, so not so far
16:24:09 <dgilmore> okay
16:24:13 <nirik> oh... one thing I noticed...
16:24:28 <nirik> ppc hub is 95% full on /
16:24:40 <dgilmore> probably logs
16:24:58 <sharkcz> nirik: it's s390 hub, ppc is 82% 1.7T free
16:25:01 <nirik> sorry, 98%
16:25:05 <dgilmore> #action masta to look at teh hub and manage space
16:25:13 <nirik> 94G   87G  2.1G  98% /
16:25:24 <sharkcz> ah, the /
16:25:24 <dgilmore> sharkcz: / not /mnt/koji
16:25:29 <nirik> those nagios emails have been going to me and dwa.
16:25:45 <nirik> should I send them to sysadmin-releng? sysadmin? specific people?
16:25:52 <sharkcz> it will be rpms from shadow
16:25:59 <dgilmore> nirik: we should s/dwa/masta/ likely there will be a new ppc person very soon
16:26:12 <nirik> ok, happy to adjust it so it's not just me. ;)
16:26:42 <nirik> (since I often don't know what to remove)
16:26:53 <dgilmore> lets do masta for now
16:27:01 * nirik nods
16:27:11 <dgilmore> #topic Secondary Architectures update - s390
16:27:17 <dgilmore> sharkcz: floor is yours
16:27:50 <sharkcz> s390 is fighting with gcc 4.9 causing build failures in gnutls, texlive, ...
16:28:13 <sharkcz> so there is progress, but it is slow
16:28:45 <sharkcz> I have people working on reducing the code so bug reports could be filled
16:30:33 <dgilmore> okay
16:30:41 <dgilmore> anything we can help with?
16:30:58 <sharkcz> probably not
16:31:52 <sharkcz> nirik: I've freed some space under /tmp on the ppc hub, it's 78% full now
16:31:53 <dgilmore> okay thanks
16:32:00 <nirik> sharkcz: thanks
16:32:06 <dgilmore> #topic Secondary Architectures update - arm
16:32:16 <sharkcz> nirik: np, shadow would crash soon
16:32:37 <sharkcz> so thanks for noticing :-)
16:32:48 <dgilmore> statistics: {'older': 358, 'local_only': 0, 'remote_only': 477, 'same': 14599, 'newer': 1, 'total_missing_builds': 551}
16:33:04 <sharkcz> very nice numbers
16:33:06 <dgilmore> so aarch64 is really close to primary for rawhide
16:34:34 <dgilmore> its moving along well
16:34:41 <nirik> cool.
16:35:00 <dgilmore> there is two issues i know of that will need kernel patches to have f21 work on hardware
16:35:50 <dgilmore> we should be good to go for alpha
16:36:03 <dgilmore> #topic Open Floor
16:36:06 <dgilmore> okay
16:36:09 <dgilmore> one issue
16:36:23 <dgilmore> branching is very very soon
16:37:02 <nirik> a week from tomorrow
16:37:02 <dgilmore> im planning to spend some time this week on trying to get a compose done and see what can be done to optimise things
16:37:20 <dgilmore> since we will need to start TC composes next week
16:39:28 <dgilmore> anyone have anything else for open floor?
16:39:53 <tyll_> I wanted to propose to get rid of cvs in FAS groups
16:40:02 <tyll_> I mentioned it on the channel once
16:40:25 <dgilmore> tyll_: changing groups in fas is a mess and not really possible
16:40:40 <tyll_> we can create new ones and not use the old ones
16:41:01 <dgilmore> we can, but would need to dig into all the places that use the cvs groups for things
16:41:32 <tyll_> there is not much stuff in ansible or puppet
16:41:48 <tyll_> I believe only the gitolite script and the pkgdb2 config
16:41:58 <dgilmore> if we do go down that route we should use scmadmin sysadmin-scm
16:42:14 <dgilmore> tyll_: there is more than that
16:42:20 <tyll_> yes, at least something generic
16:42:34 <dgilmore> at the least there is the access to pkgs01
16:42:41 <tyll_> I thought about something like pkgadmin
16:42:51 <tyll_> this is probably in the private git repo I cannot access
16:42:51 <dgilmore> tyll_: would need to change pkgdb
16:42:57 <nirik> sounds like busywork to me... but if you like, propose the exact group changes and such and we can get to it if we get to it?
16:42:58 <tyll_> pkgdb has it in a config file
16:43:10 <dgilmore> tyll_: and check bodhi
16:43:22 <tyll_> it might also be strange for newcomers to still see references to CVS
16:43:27 <dgilmore> sure
16:44:00 <dgilmore> tyll_: propose the changes and we will need to file infra tickets
16:44:15 <tyll_> but I can check bodhi and ansible/puppet and prepare patches, but someone else would need to check the private git repo for ansible/puppet secrets
16:44:27 <tyll_> at least the sudoers files are in there
16:44:31 <abadger1999> find / -group cvsadmin as well
16:44:32 <dgilmore> yeah
16:44:56 <dgilmore> there is likely files that will need ownership changed
16:45:26 <tyll_> ok, I will create a ticket and track tasks there
16:45:50 <nirik> it's not impossible, just the yummy vs trouble ratio is high... but we can add it on the radar in case we find time. ;)
16:47:59 <dgilmore> indeed
16:48:05 <dgilmore> okay, anything else?
16:49:14 <dgilmore> tyll_: something that could come out of making teh change is a better understanding of what exactly is effected and managed by those groups
16:49:28 * nirik nods.
16:49:50 * masta is back lurking (was pulled away for $dayjob)
16:50:15 <tyll_> dgilmore: yes
16:51:56 <sharkcz> nirik: would should be the procedure for the failed PSU in ppc-comm04? are you (infra) going to order the replacement or should we organize it?
16:51:57 <dgilmore> okay, if nothing else lets wrap up
16:52:12 <masta> nirik: we really should see about replacing dwa with "parasense" so I see those nags
16:52:22 <nirik> sharkcz: oh yeah, was going to mention that.
16:52:44 <nirik> smooge mailed ibm about it (it's a loaner machine), so they need to ack the issue and file service stuff on it.
16:52:54 <sharkcz> ok
16:53:27 <nirik> I'll try and keep you posted, feel free to ping me anytime for status
16:53:34 <sharkcz> thanks
16:54:03 <nirik> oh, I did have one more final thing...
16:54:13 <dgilmore> nirik: go for it
16:54:42 <nirik> releng folks might want to look at and weigh in on: https://lists.fedoraproject.org/pipermail/env-and-stacks/2014-June/000428.html
16:54:57 <nirik> someone is writing a tool to interface with koji and scratch build things.
16:55:05 <nirik> I'd never heard of it until I saw that post.
16:55:31 <nirik> thats all
16:55:48 <pingou> isn't that for auto-rebuild broken deps?
16:55:52 <dgilmore> nirik: thanks for bringing it to our attention, ive not heard of it at all
16:55:58 <masta> am I to imply a build system based on scratch builds?
16:56:13 <dgilmore> pingou: i think it is trying to find broken deps when things change
16:56:23 <nirik> yeah. not fully clear really.
16:56:41 <masta> that is kinda cool, actually. but almost abusive of the build system.
16:57:27 <dgilmore> im guessing everytime glibc gets rebuilt they will mass rebuild everything as scratch builds
16:57:36 <dgilmore> as it will effect everything
16:57:47 <dgilmore> they do say they will limit the builds
16:57:53 <masta> scratch builds a privilege, not a right. Once they are abused...
16:58:03 <dgilmore> masta: we have no such thing
16:58:17 <masta> dgilmore: ?
16:58:18 <tyll_> I believe this idea came up several times on fedora-devel
16:58:34 * pingou has had it as well for a little while
16:58:49 <masta> it's a neat idea, I've thought of it myself, but never bothered.
16:58:54 <dgilmore> masta: there is no such conecpt of " scratch builds a privilege, not a right. Once they are abused..." and we cant really stop people doing them
16:59:14 <nirik> also, we have a vastly idle setup of builders. ;)
16:59:29 * nirik is happy to have them do something as long as it's productive.
16:59:49 <masta> dgilmore: oh! yeah, I just saying my personal opinion... but really if the build system becomes over subscribed with scratch builds, it might be fair to characterize as I have... as "abuse"
17:00:04 <nirik> they are a lower priority than real builds.
17:00:11 * pingou was about to ask
17:00:16 <masta> dgilmore: sry, I always say things poorly
17:00:49 <dgilmore> i suspect that the buildsys might always be full, or they wont be able to send all the builds they want
17:00:53 <dgilmore> but I could be wrong
17:01:12 <dgilmore> its worth looking at
17:01:26 <dgilmore> I think suse does something similar
17:03:21 <dgilmore> okay, lets put it on the rader and actively engage the devs
17:03:25 <dgilmore> radar
17:04:46 <dgilmore> okay if nothing else
17:09:50 <abadger1999> dgilmore:   What do you think about this? https://fedorahosted.org/rel-eng/ticket/5894
17:10:09 <abadger1999> Using unofficial git branches in Fedora pkgs repo for non-Fedora-destined changes?
17:11:18 <dgilmore> abadger1999: not really sure. I want to enable people to do things that will eventually get to fedora, or prove its not worth doing
17:11:18 <abadger1999> and for things that are not reviewed for Fedora.
17:11:38 <abadger1999> dgilmore: We've talked about having a different git repo for things that haven't passed review yet..
17:11:44 <abadger1999> Kinda think that would be better.
17:12:11 <abadger1999> We've also thought about having a different git repo for copr.
17:12:21 <dgilmore> abadger1999: i think it wouldnt be end of the world for in fedora if we can easily throw the work away
17:12:33 <abadger1999> I guess that one we could figure out if we want a separate git repo for building copr packages or reuse fedora pkgs rpeo.
17:12:38 <dgilmore> but right now we can not throw anything away
17:12:56 <dgilmore> there is a cost to having seperate repos
17:13:03 <abadger1999> Yeah.  I was considering that (whether it's possible to throw it away) this weekend.
17:13:08 <dgilmore> but the benefits might outweigh the costs
17:13:16 <dgilmore> abadger1999: right now its not
17:13:19 <abadger1999> Yep.
17:13:41 <dgilmore> we need a plugin for koji to ensure that a commit used in a build is available via an appropriate branch
17:13:51 <dgilmore> we have an open ticket for it
17:16:01 <abadger1999> <nod> Among other things.
17:16:39 <dgilmore> abadger1999: in prinicple im not against allowing people to experiemnet
17:17:02 <dgilmore> in practice there is some things we might want to ensure happen
17:18:23 <dgilmore> maybe not polute lookaside cache
17:18:26 <dgilmore> etc
17:18:54 <dgilmore> enable all of the experimental work to be thrown away
17:21:08 <nirik> seems a bit confusing to have stuff in the fedora pkgs repo thats not shipped in fedora...
17:21:18 <dgilmore> nirik: somewhat
17:21:43 <dgilmore> nirik: but it could easily enable moving it in
17:22:07 <nirik> yeah, but what if it has legal problems or something thats not been reviewed yet?
17:22:14 <dgilmore> we are almost at 2 hours
17:22:18 <nirik> I guess if they have commit access to the package anyhow...
17:22:28 <dgilmore> nirik: maybe legal can write up something for us
17:23:02 <dgilmore> to be able to commit you are saying that it complies
17:24:05 <dgilmore> the thing we might need to make really clear is that the experimentantal work meets all fedora requirements
17:25:50 <dgilmore> lets get a ticket on this
17:25:56 <dgilmore> and sort it out
17:26:00 * nirik nods
17:26:02 <abadger1999> https://fedorahosted.org/fesco/ticket/1321
17:26:06 <abadger1999> Just filed that.
17:26:15 <dgilmore> abadger1999: okay
17:26:21 <abadger1999> But there might need to be a releng/infra ticket for the technical details as well as the policy.
17:26:34 <abadger1999> (I noted some of the technical details in the last paragraph of that ticket)
17:27:04 <nirik> those may depend on what fesco decides.
17:27:10 <abadger1999> Yeah.
17:27:23 <dgilmore> lets see what fesco says
17:27:28 <abadger1999> wfm.
17:27:31 <dgilmore> for now, we are at 2 hours
17:27:35 <dgilmore> an hour over
17:27:39 <dgilmore> lets wrap up
17:28:03 <dgilmore> #endmeeting