15:36:13 <dgilmore> #startmeeting RELENG (2014-08-25)
15:36:13 <zodbot> Meeting started Mon Aug 25 15:36:13 2014 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:36:13 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:36:26 <dgilmore> #meetingname releng
15:36:26 <zodbot> The meeting name has been set to 'releng'
15:36:30 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson
15:36:30 <zodbot> Current chairs: bochecha dgilmore masta nirik pbrobinson sharkcz tyll
15:36:41 <dgilmore> #topic init process
15:36:56 * masta is here
15:37:00 * bochecha is here
15:37:07 <tyll> Hi all
15:37:07 * nirik is here
15:37:08 * sharkcz is here
15:37:47 <dgilmore> #topic #5931 [Proposal] Move new branch and unretire requests to pkgdb2
15:37:57 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5931
15:38:04 <dgilmore> pingu ping
15:38:28 <pingou> hi
15:38:52 <pingou> so I think on the pkgdb2 side it's ready
15:38:56 <dgilmore> okay
15:39:17 <pingou> I have been working on a fedmsg-pkgdb2branch that would take pkgdb2 fedmsg messages and call pkgdb2branch on pkgs01
15:39:22 <pingou> for the packages concerned
15:39:53 <pingou> the idea is then to provide a CLI to interact with pkgdb2's AdminAction, and close bugzilla tickets
15:40:11 <pingou> the git management would then be taken care of via fedmsg
15:41:00 <pingou> I guess that's it for me :)
15:41:16 <nirik> cool.
15:41:41 <tyll> #info  < pingou> I have been working on a fedmsg-pkgdb2branch that would take pkgdb2 fedmsg messages and call pkgdb2branch on pkgs01
15:41:51 <tyll> #info < pingou> the idea is then to provide a CLI to interact with pkgdb2's AdminAction, and close bugzilla tickets
15:41:58 <pingou> nirik: does that answer your question about the fedmsg-pkgdb2branch project you had earlier?
15:42:33 <nirik> just one further one...
15:42:35 <tyll> #info < pingou> so I think on the pkgdb2 side it's ready
15:43:05 <pingou> the part I'm not sure about is: how to deal with fedmsg message missed?
15:43:07 <nirik> with other things we fedmsg enabled we have a catchall thing (ie, with fasClient we have a daily cron to make sure it's synced if it missed a fedmsg). Do we have any way to do that with branch requests?
15:43:15 <nirik> exactly. ;)
15:43:27 <pingou> so daily run of pkgdb2branch on all packages?
15:43:36 <nirik> not sure how costly that is...
15:43:39 <pingou> or bi-daily?
15:44:08 <nirik> there's a lot of packages.
15:44:13 <pingou> true there
15:44:14 <nirik> I guess we could see how long that takes...
15:44:41 <pingou> maybe not pkgdb2branch then but a dedicated one, sending a report
15:44:57 <pingou> (and/or fixing)
15:45:32 <nirik> yeah, could work.
15:45:38 <nirik> we can investigate that out of meeting?
15:45:41 <pingou> sure
15:45:50 * pingou has both scripts on his todo
15:45:55 <nirik> cool.
15:46:16 <nirik> I'm hoping to work on ansible migration for pkgs01.stg this week.
15:46:21 <nirik> so we could have a good place to test.
15:46:23 <pingou> cool :)
15:46:35 <pingou> let me know if I can help w/ that
15:46:42 <tyll> #info add daily cron script as fallback for missed fedmsg messages
15:46:47 <pingou> (although bochecha is likely in a better position for this)
15:47:02 <tyll> #info < nirik> I'm hoping to work on ansible migration for pkgs01.stg this week.
15:48:20 <bochecha> pingou, I can't help on this, it's my patches that nirik will review ;)
15:48:21 <dgilmore> anything else here?
15:50:45 <dgilmore> #topic #5959 Enable keep-alive connections for koji (primary and secondaries)
15:50:57 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5959
15:51:18 <dgilmore> pbrobinson is in the UK and they have a public holiday
15:51:30 <dgilmore> not sure if this was implemented and the results
15:51:46 <tyll> #info deferred due to public holidy in the UK
15:51:46 <dgilmore> #topic #5870 rawhide signing
15:51:54 <dgilmore> #undo
15:51:54 <zodbot> Removing item from minutes: <MeetBot.items.Topic object at 0xb6eacd0>
15:51:55 <dgilmore> #undo
15:51:55 <zodbot> Removing item from minutes: INFO by tyll at 15:51:46 : deferred due to public holidy in the UK
15:52:05 <dgilmore> #info deferred due to public holidy in the UK
15:52:10 <dgilmore> #topic #5870 rawhide signing
15:52:17 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5870
15:52:25 <dgilmore> okay so this is kinda working
15:52:33 <dgilmore> other than sigul keeps falling over
15:52:59 <dgilmore> in particular the bridge is locking up frequently
15:53:06 <tyll> yes, therefore training someone from Europ (e.g. pingou ) to restart it would be nice
15:53:31 <dgilmore> I will talk with nirik to find someone extra
15:53:44 <tyll> also maybe some automatic monitoring would be nice
15:53:53 <dgilmore> the passphrase to unlock the nss database is shared and not per user
15:53:55 <pingou> we're two -main in Europe, anyone of us could do :)
15:54:02 <tyll> e.g. ngios alerts when the bridges are broken
15:54:03 <pingou> (if that helps of course)
15:54:15 <nirik> we could possibly setup a nagios alert yeah...
15:54:36 <nirik> it seems like secondary falls over more than primary? or about the same?
15:55:06 <nirik> tyll: how close are we to all signed 21/22?
15:55:25 <tyll> secondary is used more, so I notice it easier
15:55:40 <dgilmore> secondary is signing 3 tiems the rpms
15:55:48 <dgilmore> but i think its about equal
15:56:27 <dgilmore> its the bridge locking up, we should try get some of mitr's time and see if we can't fix it
15:56:36 <nirik> yeah. that would sure be nice. ;)
15:57:00 <tyll> nirik: Currently the script is running non stop, but I did not check how many RPMs are still unsigned - if anything is currently unsigned, there is a bug (at least in primary, not sure if someone signed everything from the mass rebuilds in the secondaries)
15:57:36 <tyll> #info sigul breaks regularly, dgilmore and nirik will find additional admins that can restart it when they are not available
15:57:37 <nirik> ok, entering freeze for 21 we need to make sure everything is signed.
15:57:46 <masta> I tried last night on ppc, but something was wrong with sigul
15:57:57 <nirik> and then we can change the autosigner to concentrate on rawhide I guess.
15:58:24 <dgilmore> tyll: I know some things are unsigned.
15:58:48 <dgilmore> I had to add --nogpgcheck when installing software onto the new laptop this weekend
15:59:11 <nirik> http://koji.fedoraproject.org/mash/branched-20140825/logs/mash.log
15:59:21 <tyll> dgilmore: since there is no gating, there might also be timing issues, especially when sigul is down
15:59:22 <nirik> ( that lists the things without the right key)
15:59:47 <dgilmore> tyll: right
16:00:37 <tyll> The "None" epoch in mash#s log makes it not nice to grep in the autosigner logs
16:02:10 <dgilmore> yeah its kinda ugly
16:02:26 <dgilmore> that we expose epochs in some places and not others is ugly
16:02:36 * nirik nods.
16:03:19 <dgilmore> nirik: i need to flush the squid caches again
16:03:32 <dgilmore> probably change pungify
16:03:38 <tyll> I will take a look to check why the packages from the mash log are not signed
16:03:39 <dgilmore> thats a differnt topic though
16:03:59 <nirik> yeah, lets get f21 frozen, then we can concentrate the autosigner on rawhide again.
16:04:03 <nirik> and gating, etc
16:05:31 <dgilmore> yep
16:05:39 <dgilmore> we get auto gating with bodhi
16:05:58 <nirik> right, f21 becomes not this tickets issue once we freeze. ;)
16:06:30 <tyll> btw. Do we now have enough space on the secondary kojis for the signed RPMs?
16:07:15 <dgilmore> not 100% sure
16:07:48 <dgilmore> vtap-fedora-nfs01.storage.phx2.redhat.com:/vol/fedora_arm/data 5.0T  4.2T  761G  85% /mnt/koji
16:07:58 <sharkcz> ppc has 1.9T free
16:08:14 <nirik> so yeah, I have been working with netapp folks to get more space free on primary...
16:08:20 <dgilmore> vtap-fedora-nfs01.storage.phx2.redhat.com:/vol/fedora_s390/data 6.5T  5.8T  666G  90% /mnt/koji
16:08:30 <nirik> I removed old mashes and reduced the number of snapshots we keep.
16:08:31 <sharkcz> s390 has 600G free, but it gets expanded +0.5T when free space is below 200G
16:08:48 <nirik> sharkcz: did we have any snapshots on it?
16:09:04 <sharkcz> nirik: on s390? nope, it's disabled
16:09:25 <nirik> yeah, I see that now, ok.
16:09:26 <sharkcz> ppc has snapshots enabled
16:09:38 <sharkcz> if I see correctly
16:09:51 <nirik> we may get migrated to a new cmode filer before too long. I am talking with storage folks about that.
16:09:58 <nirik> any outage should be hopefully pretty short.
16:13:08 <dgilmore> anything else here?
16:13:33 <tyll> not from me
16:13:41 <dgilmore> #topic #5914 Move fedmsg based blocking service to Fedora Infrastructure
16:13:46 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5914
16:13:56 <tyll> There is nothing new here due to lack of time
16:14:00 <dgilmore> tyll: I need to get you a user cert right
16:14:20 <tyll> dgilmore: yes, eventually
16:14:38 <dgilmore> okay, let us know what you need
16:14:45 <dgilmore> sadly i can not make time
16:15:17 <dgilmore> #topic Secondary Architectures updates
16:15:18 <dgilmore> #topic Secondary Architectures update - ppc
16:15:29 <dgilmore> sharkcz: so i know your having issues with pungi
16:15:35 <dgilmore> anything else going on?
16:15:42 <dgilmore> you're
16:16:03 <sharkcz> seems they were cause by the mirrorlist (or outdated mirrors), looks OK with pointers to koji/mash
16:16:19 <dgilmore> sharkcz: okay, maybe some latent yum bugs
16:16:36 <sharkcz> well, almost ok, just grub2 is not included by lorax
16:16:41 <masta> I was having some problems with more recent versions of the sigulsing_unsigned.py on ppc recently, had to checkout an older version. Hoping to bisect the issue later today
16:16:57 <dgilmore> sharkcz: that would be a lorax template update then?
16:17:22 <sharkcz> don't thiok so, as the standalone lorax run did it correctly
16:17:27 <dgilmore> masta: that seems odd. im using the current code on both arm and primary
16:17:45 <dgilmore> sharkcz: maybe a pungi bug then
16:17:56 <masta> dgilmore: the error text complained about kojihelper
16:18:07 <sharkcz> dgilmore: have a log from pungi, so we can look
16:18:15 <dgilmore> we should get it fixed asap so we do not need to try push in an updated pungi
16:18:28 <sharkcz> yes
16:19:16 <dgilmore> sharkcz: there was a self introduction on the ppc list last week
16:19:20 <dgilmore> no one has responded
16:19:49 <sharkcz> ah, right, I forgot and no one else probably saw it :-)
16:19:58 <sharkcz> :-(
16:20:23 <masta> oh yes, I think somebody nominated me to be a package sponsor... I was kinda amused by that.
16:20:24 <sharkcz> it's the guy who is/was trying to revive ppc
16:20:44 <masta> dgilmore: is this the yaboot email?
16:21:01 <dgilmore> masta: well thats one of the things he wants to work on
16:21:06 <dgilmore> but we really want it dead
16:21:19 <sharkcz> ack
16:22:05 <dgilmore> at this point 32 bit ppc would be massive massive amounts of work to revive
16:22:06 <masta> yeah I was wanting some guidance on that,  firstly I don't know how I got pulled into sponsoring yaboot, or if we even want yaboot....
16:22:30 <dgilmore> yaboot is the least of the issues
16:22:42 * nirik personally thinks a $35 arm board is more functional than a 32bit ppc from ages past, but to each their own.
16:22:43 <sharkcz> dgilmore: yes, maybe as terciary arch
16:22:43 <masta> yaboot runs on a system I have here, but I was planning to upgrade that to grub2
16:22:57 <dgilmore> sharkcz: I guess so
16:23:05 <sharkcz> but not our fight ...
16:23:28 <dgilmore> sharkcz: I think we should go to FESCo and get them to say how they think it should work
16:24:17 <sharkcz> dgilmore: would make sense probably
16:25:40 <dgilmore> I do not want to add it back to the ppc hub
16:25:46 <dgilmore> it will need to be seperate
16:25:58 <dgilmore> but how we handle it I think FESCo should say
16:26:00 * sharkcz agrees
16:26:37 <dgilmore> #action dgilmore to open ticket with FESCo on how we should handle 32 bit ppc
16:27:01 <sharkcz> I don't think one man hobby project can manage it, it's more a fulltime job
16:27:18 <sharkcz> as a full "secondary" arch
16:27:20 <dgilmore> right
16:27:50 <masta> yes
16:29:52 <sharkcz> otherwise no major issues on ppc afaik, except mariadb and its tests-suite
16:29:59 <dgilmore> okay
16:30:01 <dgilmore> :(
16:30:10 <dgilmore> #topic Secondary Architectures update - s390
16:30:17 <dgilmore> sharkcz: you're up agian
16:30:20 <dgilmore> again
16:31:10 <sharkcz> yes :-) the glibc ABI is 60+% done, will have to concentrate on the perl rebootstrap, should be easier now as I can use the old rpms (pre glibc 2.19)
16:31:21 <sharkcz> glibc ABI rebuild
16:31:59 <sharkcz> it also means that currently the f21 is broken for users
16:32:47 <sharkcz> I guess it should be fixed/rebuilt next week
16:33:12 <sharkcz> I've learned the hard way 6 month ago :-)
16:34:13 <sharkcz> also no composes yet, so we don't know is the tools still work, but hopefully the ydo
16:34:24 <sharkcz> s/is/if/
16:35:04 <sharkcz> EOM
16:36:04 <dgilmore> okay thanks
16:36:12 <dgilmore> #topic Secondary Architectures update - arm
16:36:27 <dgilmore> with pbrobinson being away there is no update from him
16:37:13 <dgilmore> Planning to setup a box in phx early this week so we can get composes moving. I have done a TC compose from an external box and installed it. not expecting any big surprisses
16:37:29 <dgilmore> afaik the boot.iso and install dvd will not be bootable
16:37:47 <dgilmore> there is no support in mkisofs afaik to make bootable aarch64 media
16:37:58 <dgilmore> #topic Open Floor
16:38:06 <dgilmore> anyone have anything they want to discuss?
16:38:27 <masta> I want to learn about koji-shadow, could somebody show me?
16:38:49 <masta> either ppc64 or aarch64 ?
16:40:14 <dgilmore> masta: sure, you will need to work with the folks currently running it, which should be pbrobinson all both ppc and aarch64
16:40:36 <sharkcz> masta: see http://ppc.koji.fedoraproject.org/shadow/ - it has logs, config and script how it is run
16:40:36 <masta> okay
16:41:29 <dgilmore> #info fesco ticket for 32 bit ppc https://fedorahosted.org/fesco/ticket/1336
16:41:39 <sharkcz> masta: also http://sharkcz.livejournal.com/11410.html
16:41:44 <masta> thanks, koji-shadow is a gap in my knowledge of how things work on secondaries.
16:42:30 <sharkcz> masta: and read the source code :-)
16:43:37 <masta> sharkcz: yes indeed!
16:44:00 <masta> who is the maintainer of ks?
16:44:05 <dgilmore> me
16:44:18 <dgilmore> with many patches from the regular users
16:44:27 <sharkcz> like /me :-)
16:45:46 <sharkcz> dgilmore: did you see https://fedoraproject.org/wiki/Architectures/KojiShadowNG (I sent the link to releng list ~week ago)
16:45:53 <dgilmore> sharkcz: I did
16:46:02 <dgilmore> not had a chance to follow up on it yet
16:46:07 <sharkcz> ok, it's just a start
16:46:13 <dgilmore> last week was all about trying to get a TC done
16:46:19 <nirik> is that going to look at using fedmsg?
16:46:52 <sharkcz> like integrating koji-stalk (written by dwa)? probably yes
16:47:18 <nirik> it would be nicer for our fedmsg bus is builds on secondaries just happened around the time they did on primary.
16:47:21 <dgilmore> nirik: likely we will integrate koji-stalk and add a daemon mode
16:47:39 <nirik> right now there's a giant pile of them that happen at once when someone runs koji shadow
16:47:40 <sharkcz> nirik: problem is the something gets broken
16:47:49 <dgilmore> nirik: it would be nicer for many things, practically its not always possibe
16:47:58 <nirik> sure, agreed. would be nice to try. ;)
16:48:02 <sharkcz> and it happens all the time :-(
16:48:31 <dgilmore> I do want to have secondary arches more integrated into everything and make it more visable
16:48:40 <dgilmore> so breakages are more noticed
16:48:44 * sharkcz agrees
16:50:41 <dgilmore> that is a longer term roadmap thing
16:50:47 <dgilmore> but something to keep in mind
16:51:53 <sharkcz> same as allow multiple people to work on a arch, etc
16:52:38 <dgilmore> yeah
16:52:47 <dgilmore> needs to be easier
16:52:53 <dgilmore> and have full composes all teh time
16:53:24 <sharkcz> yeah, would be nice :-) good to know we have still plenty of work ahead :-)
16:53:41 <dgilmore> pretty sure we have years of work ahead
16:53:48 <nirik> yep
16:53:51 <dgilmore> for multiple people
16:54:32 <dgilmore> bochecha: ping, want to give an update of the composedb?
16:55:01 <bochecha> dgilmore, ah, I talked about it with dmach today
16:55:32 <bochecha> one thing he started working on is a module to parse/make composeinfo files
16:56:05 <bochecha> would be nice to look into using that (in releng/scripts/build_composeinfo for example), so that we share the same format as rhel
16:56:15 <bochecha> and then these composeinfo files can be used as the input for composedb
16:56:27 <dgilmore> bochecha: sure
16:56:54 <bochecha> only beaker uses composeinfo files currently, right?
16:57:19 <bochecha> (because dmach is changing the format, so we'll need to make sure users are ported over)
16:58:00 * sharkcz needs to leave ...
16:58:16 <dgilmore> sharkcz: cheers
16:58:28 <dgilmore> bochecha: he shouldn't change the format
16:58:41 <dgilmore> at least not without talking to the beaker devs
16:58:49 <dgilmore> as they are the only consumers i know of
16:58:58 <bochecha> I do think he's talking with the beaker devs
16:59:23 <dgilmore> I really do not like when internal releng folks change things and then try to push it back into Fedora
16:59:34 <dgilmore> we really should develop the changes in Fedora
16:59:37 <bochecha> but writing a module means that beaker can be ported to using it, so that everybody is using the same format
16:59:44 <bochecha> agreed
17:00:00 <bochecha> we're probably going to be using it before them, though ;)
17:00:27 <bochecha> and I do think it makes sense, the current composeinfo format is a pretty horrible hack to shoehorn the data into an ini file
17:00:44 <dgilmore> sure. not syaing its perfect
17:00:55 <dgilmore> but we have a good platform to effect change
17:00:58 <dgilmore> lets do it right
17:01:10 <bochecha> (in fact, I wanted to talk to about doing that, before dmach even told me about it today)
17:02:02 <dgilmore> okay
17:02:03 <bochecha> so, doing it right means 1. opensourcing this module, 2. getting it into fedora, and 3. porting build_composeinfo to it, right?
17:02:11 <dgilmore> yep
17:02:18 <bochecha> well, I'll be doing that this week then :)
17:02:19 <dgilmore> making sure beaker devs are on board
17:02:23 <bochecha> of course
17:02:31 <dgilmore> they do import fedora into internal beaker
17:02:43 <dgilmore> its the only reason we make the .composeinfo file in fedora
17:02:55 <bochecha> well, I'm hoping to use it as the input for composedb
17:03:06 <bochecha> as it contains most of the stuff that I'd need in it
17:03:06 <dgilmore> :) that is fine
17:03:47 <dgilmore> 5
17:04:34 <dgilmore> bochecha: we can add whats missing
17:04:44 <bochecha> yes
17:04:56 <bochecha> and that's pretty much it for me, I'm still wondering if/how we could reuse internal work on this, and at the same time hacking on a quick poc for a composedb, in case we decide not to reuse it
17:05:22 * bochecha isn't sure how much he is allowed to say at this point, given that the internal work is not FOSS yet :x
17:05:27 <dgilmore> bochecha: okay, depends how modular thinsg are I guess
17:05:38 <bochecha> yeah
17:05:57 <dgilmore> so that things can be extended/swapped out as need be for different use cases
17:06:28 <dgilmore> I am hoping that we can get the composedb in place for F22
17:07:01 * bochecha gets cranking
17:07:55 <dgilmore> bochecha: having frontend/backend totally seperate is fine also
17:08:09 <bochecha> sure
17:09:30 <dgilmore> I have mentioned a few times i think having pkgdb/bodhi/koji/composedb as one integrated user interface would be awesome
17:09:43 <dgilmore> and get rid of the seperate versions
17:10:23 <bochecha> fedora community? :P
17:10:32 <dgilmore> ive said it before
17:10:50 <dgilmore> have it completely replace one thing
17:10:54 <dgilmore> then add the rest
17:11:44 * bochecha has to go soon
17:11:53 <bochecha> (and it seems we lost everyone else ^_^)
17:13:01 <dgilmore> yeah lets wrap up
17:13:16 <dgilmore> I just wante to get some more visability on the composedb
17:13:34 <dgilmore> will close in a minute unless someone has something else
17:13:42 <bochecha> right, forgot the link: moz-action:switchtab,https://fedoraproject.org/wiki/Compose_Database
17:13:55 <bochecha> huh? what's that moz-action stuff...
17:14:02 <bochecha> #link https://fedoraproject.org/wiki/Compose_Database
17:14:18 <dgilmore> no clue
17:14:24 <dgilmore> #endmeeting