15:36:13 #startmeeting RELENG (2014-08-25) 15:36:13 Meeting started Mon Aug 25 15:36:13 2014 UTC. The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:36:13 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:36:26 #meetingname releng 15:36:26 The meeting name has been set to 'releng' 15:36:30 #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson 15:36:30 Current chairs: bochecha dgilmore masta nirik pbrobinson sharkcz tyll 15:36:41 #topic init process 15:36:56 * masta is here 15:37:00 * bochecha is here 15:37:07 Hi all 15:37:07 * nirik is here 15:37:08 * sharkcz is here 15:37:47 #topic #5931 [Proposal] Move new branch and unretire requests to pkgdb2 15:37:57 https://fedorahosted.org/rel-eng/ticket/5931 15:38:04 pingu ping 15:38:28 hi 15:38:52 so I think on the pkgdb2 side it's ready 15:38:56 okay 15:39:17 I have been working on a fedmsg-pkgdb2branch that would take pkgdb2 fedmsg messages and call pkgdb2branch on pkgs01 15:39:22 for the packages concerned 15:39:53 the idea is then to provide a CLI to interact with pkgdb2's AdminAction, and close bugzilla tickets 15:40:11 the git management would then be taken care of via fedmsg 15:41:00 I guess that's it for me :) 15:41:16 cool. 15:41:41 #info < pingou> I have been working on a fedmsg-pkgdb2branch that would take pkgdb2 fedmsg messages and call pkgdb2branch on pkgs01 15:41:51 #info < pingou> the idea is then to provide a CLI to interact with pkgdb2's AdminAction, and close bugzilla tickets 15:41:58 nirik: does that answer your question about the fedmsg-pkgdb2branch project you had earlier? 15:42:33 just one further one... 15:42:35 #info < pingou> so I think on the pkgdb2 side it's ready 15:43:05 the part I'm not sure about is: how to deal with fedmsg message missed? 15:43:07 with other things we fedmsg enabled we have a catchall thing (ie, with fasClient we have a daily cron to make sure it's synced if it missed a fedmsg). Do we have any way to do that with branch requests? 15:43:15 exactly. ;) 15:43:27 so daily run of pkgdb2branch on all packages? 15:43:36 not sure how costly that is... 15:43:39 or bi-daily? 15:44:08 there's a lot of packages. 15:44:13 true there 15:44:14 I guess we could see how long that takes... 15:44:41 maybe not pkgdb2branch then but a dedicated one, sending a report 15:44:57 (and/or fixing) 15:45:32 yeah, could work. 15:45:38 we can investigate that out of meeting? 15:45:41 sure 15:45:50 * pingou has both scripts on his todo 15:45:55 cool. 15:46:16 I'm hoping to work on ansible migration for pkgs01.stg this week. 15:46:21 so we could have a good place to test. 15:46:23 cool :) 15:46:35 let me know if I can help w/ that 15:46:42 #info add daily cron script as fallback for missed fedmsg messages 15:46:47 (although bochecha is likely in a better position for this) 15:47:02 #info < nirik> I'm hoping to work on ansible migration for pkgs01.stg this week. 15:48:20 pingou, I can't help on this, it's my patches that nirik will review ;) 15:48:21 anything else here? 15:50:45 #topic #5959 Enable keep-alive connections for koji (primary and secondaries) 15:50:57 https://fedorahosted.org/rel-eng/ticket/5959 15:51:18 pbrobinson is in the UK and they have a public holiday 15:51:30 not sure if this was implemented and the results 15:51:46 #info deferred due to public holidy in the UK 15:51:46 #topic #5870 rawhide signing 15:51:54 #undo 15:51:54 Removing item from minutes: 15:51:55 #undo 15:51:55 Removing item from minutes: INFO by tyll at 15:51:46 : deferred due to public holidy in the UK 15:52:05 #info deferred due to public holidy in the UK 15:52:10 #topic #5870 rawhide signing 15:52:17 https://fedorahosted.org/rel-eng/ticket/5870 15:52:25 okay so this is kinda working 15:52:33 other than sigul keeps falling over 15:52:59 in particular the bridge is locking up frequently 15:53:06 yes, therefore training someone from Europ (e.g. pingou ) to restart it would be nice 15:53:31 I will talk with nirik to find someone extra 15:53:44 also maybe some automatic monitoring would be nice 15:53:53 the passphrase to unlock the nss database is shared and not per user 15:53:55 we're two -main in Europe, anyone of us could do :) 15:54:02 e.g. ngios alerts when the bridges are broken 15:54:03 (if that helps of course) 15:54:15 we could possibly setup a nagios alert yeah... 15:54:36 it seems like secondary falls over more than primary? or about the same? 15:55:06 tyll: how close are we to all signed 21/22? 15:55:25 secondary is used more, so I notice it easier 15:55:40 secondary is signing 3 tiems the rpms 15:55:48 but i think its about equal 15:56:27 its the bridge locking up, we should try get some of mitr's time and see if we can't fix it 15:56:36 yeah. that would sure be nice. ;) 15:57:00 nirik: Currently the script is running non stop, but I did not check how many RPMs are still unsigned - if anything is currently unsigned, there is a bug (at least in primary, not sure if someone signed everything from the mass rebuilds in the secondaries) 15:57:36 #info sigul breaks regularly, dgilmore and nirik will find additional admins that can restart it when they are not available 15:57:37 ok, entering freeze for 21 we need to make sure everything is signed. 15:57:46 I tried last night on ppc, but something was wrong with sigul 15:57:57 and then we can change the autosigner to concentrate on rawhide I guess. 15:58:24 tyll: I know some things are unsigned. 15:58:48 I had to add --nogpgcheck when installing software onto the new laptop this weekend 15:59:11 http://koji.fedoraproject.org/mash/branched-20140825/logs/mash.log 15:59:21 dgilmore: since there is no gating, there might also be timing issues, especially when sigul is down 15:59:22 ( that lists the things without the right key) 15:59:47 tyll: right 16:00:37 The "None" epoch in mash#s log makes it not nice to grep in the autosigner logs 16:02:10 yeah its kinda ugly 16:02:26 that we expose epochs in some places and not others is ugly 16:02:36 * nirik nods. 16:03:19 nirik: i need to flush the squid caches again 16:03:32 probably change pungify 16:03:38 I will take a look to check why the packages from the mash log are not signed 16:03:39 thats a differnt topic though 16:03:59 yeah, lets get f21 frozen, then we can concentrate the autosigner on rawhide again. 16:04:03 and gating, etc 16:05:31 yep 16:05:39 we get auto gating with bodhi 16:05:58 right, f21 becomes not this tickets issue once we freeze. ;) 16:06:30 btw. Do we now have enough space on the secondary kojis for the signed RPMs? 16:07:15 not 100% sure 16:07:48 vtap-fedora-nfs01.storage.phx2.redhat.com:/vol/fedora_arm/data 5.0T 4.2T 761G 85% /mnt/koji 16:07:58 ppc has 1.9T free 16:08:14 so yeah, I have been working with netapp folks to get more space free on primary... 16:08:20 vtap-fedora-nfs01.storage.phx2.redhat.com:/vol/fedora_s390/data 6.5T 5.8T 666G 90% /mnt/koji 16:08:30 I removed old mashes and reduced the number of snapshots we keep. 16:08:31 s390 has 600G free, but it gets expanded +0.5T when free space is below 200G 16:08:48 sharkcz: did we have any snapshots on it? 16:09:04 nirik: on s390? nope, it's disabled 16:09:25 yeah, I see that now, ok. 16:09:26 ppc has snapshots enabled 16:09:38 if I see correctly 16:09:51 we may get migrated to a new cmode filer before too long. I am talking with storage folks about that. 16:09:58 any outage should be hopefully pretty short. 16:13:08 anything else here? 16:13:33 not from me 16:13:41 #topic #5914 Move fedmsg based blocking service to Fedora Infrastructure 16:13:46 https://fedorahosted.org/rel-eng/ticket/5914 16:13:56 There is nothing new here due to lack of time 16:14:00 tyll: I need to get you a user cert right 16:14:20 dgilmore: yes, eventually 16:14:38 okay, let us know what you need 16:14:45 sadly i can not make time 16:15:17 #topic Secondary Architectures updates 16:15:18 #topic Secondary Architectures update - ppc 16:15:29 sharkcz: so i know your having issues with pungi 16:15:35 anything else going on? 16:15:42 you're 16:16:03 seems they were cause by the mirrorlist (or outdated mirrors), looks OK with pointers to koji/mash 16:16:19 sharkcz: okay, maybe some latent yum bugs 16:16:36 well, almost ok, just grub2 is not included by lorax 16:16:41 I was having some problems with more recent versions of the sigulsing_unsigned.py on ppc recently, had to checkout an older version. Hoping to bisect the issue later today 16:16:57 sharkcz: that would be a lorax template update then? 16:17:22 don't thiok so, as the standalone lorax run did it correctly 16:17:27 masta: that seems odd. im using the current code on both arm and primary 16:17:45 sharkcz: maybe a pungi bug then 16:17:56 dgilmore: the error text complained about kojihelper 16:18:07 dgilmore: have a log from pungi, so we can look 16:18:15 we should get it fixed asap so we do not need to try push in an updated pungi 16:18:28 yes 16:19:16 sharkcz: there was a self introduction on the ppc list last week 16:19:20 no one has responded 16:19:49 ah, right, I forgot and no one else probably saw it :-) 16:19:58 :-( 16:20:23 oh yes, I think somebody nominated me to be a package sponsor... I was kinda amused by that. 16:20:24 it's the guy who is/was trying to revive ppc 16:20:44 dgilmore: is this the yaboot email? 16:21:01 masta: well thats one of the things he wants to work on 16:21:06 but we really want it dead 16:21:19 ack 16:22:05 at this point 32 bit ppc would be massive massive amounts of work to revive 16:22:06 yeah I was wanting some guidance on that, firstly I don't know how I got pulled into sponsoring yaboot, or if we even want yaboot.... 16:22:30 yaboot is the least of the issues 16:22:42 * nirik personally thinks a $35 arm board is more functional than a 32bit ppc from ages past, but to each their own. 16:22:43 dgilmore: yes, maybe as terciary arch 16:22:43 yaboot runs on a system I have here, but I was planning to upgrade that to grub2 16:22:57 sharkcz: I guess so 16:23:05 but not our fight ... 16:23:28 sharkcz: I think we should go to FESCo and get them to say how they think it should work 16:24:17 dgilmore: would make sense probably 16:25:40 I do not want to add it back to the ppc hub 16:25:46 it will need to be seperate 16:25:58 but how we handle it I think FESCo should say 16:26:00 * sharkcz agrees 16:26:37 #action dgilmore to open ticket with FESCo on how we should handle 32 bit ppc 16:27:01 I don't think one man hobby project can manage it, it's more a fulltime job 16:27:18 as a full "secondary" arch 16:27:20 right 16:27:50 yes 16:29:52 otherwise no major issues on ppc afaik, except mariadb and its tests-suite 16:29:59 okay 16:30:01 :( 16:30:10 #topic Secondary Architectures update - s390 16:30:17 sharkcz: you're up agian 16:30:20 again 16:31:10 yes :-) the glibc ABI is 60+% done, will have to concentrate on the perl rebootstrap, should be easier now as I can use the old rpms (pre glibc 2.19) 16:31:21 glibc ABI rebuild 16:31:59 it also means that currently the f21 is broken for users 16:32:47 I guess it should be fixed/rebuilt next week 16:33:12 I've learned the hard way 6 month ago :-) 16:34:13 also no composes yet, so we don't know is the tools still work, but hopefully the ydo 16:34:24 s/is/if/ 16:35:04 EOM 16:36:04 okay thanks 16:36:12 #topic Secondary Architectures update - arm 16:36:27 with pbrobinson being away there is no update from him 16:37:13 Planning to setup a box in phx early this week so we can get composes moving. I have done a TC compose from an external box and installed it. not expecting any big surprisses 16:37:29 afaik the boot.iso and install dvd will not be bootable 16:37:47 there is no support in mkisofs afaik to make bootable aarch64 media 16:37:58 #topic Open Floor 16:38:06 anyone have anything they want to discuss? 16:38:27 I want to learn about koji-shadow, could somebody show me? 16:38:49 either ppc64 or aarch64 ? 16:40:14 masta: sure, you will need to work with the folks currently running it, which should be pbrobinson all both ppc and aarch64 16:40:36 masta: see http://ppc.koji.fedoraproject.org/shadow/ - it has logs, config and script how it is run 16:40:36 okay 16:41:29 #info fesco ticket for 32 bit ppc https://fedorahosted.org/fesco/ticket/1336 16:41:39 masta: also http://sharkcz.livejournal.com/11410.html 16:41:44 thanks, koji-shadow is a gap in my knowledge of how things work on secondaries. 16:42:30 masta: and read the source code :-) 16:43:37 sharkcz: yes indeed! 16:44:00 who is the maintainer of ks? 16:44:05 me 16:44:18 with many patches from the regular users 16:44:27 like /me :-) 16:45:46 dgilmore: did you see https://fedoraproject.org/wiki/Architectures/KojiShadowNG (I sent the link to releng list ~week ago) 16:45:53 sharkcz: I did 16:46:02 not had a chance to follow up on it yet 16:46:07 ok, it's just a start 16:46:13 last week was all about trying to get a TC done 16:46:19 is that going to look at using fedmsg? 16:46:52 like integrating koji-stalk (written by dwa)? probably yes 16:47:18 it would be nicer for our fedmsg bus is builds on secondaries just happened around the time they did on primary. 16:47:21 nirik: likely we will integrate koji-stalk and add a daemon mode 16:47:39 right now there's a giant pile of them that happen at once when someone runs koji shadow 16:47:40 nirik: problem is the something gets broken 16:47:49 nirik: it would be nicer for many things, practically its not always possibe 16:47:58 sure, agreed. would be nice to try. ;) 16:48:02 and it happens all the time :-( 16:48:31 I do want to have secondary arches more integrated into everything and make it more visable 16:48:40 so breakages are more noticed 16:48:44 * sharkcz agrees 16:50:41 that is a longer term roadmap thing 16:50:47 but something to keep in mind 16:51:53 same as allow multiple people to work on a arch, etc 16:52:38 yeah 16:52:47 needs to be easier 16:52:53 and have full composes all teh time 16:53:24 yeah, would be nice :-) good to know we have still plenty of work ahead :-) 16:53:41 pretty sure we have years of work ahead 16:53:48 yep 16:53:51 for multiple people 16:54:32 bochecha: ping, want to give an update of the composedb? 16:55:01 dgilmore, ah, I talked about it with dmach today 16:55:32 one thing he started working on is a module to parse/make composeinfo files 16:56:05 would be nice to look into using that (in releng/scripts/build_composeinfo for example), so that we share the same format as rhel 16:56:15 and then these composeinfo files can be used as the input for composedb 16:56:27 bochecha: sure 16:56:54 only beaker uses composeinfo files currently, right? 16:57:19 (because dmach is changing the format, so we'll need to make sure users are ported over) 16:58:00 * sharkcz needs to leave ... 16:58:16 sharkcz: cheers 16:58:28 bochecha: he shouldn't change the format 16:58:41 at least not without talking to the beaker devs 16:58:49 as they are the only consumers i know of 16:58:58 I do think he's talking with the beaker devs 16:59:23 I really do not like when internal releng folks change things and then try to push it back into Fedora 16:59:34 we really should develop the changes in Fedora 16:59:37 but writing a module means that beaker can be ported to using it, so that everybody is using the same format 16:59:44 agreed 17:00:00 we're probably going to be using it before them, though ;) 17:00:27 and I do think it makes sense, the current composeinfo format is a pretty horrible hack to shoehorn the data into an ini file 17:00:44 sure. not syaing its perfect 17:00:55 but we have a good platform to effect change 17:00:58 lets do it right 17:01:10 (in fact, I wanted to talk to about doing that, before dmach even told me about it today) 17:02:02 okay 17:02:03 so, doing it right means 1. opensourcing this module, 2. getting it into fedora, and 3. porting build_composeinfo to it, right? 17:02:11 yep 17:02:18 well, I'll be doing that this week then :) 17:02:19 making sure beaker devs are on board 17:02:23 of course 17:02:31 they do import fedora into internal beaker 17:02:43 its the only reason we make the .composeinfo file in fedora 17:02:55 well, I'm hoping to use it as the input for composedb 17:03:06 as it contains most of the stuff that I'd need in it 17:03:06 :) that is fine 17:03:47 5 17:04:34 bochecha: we can add whats missing 17:04:44 yes 17:04:56 and that's pretty much it for me, I'm still wondering if/how we could reuse internal work on this, and at the same time hacking on a quick poc for a composedb, in case we decide not to reuse it 17:05:22 * bochecha isn't sure how much he is allowed to say at this point, given that the internal work is not FOSS yet :x 17:05:27 bochecha: okay, depends how modular thinsg are I guess 17:05:38 yeah 17:05:57 so that things can be extended/swapped out as need be for different use cases 17:06:28 I am hoping that we can get the composedb in place for F22 17:07:01 * bochecha gets cranking 17:07:55 bochecha: having frontend/backend totally seperate is fine also 17:08:09 sure 17:09:30 I have mentioned a few times i think having pkgdb/bodhi/koji/composedb as one integrated user interface would be awesome 17:09:43 and get rid of the seperate versions 17:10:23 fedora community? :P 17:10:32 ive said it before 17:10:50 have it completely replace one thing 17:10:54 then add the rest 17:11:44 * bochecha has to go soon 17:11:53 (and it seems we lost everyone else ^_^) 17:13:01 yeah lets wrap up 17:13:16 I just wante to get some more visability on the composedb 17:13:34 will close in a minute unless someone has something else 17:13:42 right, forgot the link: moz-action:switchtab,https://fedoraproject.org/wiki/Compose_Database 17:13:55 huh? what's that moz-action stuff... 17:14:02 #link https://fedoraproject.org/wiki/Compose_Database 17:14:18 no clue 17:14:24 #endmeeting