16:40:18 #startmeeting releng 16:40:18 Meeting started Mon Mar 17 16:40:18 2014 UTC. The chair is masta. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:40:18 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:40:18 #meetingname releng 16:40:18 The meeting name has been set to 'releng' 16:40:18 #topic === Releng Roll Call === 16:40:30 Howdy folks 16:40:54 hi 16:40:57 Hey 16:41:00 hi 16:41:02 * nirik will be around in a few. 16:41:37 #chair dgilmore dwa nirik tyll sharkcz bochecha 16:41:37 Current chairs: bochecha dgilmore dwa masta nirik sharkcz tyll 16:42:40 .chair bochecha 16:42:40 bochecha is seated in a chair with a nice view of a placid lake, unsuspecting that another chair is about to be slammed into them. 16:42:46 :-p 16:43:14 « nice view of a placid lake »... if only :] 16:43:53 #topic === releng Announcements === 16:44:25 anything? 16:44:44 not sure that counts as a releng announcement, but there's the bodhi2 / taskotron hackfest coming 16:44:47 https://fedoraproject.org/wiki/FAD_Bodhi2_Taskotron_2014 16:44:52 ok, I am here now. ;) 16:44:59 bodhi is slightly related to what releng does, so... 16:45:08 yeah, if any releng folks are wanting to help with that or attend, please add your info 16:45:32 cool 16:46:10 lets get rolling 16:46:17 #topic === releng Tickets === 16:46:17 #link https://fedorahosted.org/rel-eng/report/10 16:46:54 https://fedorahosted.org/rel-eng/ticket/5854 16:47:24 looks like I own that one, and I believe we agreed previously to not patch the script.... so I might close 16:48:01 #5362 Use XZ for repository metadata compression 16:48:09 I believe we discussed when and where to run it, but not whether we patch the script 16:48:28 #topic https://fedorahosted.org/rel-eng/ticket/5362 16:49:12 I'm ok with patching it even if we don't run it... it makes it nicer to use in the event someone wanted to run it one off. 16:49:28 As far as I know we have identified how to solve the xz compression, just need to actually do it. 16:49:40 * nirik nods 16:50:02 * masta will go ahead and patch (ticket 5854) 16:50:22 #action masta will go ahead and patch (ticket 5854) 16:50:35 for the record, the solution is to update the masher script, and configuration knobs, etc. 16:51:24 dgilmore wanted to address 5362 16:51:28 tyll: thanks 16:52:21 it would be nice to get that finally going in rawhide well before we have any milestones. 16:52:21 tyll: yeah, we can help him out by writing the patch, he would likely be happy to apply what we send (but I won't speak for him) 16:53:25 ok, so anybody want to volunteer? 16:53:51 I'll try to get to it, but probably not going to formally action myself =) 16:54:05 ok, so moving on.... 16:54:10 I guess I could get the mash part done, but I do not know anything about the createrepo API 16:54:34 #action till will look into preparing a patch for mash 16:54:36 I think it's just mash needing to pass the option to createrepo. 16:54:51 tyll: what nirik said 16:55:03 yes, this looks easy, but there is also a pungi part 16:55:52 I wouldn't mind looking into it, but I have quite a few pending things still, and my availability will change drastically in April/May, so... 16:56:08 hey al 16:56:10 all 16:56:11 #topic #5859 Releng improvements ideas 2014 (https://fedorahosted.org/rel-eng/ticket/5859) 16:56:13 hi dgilmore 16:56:20 Hi dgilmore 16:56:22 * masta waves to dgilmore 16:56:33 hey :) 16:57:32 lol ... looks like we are having a meeting right now that contradicts this ticket 16:57:46 masta: then the ticket worked? ;) 16:58:20 i think this can be closed 16:58:37 bochecha: I guess so... I certainly do not disagree with the ticket... if anybody wants to chair a meeting... it's really not so hard. 16:58:44 need to work on doing better with communications and being organise 16:58:47 organised 16:59:13 dgilmore: you planned to collect feedback about improvements in releng and e.g. write a e-mail - this is also why I opened the ticket 16:59:14 I'd be happy to help mentor on fedora meetings or whatever, the hard part is the follow-up (sending meetign minutes, etc) 17:00:13 tyll: yeah. we have lots of improvements needed 17:00:28 we can leave it open and document things 17:00:45 add the meeting keyword back when there is new things to discuss 17:01:04 sounds good 17:01:45 yeah, sounds good 17:02:16 does anybody have a topic not covered by a ticket, or move to open floor? 17:02:39 I have a ticket, to which I didn't add the meeting keyword 17:02:49 * bochecha just realizes that's the process 17:03:00 bochecha: ;-) 17:03:41 ok, so... 17:03:48 #topic 5846 17:04:10 I just redid the change today, as per our discussion from 2 weeks ago: https://fedorahosted.org/rel-eng/ticket/5846#comment:16 17:04:30 I didn't change anything in pyrpkg, as you wanted dgilmore :) 17:04:54 review would be nice, I tested it on my lookaside and it's working 17:06:45 * bochecha wonders whether he killed the meeting :x 17:06:54 bochecha: in https://lists.fedoraproject.org/pipermail/infrastructure/2014-March/014188.html in the notifications to stderr the actual hash algorithm should be noted, not only CHECKSUM= - maybe add HASHALGO=md5 or someting similar 17:07:24 bochecha: not dead, I was reading the chain of text related to the ticket =) 17:07:31 I thought I was writing MD5SUM or SHA512SUM? 17:07:32 bochecha: will review again 17:07:54 tyll: right, the next patch changes that line again 17:08:03 tyll: and specifies the hash type 17:08:33 first patch was just renaming all the hardcoded "md5" everywhere, and the second one adding support for other hashes 17:08:42 I could have probably changed these log lines only once, though 17:09:21 I see 17:09:54 the hashdir should probably also contain the hash algorithm 17:10:28 tyll: that sounds like a good idea 17:10:33 tyll: possibly 17:11:02 something like package/sourcename/hashtype:hash/sourcename ? 17:12:37 :'s in path names are anoything 17:12:39 anoying 17:12:49 you cannot scp them easily for example 17:13:26 oh, I meant /, not : (same key on the French keyboard, I just failed to hit shift) 17:13:43 ok. ;) 17:14:33 I would be okay with that 17:15:03 ok, I'll send a revised patch tomorrow that would do that, then 17:15:07 bochecha: do you have a fedorapeople repo with the changes? Might be easier to follow 17:15:11 it's good to be imperative/explicit with regards to the hash also being used (or previously used), nobody likes to guess 17:15:14 just one should need to be changed 17:15:18 tyll: I don't, but I can do that 17:16:12 #action bochecha will add the hash type to the path name on the lookaside cache 17:16:39 bochecha: cool 17:16:59 now we move to our favorite thing, Open Floor! 17:17:03 another possible improvement would be to calculate several hash values at once 17:17:18 then you can store it at the proper paths for all enabled hash algos 17:17:31 tyll: and hardlink it all? 17:17:44 bochecha: yes 17:18:08 that's certainly doable, I'd have to replace the « if/elif » for the hashes by just a series of « if » 17:18:49 #topic Open floor 17:18:56 * nirik would prefer just moving to the 'better' hash, not providing the old one too 17:19:38 nirik: id like to convert the existing content over to the stronger hash. 17:19:45 but id rather just the newer 17:19:55 dgilmore: that's the other part, I was planning to look at that during the week 17:20:00 making sure we dont break existing fedpkg sources 17:20:01 shouldn't be too hard 17:20:13 bochecha: wont be hard 17:20:25 but will need to rehash every tarball/file 17:20:26 sure, but if people still want md5, they can do it themselves, we shouldn't keep providing it. ;) 17:20:40 for a transitional time I'd be fine with doing both, temporarily... do think hard switching to the stronger is ideal. 17:20:45 it might make it easier to compare our sources with others 17:21:01 i dont think there will be a transitional time 17:21:02 nirik: « fedpkg sources » will try to verify the md5sum, if that's what is in the « sources » file 17:21:17 I think we will require a new fedpkg to upload 17:21:27 for this it would also be good to have sha256 because it is more common than sha512 17:21:43 bochecha: sure, thats fine for already existing md5. I just don't think we should also make it moving forward for new uploads. Seems pointless to me. 17:21:54 nirik: ah, yes 17:22:04 dgilmore: will you also want to commit to all repositories a new version of the « sources » file, to move them all to a stronger hash? 17:22:30 tyll: I'm not set on the hash, I just went for the biggest provided by Python's hashlib, but it would be easy to change to something else, add more,... 17:22:40 bochecha: no 17:22:55 dgilmore: ok, so it's good I added a fallback to md5 for fedpkg :) 17:22:58 bochecha: I think it is good to support multiple hash values at once 17:23:29 sha256 would be great, I'd rather not have to see 128 characters in an 80 char terminal 17:23:39 tyll: multiple, without counting the one we're moving away from? 17:24:58 bochecha: yes, for example storing sha256 and sha512 hashes would be nice - but additionally md5 as well, but I would not insist. It just makes it easier to compare sources with other projects 17:26:57 * nirik isn't sure we can meet everyone on the planets needs for checksum type. 17:27:35 it could be useful if upstream only publishes md5 or sh256 summed checksums for their tarballs to verify that what we have matches upstream 17:28:03 there are certainly project still (only) using md5 17:28:03 I suppose, but the main time that would be checked is by the packager... _before_ they upload 17:28:08 but the effort to run sha25sum or md5sum is not hard to manually do so 17:28:35 right 17:28:39 but you have to download the actual files then 17:28:52 true 17:29:10 hardlinking is not too expensive 17:29:13 I think it would be great for distros to add gravity to sha256, over time people will gravitate to sha256. 17:29:38 computing md5 sha1 sha256 sha512 is much more expensive server side 17:29:42 well, hardlinking doesn't even need to be done, the netapp is doing that on the backend. 17:29:56 true 17:30:02 folks can checksum however they want, but WE will use the best checksum regardless (or in spite of) 17:30:07 but it adds complexity to have all those... 17:30:22 for 99% of the time no use 17:30:28 nirik: indeed 17:31:16 so how about this: fedpkg talks to the lookaside using a strong hash (right now I've implemented it with sha512, we could downgrade to sha256), and then the lookaside takes care of creating all those hashes/folders/hardlinks if it can verify the upload? 17:32:28 I'd personally prefer just using sha512 and leaving it at that... but if everyone else wants the compat hashes, I'm happy to move on. 17:32:47 nirik: I'm happy with just one too, just trying to get something decided :) 17:33:03 id rather just 1 17:33:34 +1 sha256 (64 chars fit in 80 char wide terminals, is strong enough) 17:33:57 id rather sha512 17:35:15 the hashes can be calculated in parallel so it does not add load to i/o only to the CPU, but then hash values are usually fast to calculate 17:36:25 pkgs.fp.o has 4 core 17:36:26 s 17:36:42 some sources are... large 17:36:59 ~1.5T for texlive 17:37:00 depends on the size of input, but yea.. they are fast on small input 17:37:01 but yeah, checksums are quick 17:37:08 nirik: you need to read them only once and can feed them at the same time too all hashlib instances 17:37:15 there is a bunch that are hundreds of Mb 17:38:26 checksums should less than O(2n) complex... but we should be mindful to use the least expensive, and good enough checksum... as long as there are no collisions I'm good with the least expensive 17:39:31 maybe we discuss this on list? 17:39:38 sure 17:39:41 well, there's not much difference between 256 and 512 IMHO on the compute site 17:39:43 see if we can get a consensus 17:39:43 side 17:40:04 * bochecha is happy to implement whatever ends up being the consensus 17:40:10 * nirik just ran some test runs on his sourcecheck vm... it's really quick. The real storage is likely slower, but still. 17:40:22 wc -l sources 17:40:22 6157 sources 17:40:36 texlive has a lot of tarballs 17:40:37 I won't have access to a lookaside cache any more after April 4th, though, so I hope it can be done by then :) 17:40:57 dgilmore: about 10 seconds here locally. ;) time sha512sum *.xz -> real 0m9.235s 17:41:04 they are all small. 17:43:28 given the equivalence... I'd still opt for the smaller result string. 17:43:46 example of calculating multiple hashes at once: http://ur1.ca/gve97 17:43:59 * nirik would prefer to just change it once now rather than again in a few years. 17:44:27 masta: can you compare 256's by eyeballing? ;) usually when you are checking you would use -c no? 17:44:56 It might be a good idea to change to SHA-3 anyhow in the future 17:45:51 tyll: python's hashlib doesn't support that, though, so it would be less trivial, but yeah, we'll likely end up changing regularly as older hashes get broken and newer ones get created 17:46:23 bochecha: sha-3 is not yet completely final, but I am sure it will be suppported then 17:46:30 nirik: sometimes I look at the checksums... but your point is accepted... is mostly automatic -c use 17:47:02 can someone please volunteer to start a thread on teh releng list and we can discuss there 17:47:30 dgilmore: I've sent the patches, I guess I can do that 17:47:39 or we could just use the thread of the patches, I guess 17:47:55 bochecha: thats fine too 17:48:00 so... 17:48:25 anything for open floor? 17:49:01 I setup a ticket with anaconda folks to remove missing packages from lorax templates, they agreed to remove firstaidekit, but not *-dracut-whateveritwas 17:49:14 okay 17:49:22 should improve those pesky errors nirik mentioned a few meetings ago 17:49:34 (pungi logs) 17:49:37 ok 17:49:42 * nirik hasn't looked in a while... 17:49:48 anyone is welcome to tho. :) 17:50:10 I want to look at the fstrim thing next 17:50:15 not really an error its a warning 17:50:26 fstrim thing? 17:50:53 dgilmore: yeah, it was suggested that we could call fstrim on live media at least in the process to make them more uniform sized. 17:51:24 nirik: okay, I think it doesnt work 17:51:40 nirik: the cloud images fill the disk with zeros 17:52:16 echo "Zeroing out empty space." 17:52:16 # This forces the filesystem to reclaim space from deleted files 17:52:16 dd bs=1M if=/dev/zero of=/var/tmp/zeros || : 17:52:16 rm -f /var/tmp/zeros 17:52:16 echo "(Don't worry -- that out-of-space error was expected.)" 17:52:27 well, we were seeing live images vary with few changes. 17:52:48 it would need to be done before the squashing 17:52:54 nirik: right, but from memory using fstrim on the appliance images didnt help 17:53:04 which is why they do what i pasted above 17:53:31 there are better tools for this, e.g. zerofree or some sparsify tool from libvirt something 17:53:34 which is likely what needs doing 17:54:01 well I'll give it a whirl and if we establish it's not helpfull... I can chalk it up to A-for-effort and move on 17:54:01 could be. 17:54:17 it would be nice to have them stay the same size from compose to compose. 17:54:52 nirik: it would and its not a big issue except when they are really close to the size limit 17:55:09 right. 17:56:47 hey one thing I noticed recently, systemd can now mount the system partitions without /etc/fstab, using gpt partition types. Is that something we want to implement eventually? 17:57:25 masta: not up to us 17:57:28 implement where? by who? 17:57:38 thats anaconda land 17:57:49 yeah, anaconda changes for new installs... 17:57:57 nirik: it would be a neat thing to add into appliance tool for cloud images, maybe as a starting point 17:58:20 masta: cloud images in fedora 21 are supposed to use anaconda 17:58:39 you guys are right of course... I'm just rambling about that weird new systemd thing.... 17:58:49 waiting in a new koji release to move to oz and image factory for image creation 17:58:55 dgilmore: ah! 17:59:10 but arm will be appliance-creator still 17:59:17 at least for now 18:00:07 we are over our scheduled stop time 18:00:13 yeah 18:00:21 any objections to ending the meeting ? 18:00:27 I think we should call it a wrap 18:00:34 cool 18:00:43 #endmeeting