17:00:13 #startmeeting F25-Final-Go/No-Go-meeting 17:00:13 Meeting started Thu Nov 10 17:00:13 2016 UTC. The chair is jkurik. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:00:13 Useful Commands: #action #agreed #halp #info #idea #link #topic. 17:00:13 The meeting name has been set to 'f25-final-go/no-go-meeting' 17:00:15 #meetingtopic F25 Final Go/No-Go meeting 17:00:16 #topic Roll Call 17:00:18 .hello jkurik 17:00:19 jkurik: jkurik 'Jan Kurik' 17:00:38 nirik, adamw, mboddu: are you around ? 17:00:47 .hello mohanboddu 17:00:48 mboddu: mohanboddu 'Mohan Boddu' 17:00:51 yes. I am here. 17:01:00 * nirik thought this was in an hour. ;) pesky DST 17:01:06 hi 17:01:47 * pschindl is here 17:01:53 so, we have FESCo and RelEng representative and we need someone from QA as well 17:02:01 * mattdm is here watching 17:02:11 .hello sgallagh 17:02:12 sgallagh: sgallagh 'Stephen Gallagher' 17:02:50 pschindl: Hi, can you represent QA, till adamw appears ? 17:03:04 hola 17:03:05 i'm here, i'm here 17:03:08 .hello adamwill 17:03:09 adamw: adamwill 'Adam Williamson' 17:03:10 ok, ok 17:03:21 so, hello everybody 17:03:29 #topic Purpose of this meeting 17:03:35 #info Purpose of this meeting is to check whether or not F25 Final is ready for shipment, according to the release criteria. 17:03:41 #info This is determined in a few ways: 17:03:46 #info No remaining blocker bugs 17:03:51 #info Test matrices are fully completed 17:03:57 #info Final RC compose is available 17:04:06 #topic Current status 17:04:10 .hello pfrields 17:04:11 stickster: pfrields 'Paul W. Frields' 17:04:12 sorry to be late 17:04:30 hi stickster 17:04:34 * kparal is also here 17:04:40 #info The F25 Final RC is available: 17:04:42 #link https://fedorahosted.org/rel-eng/ticket/6489 17:04:43 #link http://dl.fedoraproject.org/pub/alt/stage/25_RC-1.2 17:04:56 #info Test results for F25 Final are available: 17:04:58 #link https://fedoraproject.org/wiki/Test_Results:Fedora_25_RC_1.2_Summary 17:04:59 #link https://fedoraproject.org/wiki/Test_Results:Fedora_25_RC_1.2_Installation 17:05:01 #link https://fedoraproject.org/wiki/Test_Results:Fedora_25_RC_1.2_Base 17:05:02 #link https://fedoraproject.org/wiki/Test_Results:Fedora_25_RC_1.2_Server 17:05:04 #link https://fedoraproject.org/wiki/Test_Results:Fedora_25_RC_1.2_Cloud 17:05:05 #link https://fedoraproject.org/wiki/Test_Results:Fedora_25_RC_1.2_Desktop 17:05:07 #link https://fedoraproject.org/wiki/Test_Results:Fedora_25_RC_1.2_Security_Lab 17:05:09 There are still some blockers 17:05:16 #info We have 9 accepted blockers and 3 proposed blockers 17:05:23 #link https://qa.fedoraproject.org/blockerbugs/milestone/25/final/buglist 17:05:34 Let's start with Mini-blocker review 17:05:41 adamw: may I ask you please to chair the mini-blocker review ? 17:05:45 sure 17:05:51 #topic Mini-Blocker Review 17:06:00 adamw: thanks 17:06:00 #info starting with proposed blockers 17:06:02 did you chair me? 17:06:19 * coremodule is here. 17:06:26 ah, sorry .... 17:06:38 #chair adamw mboddu nirik 17:06:38 Current chairs: adamw jkurik mboddu nirik 17:06:41 #topic (1392885) External display doesn't disconnect after undocking with kernel 4.8.6 17:06:41 #link https://bugzilla.redhat.com/show_bug.cgi?id=1392885 17:06:42 #info Proposed Blocker, kernel, NEW 17:06:48 yeah, no. -1 17:07:31 -1 here too... but FYI jforbes has been working on this and is going to try to have an update on or near 0-day 17:07:41 I thought this was largely covered in the BZ, but -1 blocker 17:07:47 -1 17:07:52 -1 17:07:53 -1 pile-on! 17:07:54 -1 blocker here too 17:07:57 -1 blocker 17:07:58 ha 17:08:19 well, seems like a clear blocker 17:08:22 :P 17:08:41 proposed #agreed 1392885 - RejectedBlocker (Final) - this clearly doesn't violate any of the criteria and can reasonably be fixed with an update 17:08:46 ack 17:08:50 ack 17:08:50 (it's a bit sad that we introduced this by pulling an FE, but ah well) 17:08:53 ack 17:08:58 ack 17:09:00 #agreed 1392885 - RejectedBlocker (Final) - this clearly doesn't violate any of the criteria and can reasonably be fixed with an update 17:09:11 #topic (1393846) no Fedora boot menu in Mac OS X dual boot install 17:09:11 #link https://bugzilla.redhat.com/show_bug.cgi?id=1393846 17:09:11 #info Proposed Blocker, mactel-boot, NEW 17:09:23 this one's a bit trickier. we have precisely one data point so far, but it's a very clear violation of the criterion. 17:10:26 i can't really vote -1 on this under the current criteria unless someone tries on a newer mac and it works. but given https://www.happyassassin.net/testcase_stats/25/Installation/QA_Testcase_dualboot_with_OSX_Miscellaneous.html , i am entirely open to dropping the criterion on the basis of failure to test. 17:11:02 * nirik reads the logs 17:11:38 looks like /boot was read-only? so it couldn't copy grub configs 17:11:39 As I mentioned in the BZ, I'd prefer to drop this criterion for the future. 17:12:02 I've never understood why we would block on failures to run on hardware that actively tries to make it difficult to run Linux on. 17:12:10 sgallagh: i put a link in to the original thread. 17:12:11 yeah, for me it makes sense to drop the criterion as well 17:12:14 * satellit install with usb to external USB HD as a substitute? 17:12:15 nirik: I thought it was somehow related to the mactel-boot failure. but it's possible that the cause was different 17:12:18 sgallagh: because users 17:12:19 er, or i thought i did. 17:12:25 oh damn. never hit enter. 17:12:36 https://lists.fedoraproject.org/pipermail/test/2014-August/122496.html 17:13:35 kparal: yeah, that's how i'd read it. 17:14:00 though, hmm, nirik is right, all those rsync errors come first. 17:14:03 not sure what the crap that is. 17:14:21 oh 17:14:24 HFS+ driver issue? dirty fs? 17:14:27 * stickster has a mid-2015 MBP, is that new enough? 17:14:31 it seems to be part of the actual install process... 17:14:34 stickster: yese 17:15:06 OK, I'll see if I can reproduce. I'm in this meeting upstairs, *on* that MBP while I shove food in my mouth right now. But once done, I can try it 17:15:07 oh. huh 17:15:10 that's how we install live images now. 17:15:16 who knew. well, someone, presumably. 17:15:21 Oh wait, do I have to install to check this? 17:15:26 oh, /boot is ext4 17:15:44 stickster: since it's about dual booting, i'm gonna say yes. 17:15:46 um, rsync has been in use for live install for years 17:15:55 if that's what you mean 17:15:55 well, i either didn't know or forgot. 17:15:59 those might be perfectly normal errors i guess 17:16:00 whoops, no can do then. This machine has to stay unmolested, it's my Pro Tools right :-( 17:16:03 rig 17:16:09 * adamw checks a normal live install 17:16:42 hmm, no. so it does seem like there might be something odd going on there. 17:16:43 huh... it makes /boot on sda5, then makes sda5 a lvmpv? 17:17:52 /boot/efi is not part of /boot 17:18:05 it's a mount of the pre-existing mac-ish ESP 17:18:08 right 17:18:19 13:07:33,429 DEBUG blivet: fixing size of existing 201.55 GiB partition sda2 (29) with existing macefi filesystem mounted at /boot/efi 17:18:31 so seems like that got mounted RO for some reason, which is obviously gonna bust stuff 17:18:38 sda1-3 existed from macos installation, sda4 and 5 were created by anaconda I believe 17:18:58 kparal: it did 13:09:18,704 INFO program: Running... mount -t hfsplus -o defaults /dev/sda2 /mnt/sysimage/boot/efi 17:18:58 13:09:18,800 DEBUG program: Return code: 0 17:19:14 adamw: the behaviour should be the same for any uefi based system right? 17:19:15 200 GB macefi ?? 17:19:15 kparal: can you maybe try that on the affected box from the live image or something and see if it turns up read-only? and figure out why? 17:19:20 and then just fix the bug, kthx 17:19:30 it's also possible that some error occurred during restoring macos to the drive. I have it backed up so that I can restore it to original layout 17:19:32 mkolman_: huh. that's odd. 17:19:39 dgilmore: nah, macs are weird. 17:19:41 but the macos system booted just fine and no error was shown 17:19:43 dgilmore: note that 'macefi' type. 17:19:46 dgilmore: i don't remember all the details. 17:20:03 oh that's right, Apple decided to have HFS+ formatted 'ESP'. because they're assholes. 17:20:13 oh 17:20:20 that makes sense I guess 17:20:33 if their mutant of a firmware can read that 17:21:26 you know, i think it misidentified the partitions for some reason 17:21:31 sda1 looks a lot more like the actual ESP to me 17:21:44 that could be it 17:22:02 it might think the Mac root is actually the EFI partition 17:22:04 yay 17:22:17 yeah 17:22:19 that's what' 17:22:22 s going on i think 17:22:24 now, to figure out *why* 17:22:39 it correctly (i think) determines that sda1 is a regular EFI system partition 17:22:51 but then it decides sda2 is a weird mac HFS+ EFI system partition and uses that instead 17:24:48 and it gets mounted r/o, which turns out to be good. 17:25:32 /me thought this was a blocker review 17:25:58 sgallagh: well, my point here is to try and figure out if this is something specific to kparal's setup, or if it's gonna affect all macs. 17:25:59 Given that we have to make a Go or No-Go decision, the real question is: is this a blocker or not? 17:26:08 which makes a big difference to whether it's a blocker. 17:26:08 ok 17:26:13 right, we need info to tell 17:26:18 although we can discuss dropping the criterion in parallel, if you like 17:26:23 and if we agree to do that we can stop debugging 17:26:30 /me nods 17:26:41 * adamw goes back to figuring out how this detection works 17:27:21 Frankly, I think that the criterion should just read some variation on: "If the user attempts to set up a dual-boot with OSX, the OSX installation must remain bootable" 17:27:30 last I looked linux defaulted to ro for HFS+ with journal turned on 17:27:45 because it does not support the journal 17:27:47 Remove the mandate that we must *succeed* at dual-booting and just ensure we cause no harm 17:28:09 yeah, it seems like a good proposal 17:28:11 "anaconda takes the Hippocratic oath" 17:28:50 i'd like proposals to at least consider the original rationale for accepting the criterion 17:28:53 otherwise we're just ping ponging 17:29:06 do we have that somewhere? 17:29:43 * nirik reads all the nice references on the wiki page 17:30:00 "We currently do not have any release criterion that applies to dual booting with OS X. Since our Mac support is very poor and has no prospect of near term improvement -- in particular, we have concerns that running Fedora on a Mac has caused at least one Mac to overheat and die -- the consensus seems to be that dual booting with OS X should not be a requirement. However, it's also not OK to destroy the user's OS X without warning 17:30:06 from https://lists.fedoraproject.org/pipermail/test/2014-August/122496.html 17:30:38 note this is ultimately rooted in the Workstation tech spec 17:30:39 https://fedoraproject.org/wiki/Workstation/Technical_Specification 17:30:50 which says "One aspect of storage configuration that will be needed is support for dual-boot setups (preserving preexisting Windows or OS X installations), since e.g. students may be required to run software on those platforms for their coursework." 17:31:25 we should at least pull someone from workstation into this decision 17:31:30 * adamw goes to find someone 17:31:44 adamw: we have stickster here 17:32:18 * adamw pinged mcatanzaro, cschalle and mclasen 17:32:20 "preserving preexisting Windows or OS X installations" - may I read it as "do not harm the existing Windows or OS X installtion" ? 17:32:58 sooooo 17:33:04 cschalle is incoming 17:33:08 * mattdm summons cschalle since he's sitting 20 feet from me 17:33:21 * stickster is here 17:33:25 if i'm reading this right, blivet will decide that any HFS+ partition marked as bootable is a 'macefi' partition 17:33:41 yo 17:33:53 jkurik: well, yes, but 'support for dual-boot setups' rather heavily implies that fedora works. 17:34:01 jkurik: it's not a 'dual-boot setup' if you can't actually boot fedora. :P 17:34:08 BRB, just located mac install iso... 17:34:16 adamw: but you can at least install it :-) 17:34:28 kparal: is the HFS+ partition on your system marked 'bootable'? 17:34:35 and i guess, the million dollar question, is that common? 17:34:49 adamw: I don't know and I'm no longer in the office :/ 17:35:09 however, I restored the macos from the backup, the same way as I used to do each cycle 17:35:31 so unless the restore tools screwed up something, it used to work and now it doesn't 17:35:32 well, for the record, code is https://github.com/rhinstaller/blivet/blob/2.1-devel/blivet/populator/helpers/boot.py 17:35:50 note how the 'match' function works there, and the MacEFIFormatPopulator class is the one we're dealing with 17:36:31 do we know how hard it would be to fix? I mean this being even on the table as a blocker I would think is only true if its a relatively trivial fix. If its 2 Months of reverse engineering Macs I guess this is not even a question 17:36:55 Did the same setup work in F24? 17:36:59 adamw: however, the last time I tested this was with F23. in F24 all the testing was done by cmurf 17:37:09 well, we generally set things as blockers not based on effort but based on requirements 17:37:18 cschalle_: well, if the bug is what i think it is, i can think of a few potential 'fixes' already, though the consequences are a bit hard to guess 17:37:25 stickster: well, it's a hybrid. 17:37:26 but of course there is the need to be practical 17:37:31 adamw: *jinx 17:37:51 so there is one important consideration here, which is that we can't enforce criteria we don't test 17:37:56 and this has not been tested for F25 until today 17:38:00 *nod 17:38:01 which is obviously not sustainable 17:38:06 and not many folks have HW 17:38:15 so if we really want to keep this criterion, whoever wants to keep it needs to commit to providing the testing 17:38:29 If nobody has found this yet for F25, it casts doubt on this being a valid release requirement 17:38:45 cmurf usually runs os x tests, not sure why he didn't this cycle, but 17:38:55 otaylor: that depends on the size of the testing group 17:39:19 i suspect the ratio of pre-release:post-release testers for macs is unusual 17:39:25 but this is getting philosophical I guess... 17:39:32 Well for Fedora Workstation we are really targeting developers using macOS 17:39:34 (i.e. it seems we have quite a lot of people who want to install os x on macs post-release, but almost no-one who wants to test pre-release) 17:39:39 i guess this is because 'burner macs' are rare 17:39:52 If dual-boot doesn't work that is a serious problem for attracting that target audience. 17:39:55 people generally want to install to their personal laptop, and obviously don't want to use it for testing 17:39:57 The hardware is very expensive, yes 17:39:57 yeah, that's unfortunate. and I understand, being in that pickle myself 17:40:00 stickster: I'm saying that if there's 0 representation on people trying the beta, I can't imagine that the representation for final release is more than a tiny fraction 17:40:09 otaylor: that's what i'm considering above :) 17:40:14 what is mactel-boot-setup? usually on my mac I just run the regular live and install from it 17:40:26 adamw: But at beta release time, we're not talking about "participate in QA and wipe my laptop" 17:40:29 yep, but that's also why we attempt to push into that area :-) 17:40:36 nb: it deals with the mac's special graphical boot menu, i think, but it's not relevant here really 17:40:43 oh ok 17:40:43 nb: the bug is that anaconda thinks the wrong partition is the ESP 17:41:05 otaylor: sorry? don't quite parse that 17:41:14 * stickster can live with this not blocking, but has a question whether this might be fixable and picked up in a community respin 17:41:24 btw, in terms of fixing this: one obvious thing would be to slap a 'maximum size' on the macefi format type 17:41:27 I know we don't rely on those, but we know people care about and use them 17:41:27 it currently doesn't have one 17:41:38 * nb has a mac, but all I test is using dnf system-upgrade usually, since i don't want to wipe it 17:41:42 if we put a maximum size on it, blivet would only identify partitions smaller than that size as 'macefi' 17:41:55 stickster, the respins would pick up any updates to the package set, so yeah, it should be picked up if fixed 17:41:58 stickster: it would probably be fixable with an updates.img 17:42:18 adamw: I was saying that people on macs are likely to upgrade to beta releases, but if this is only an initial install issue, then that's less relevant 17:42:18 but the big question is how common this 'hfs+ partition marked as bootable' case is (assuming my understanding of the bug is correct) 17:42:25 otaylor: yeah, it's initial install 17:42:40 * adamw goes to hack up a test script 17:43:49 good thing 17:43:52 grr 17:44:16 so yeah, there is another point here, a good point nirik made in passing 17:44:18 * stickster reiterates -1 to block; my assumption is there's no time to get any fix in at this point. if I'm wrong, I'd support a trivial fix but not an all-hours scramble. 17:44:32 and it gets mounted r/o, which turns out to be good. 17:44:51 * mattdm is against an all-hours scramble. 17:45:00 if the misidentified partition had been mounted r/w, this could have done something *much worse*, potentially 17:45:08 then we would've been writing to the OS X partition, thinking it was the ESP 17:45:16 which...i dunno what consequences it could have, but probably not good ones 17:45:16 otaylor: Just because existing Fedora developers are not testing it currently doesn't mean it's unimportant. It is very important for attracting new users who do not already use Fedora. If this only affects a small subset of Mac users, then -1 blocker, but otherwise I would prefer to slip. We don't want to have to tell Mac users "sorry we broke it, you cannot dual boot until F26". 17:45:47 cschalle_: if you're OK with it, I can buy another SSD for my MBP and I could at least help with some future testing ;-) 17:45:50 .fas lailah 17:45:51 so if we aren't very sure that all OS X system partitions will be mounted r/o , this could potentially be more serious 17:45:51 Kohane: lailah 'Sylvia Sánchez' 17:46:09 mcatanzaro, I suppose there is an ugly workaround: install f24 and dnf system-upgrade to f25 17:46:12 * stickster can't endanger his current drive, though, it's a critical appliance for IRL stuff 17:46:17 stickster: yeah, we can't fix this without slipping, obv. taking it as a blocker means a slip. 17:46:28 adamw: that's what I figured at this point, thanks 17:46:52 so this could be a "common bug" + updates.img, as well as possibly fixed in a cmty respin 17:46:55 nb: If we tell users to do that, we're basically saying "go use Ubuntu instead" 17:47:06 yeah, that's too ugly 17:47:10 yeah 17:47:40 we've spent quite a while on this bug now, are there others still lurking? 17:47:41 even we do not know whether the same issue is not on F24 as well as noone have tested it 17:47:56 stickster: nothing super significant, assuming we're downgrading the outstanding sddm bug 17:48:01 that's the other major one 17:48:27 stickster: i would definitely like to look into that thing i just mentioned about whether the wrongly-identified partition is always mounted r/o before making a decision 17:48:41 yeah, and that sddm/qxl issue is -1 from me as well 17:48:43 if there's any possibility we mark the OS X system partition as the ESP, mount it read/write, and then start doing stuff to it, i am worried 17:48:45 so am i understanding correctly that the bug is only happening on some macs with a partition marked a certain way? 17:49:00 nb: we aren't sure. ;( 17:49:03 oh 17:49:03 nb: my reading is that it will happen any time there's a HFS+ partition marked bootable 17:49:12 nb: we do not know how common that is 17:49:12 nb: I think the question is whether other macs *are* marked that way 17:49:16 mabye? ^ 17:49:26 yeah 17:49:33 adamw: hey, would it be at all helpful if I booted this Mac with a Live and dumped the partition table? 17:49:41 if sgallagh_ can get a fresh install to test with there that would be some good data. 17:49:41 Is there an easy way to tell if stickster's looks that way? 17:49:44 (jinx) 17:49:49 nothing has happened to this Mac that didn't come from Apple 17:49:54 stickster, sure, go ahead and buy that SSD 17:49:54 nirik: I can't in meaningful time, sorry. 17:50:01 cschalle_: :-) 17:50:01 ah well. 17:50:37 stickster: What do you mean with a Mac that didn't come from Apple? 17:50:43 This USB stick I was creating was maxing out at 1MB/sec for an 8GB image and it would take upwards of an hour to do the install 17:50:46 So minimum three hours 17:50:54 sgallagh: :-/ 17:50:54 hum... I just remembered, I have this mini here for mediawriter building/signing... 17:50:58 Kohane: He means an apple-caused bug 17:50:59 damnit, does anyone know what provides 'mount -t hfsplus'? 17:51:04 Ah! 17:51:05 Kohane: it hasn't ever been dual boot installed or otherwise changed from what they deliver 17:51:08 I could try installing on it... not sure how long it would take 17:51:14 adamw: hfsplusutils? 17:51:14 if anyone *has a mac* at all they can provide us some useful data here, without running an install test 17:51:18 adamw: or hfsutils? 17:51:27 stickster: but anaconda requires hfsplus-tools , which conflicts with hfsplusutils 17:51:28 * adamw is confused 17:52:21 * Kohane is confused too 17:53:28 to move on ... what options we have ? 17:53:31 blocker / common_bug / drop_criterion - anything else we can vote for ? 17:53:58 To me is blocker. 17:54:13 * mcatanzaro does not want to drop the criterion. 17:54:21 adamw: What would one need to do to get the information? 17:54:26 mcatanzaro: +1, that doesn't mean we can't grant exception this time only 17:54:40 /me wonders if he can boot off a Workstation live... 17:54:44 sgallagh: just run fdisk or parted and see if the main OS X partition is marked bootable 17:55:32 for me it is a blocker as well as there is a serious potential it might corrupt the OS X filesystem 17:55:56 OK, give me a couple minutes 17:56:03 and we can not be sure it does not without a timeconsuming investigation 17:56:38 * nirik is for gathering more info which he is trying to do here with a new mini. 17:56:42 gajh 17:56:46 so i think i know what broke this 17:57:01 \o/ 17:57:21 https://github.com/rhinstaller/blivet/commit/368a4db6141c7fdcb31ed45fe6be207ccc08ad30#diff-c0cef2bf2f989e2f94b5d1cca9c8115eL1112 17:57:28 that change rejigged all this detection stuff 17:57:36 * nirik looks for a dammed terminal app 17:57:37 and it looks to me like it lost a rather important condition, the highlighted line 17:57:47 * stickster had to burn a USB, almost there 17:58:11 before that change, we'd only take partitions whose 'partition.name' (I guess the partition label?) was 'Linux HFS+ ESP' 17:58:18 in the current code, afaict, that condition no longer applies 17:59:39 so i can certainly believe this is a significant breakage since whatever fedora that change landed in. 17:59:53 yes 18:03:45 adamw: OK, booting this MBP now. What do I need to be looking for in e.g. sfdisk? boot flag enabled on a partition? 18:04:14 sudo parted /dev/sda p 18:04:37 stickster: yeah 18:05:00 adamw: the commit happened yesterday, so it was probably part of FE ? 18:05:13 I can't easily dump here, but /dev/sda1 is file system FAT32, EFI System Partition, flags: boot, esp 18:05:41 /dev/sda2 is the big 250 GB "Customer" partition, hfs+ 18:05:41 jkurik: 2015 18:05:58 Which suggests that this has been an issue for F23 and F24... 18:05:59 sgallagh: ah, thanks :) 18:06:02 /dev/sda3 is 650MB hfs+ "Recovery HD" partition, hfs+ 18:06:15 stickster: you can use http://paste.fedoraproject.org/ 18:07:01 http://paste.fedoraproject.org/477461/01208147 18:07:04 aha, ok, the 'partition name' we're dealing with here is an EFI property, apparently 18:07:40 stickster: hmm, so doesn't look like yours is bootable 18:07:52 http://paste.fedoraproject.org/477463/01252147/ is diskutil list here, I can find a usb key and boot it on fedora if it would help? 18:07:56 sgallagh: the date isn't sufficient for guessing when a blivet change affected fedora 18:08:07 adamw: ok 18:08:09 sgallagh: as blivet can develop quite far ahead of what's landed in the distro 18:08:11 i can look it up 18:08:46 adamw: `git desc` ? 18:09:12 sgallagh: that would be the *sensible* way 18:09:17 * adamw just checks out the tag for the f24 package 18:09:34 latest tagged blivet for f24 is python-blivet-1.20.3-1.fc24 18:09:47 blivet 1.20.3-1 tag has the old code 18:09:51 so this is indeed an f25 regression 18:09:57 (again, assuming my analysis is correct) 18:10:40 * nirik is grabbing a f25 Xfce live to boot this thing with and install if you like. 18:11:39 /me finally got a live image booting on that Mac 18:11:47 I'll check the partitions momentarily 18:12:26 hmm. I *suspect* the most likely scenario here is some older OS X installers marked the system partition as bootable so the disk would boot on PowerPC macs 18:12:34 but i'm really making guesses based on google searches, here 18:13:02 adamw: Also not bootable. 18:13:22 sgallagh: yeah, if this recent wild-ass guess is correct, older macs would be the ones likely to have 'bootable' system partitions 18:13:32 This is a 2008/2009-era mac 18:13:59 sgallagh: what OS X release? 18:14:04 Not that old I would dare to say.... 18:14:14 kparal: you said yours is like 2012, right? 18:14:24 could the partition scheme somehow be older? 18:14:32 adamw: yes, about that 18:14:41 2011-2012 I guess 18:14:47 hrrrmmm 18:15:13 the partition scheme was backed up in the original state 18:15:18 * adamw can't account for an older Mac having non-bootable OS X partition but a newer one having bootable OS X partition. that doesn't make sense. 18:15:21 adamw: Not sure originally. somewhere between 10.6 and 10.8 18:15:26 kparal: as in, the state it came from the factory? 18:15:33 but I can't say if one of the restore tools could set an extra flag while restoring 18:15:37 adamw: yes 18:15:39 the one I have here is super new. 18:16:01 sgallagh: wikipedia says 10.5 was the last OS X that worked on powerpcs, so logically speaking you'd think if anything, they'd stop marking the OS X partition bootable with 10.6 18:16:09 kparal: i guess that's possible :/ 18:16:11 adamw: I also have full-disk dd backup. it takes a long time to restore, but I could do that tomorrow and check the boot flag (and do the dual-boot install again) 18:16:13 but that's tomorrow 18:16:16 kparal: yeah 18:16:25 that would be useful if only to satisfy my ocd 18:16:45 nirik: did you check if yours is marked bootable? 18:17:06 Oh, hang on. I wasn't paying close enough attention. DIsregard my information. I was looking at the partition layout of the USB stick 18:17:08 I am writing a usb stick now to boot linux on it. 18:17:10 adamw: for ref, my Mac is on El Capitan, which is (I think?) 10.11.x 18:17:12 sgallagh: .... 18:17:20 again, I haven't molested the partition table ever 18:17:21 Sorry 18:17:23 * adamw smacks sgallagh around a bit with a wet fish 18:17:29 /me takes it stolidly 18:17:44 Turns out the reason this thing wouldn't boot is that the disk doesn't show up at all! 18:17:44 is there any trick needed to make a mac boot from usb? 18:17:48 /me facepalms 18:17:59 nirik: Hold down "option" when powering on 18:18:22 yeah, when you hear the chime hit it immediately 18:18:25 I have no apple keyboard. Is there a normal keyboard equivilent? 18:18:35 Left Alt/Meta? 18:18:35 alt, I think 18:18:37 surely os x has fdisk, right? right?! 18:18:43 adamw: it doesn't. 18:18:45 ... 18:18:49 yak farm. 18:18:55 well, it does... but it does something totally different 18:18:56 adamw: no, but I booted Fedora WS Live 25 Beta and just ran parted 18:19:05 stickster: yeah, that's fine so long as you run it on the right disk 18:19:08 nirik: right alt 18:19:08 heh, I did 18:19:13 stickster: i meant for nirik, since he's having trouble booting fedora 18:19:17 thanks kparal 18:19:18 Yeah, yeah. I'm an idiot. 18:19:19 oh sorry 18:19:20 Apple Option +R actually 18:19:21 stickster: what was your result? i missed it 18:19:21 * stickster shuts up 18:19:40 linuxmodder: what? That's for recovery mode, isn't it? 18:19:41 adamw: http://paste.fedoraproject.org/477461/01208147 -- from a MacBook Pro retina mid-2015 18:19:47 sgallagh: yes, it is 18:19:53 it will confirm a bootable disk too tho 18:19:57 stickster: okay, so score one for 'not bootable' 18:19:59 Commadn + R 18:20:16 it's option-command-r for internet recovery, IIRC 18:20:28 yup on newer macs 18:20:46 if we get one more 'not bootable' result i'm gonna say this is fine 18:20:51 ack 18:20:58 ss shipit 18:21:02 * nirik is waiting on this dd still... I think it's about 75% done 18:21:10 and hope to god no-one has an OS X partition that's a) marked bootable and b) not journalled 18:21:38 aren't all hfs partitions these days journalled? 18:22:04 I'm about to try a clean install on a MacBook Pro, if that's still of use 18:22:37 barq++ 18:22:51 barq: what age is that, out of curiosity? 18:23:01 The MBP? 18:23:17 mid 2010 15inch, but it's running 10.12.1 Sierra 18:23:32 And I'm dualing booting f24 right now 18:23:34 Quite new then. 18:23:50 not 18:23:58 barq: could you list your partitions and paste it to http://paste.fedoraproject.org/ ? 18:24:10 sudo parted /dev/sdX p 18:24:44 * nirik should pick up some faster usb drives. This is a Red Hat branded one I got at new hire... it's a bit slow now. 18:24:51 could not stat device 18:25:14 barq, X is not literal it is likely sda or sdb 18:25:20 the X is a placeholder 18:26:17 http://paste.fedoraproject.org/477471/14788023/ 18:26:17 linuxmodder: i hope like crap that they are. 18:26:17 like this from my VPS http://paste.fedoraproject.org/477472/88023671/ 18:26:30 finally 18:27:28 and install is moving along barg ? 18:27:46 Formatting the usb now 18:28:16 * stickster brb, have to take care of dog 18:28:35 Should I ask about the update here or in #fedora ? 18:28:47 s/update/install 18:30:15 ah shit. 18:30:16 nirik: We're in suspense here 18:30:16 shit shit shit. 18:30:18 barq, not here 18:30:20 triple shit. 18:30:22 adamw: ... what? 18:30:23 i missed something quite important 18:30:25 adamw, ? 18:30:38 i read the _bootable definition out of AppleBootFormatPopulator not MacEFIFormatPopulator . 18:30:39 sigh... it booted, but no video on hdmi... 18:30:49 so...again if i'm right...we don't actually care if the partition is bootable after all. 18:30:59 Doh... 18:31:19 it shouldn't matter if its pre-existing as such iirc 18:31:24 the only relevant conditions are 'is it hfs+' and 'is it in the valid size range' (which for this case is just 'larger than 50Mib' 18:32:02 so...yeah. anyone is welcome to check my work, but my diagnosis here is that blivet is going to think absolutely any HFS+ partition larger than 50MiB is an HFS+ EFI system partition. 18:32:09 * stickster back 18:32:14 sorry for the mistake 18:32:15 the /boot or the ESP only need be 50Mb? 18:32:31 /me gives up and goes to lucnh 18:32:33 *lunch 18:32:35 linuxmodder: no. that's not what i'm saying. it's complicated, if you're not following, please don't make me explain it again 18:32:43 upshot of that is: this is gonna be broken on pretty much any mac, i suspect. 18:32:52 so we're back to either block the release or dump the criterion. 18:33:03 ugh. 18:33:07 again, this is all based on my analysis, please do check my work. 18:33:20 adamw: and if you're correct, it's a regression from F24 18:33:21 ? 18:33:35 adamw: is there still the risk it can demage the existing osx installation ? 18:35:19 I mean, on the system kparal used the FS has been mounted as read only. Is there a risk it can be mounted as writable one ? 18:35:37 ok, I am in. 18:36:09 http://paste.fedoraproject.org/477478/47880296/ 18:36:13 parted info ^ 18:36:33 oh wait, thats the stick 18:36:50 jkurik: theoretically there is 18:36:51 nirik: thats a 500G drive 18:36:53 no, thatt the drive 18:36:57 nirik: sorry, turns out we don't need that any more. my bad :/ 18:37:05 i was misreading a thing. 18:37:36 no worries. Any other info I could gather? or would an install be of use? 18:37:45 * nirik reads back up 18:37:52 nirik: not from my perspective, I *think* I understand what's going on here fully now 18:38:04 of course, i could be entirely wrong. please do go poke through blivet and check my work, i gave rreferences 18:38:21 so, them imo even if we drop the criterion we should block on this as we can not risk users will demage their existing installtions 18:38:22 but my reading is that this will basically happen on every Mac. 18:38:29 jkurik: i'm fairly sure it's a very small risk. 18:38:58 i don't think there's any 'normal' scenario where you have journalling disabled on an HFS+ partition, it's something that has to be done manually if you want to do something weird like write to it from Linux. 18:39:07 but it is, theoretically, there. 18:39:16 i can probably whip up an updates.img for this in, like, an hour. 18:39:17 can this be fixed after release ?with a update 18:39:17 Well, I can't check because I have no Apple computers. Not even friends with one to take borrowed for a while. I'm sorry. But if I can do anything of use... 18:39:27 Southern_Gentlem: not fully, it's an installer bug. we can provide an updates.img . 18:39:41 and a respin should work too 18:40:01 nb: with respin we should slip the release 18:40:07 assuming we build a package with the fix and push it stable, yes. 18:40:07 my efi part is fat32... is it only the older ones that use hfs+ 18:40:15 jkurik: he means the unofficial post-install respins 18:40:22 ah, sorry 18:40:22 jkurik, the unofficial respins 18:40:33 nirik: yes, but that doesn't matter: the point is that blivet/anaconda will misidentify your HFS+ *OS X partition* as one of those older HFS+ ESPs 18:41:13 I think i am +0 18:41:17 adamw, i am willing to do a f25 updates spin once that if fixed and pushed for the mac users 18:41:41 If we think it will always mount the partition as ro, I think a weak -1 18:41:59 i don't think it's logically plausible to vote anything but +1 as long as we have the criterion. 18:42:04 so this comes back down to, do we dump the criterion. 18:42:09 true 18:42:24 (or, i guess, do something like say it's dumped for f24 but back in f25 and we promise to test it this time). 18:42:29 I don't think so, we shouldn't dump the criterion. 18:42:29 er, f25/f26 18:42:54 adamw, I am kinda thinking (or, i guess, do something like say it's dumped for f25 but back in f26 and we promise to test it this time). 18:43:33 nb -1 18:43:36 and if we push does that push us to the 29th? 18:43:43 I doubt we will until last minute again 18:43:45 adamw: This (or, i guess, do something like say it's dumped for f24 but back in f25 and we promise to test it this time) sounds really awful to me. 18:44:05 Southern_Gentlem: no 18:44:25 * nirik can't mount the hfs+ osx partition at all here. 18:44:29 we would go one week 18:44:39 nirik: not even ro? 18:44:45 dgilmore: nope. 18:44:49 weird 18:44:55 dgilmore, ok wasnt sure since alot of people take off the next week for vacation 18:45:03 anyhow. I guess if the WS working group wants to keep this criteria then +1 blocker. 18:45:41 I don't suppose there's much way to get a fix, respin and try tomorrow? but I guess we agreed to stop that hero junk... 18:46:09 yeah, we agreed on no-heroing 18:46:43 I suppose it'd have to be +1 blocker, since the criteria is there 18:47:12 +1 blocker 18:47:15 so, then it is a nirik pointed. If WS WG does not want to drop the criterion, it is a blocker 18:47:39 what nirik said 18:47:51 although it seemed like mcatanzaro was the one really wanting to keep it and others didn't care as much? 18:47:52 I dislike that this criterion basically holds the release hostage to a supposedly important point that isn't accompanied by people testing (i.e. Workstation group) 18:47:54 stickster: what say you? 18:48:08 IOW, if it's important, pony up and help 18:48:12 stickster, I agree 18:48:37 Ideally it would be tested by QA team 18:48:47 mcatanzaro: Ideally we'd have infinite QA people. 18:48:50 ;-) 18:49:00 but "someone else's problem" doesn't scale :-) 18:49:02 mcatanzaro, what is the actual importance factor tho for it ? 18:49:05 If we're going to drop the criterion we should drop our goal of recruiting macOS users. 18:49:13 I recently ran into this exact issue. I'd be willing to work with people on it, test, etc., but I'm new to it all. 18:49:16 mcatanzaro: Yeah, I don't like that either... don't get me wrong 18:49:18 And if we keep the criterion, /s/OS X/macOS/ 18:49:23 Kohane: the rationale for that is that no-one actually tested this throughout the whole cycle until like today 18:49:24 mcatanzaro: I think we basically *have* to +1 blocker 18:49:36 and it annoys the heck out of me because I don't like eleventh hour surprises 18:49:46 And I think we do not want to drop our goal of recruiting macOS users. 18:49:48 But 18:49:57 Maybe we should stop accepting new blockers at the 11th hour 18:50:03 Something to consider in the future. 18:50:19 * nirik thinks we should have gotten a nice big long game for kparal last week. ;) (just kidding kparal, thank you for finding this!) 18:50:28 hero'ing might be at least *possible* here as a fix should be very specific to the OS X code 18:50:36 mcatanzaro: then we need resources to test earlier 18:50:39 with very high confidence 18:50:42 adamw: Yes, I know. But Apple stuff isn't that common. I for myself have no Apple laptops and I know nobody with one. How can you test anything without machine? 18:50:45 well, we stop accepting blockers when we go. ;) 18:50:45 perhaps, but in practical terms our release quality over the years has been AMAZINGLY good, and part of that owes to the blocker process 18:50:49 so everything else should remain untouched. i'm not in love with the idea, just mentioning it. 18:50:50 nirik: right 18:51:18 Kohane: I don't know where on the planet you live, but I see more Apple laptops than Windows nowadays. It is very important. 18:51:26 adamw: so results could carry over? 18:51:26 Kohane: for sure, but the point is, if no-one can test it earlier than 'the day we decide whether to release', then clearly it's silly to block the release on it. for the long term, if we're going to block on this we need to ensure it's being tested. 18:51:28 kparal++ 18:51:28 stickster: Karma for kparal changed to 7 (for the f24 release cycle): https://badges.fedoraproject.org/tags/cookie/any 18:51:31 Kohane: you know someone now. How can I help? 18:51:38 nirik: yeah, in theory we could just carry over results for every test besides OS X dual boot. 18:51:48 and we have openQA as a sanity check. 18:51:57 the only risk would be some kind of nutty compose process bitflip or human error. 18:52:01 mcatanzaro: +1 18:52:10 which is, you know, a non-zero possibility. 18:52:19 mcatanzaro: Frankfurt, Germany, I live in that part of the planet. 18:52:23 (for the love of god no-one merge anything to comps or kickstarts) 18:52:26 there's lots of apple laptops. I don't know how many peope dual boot them with fedora tho. 18:52:31 adamw: +10^100 18:52:39 adamw: Nonzero but quite low. I think we can risk it. 18:52:48 (if we have a hero) 18:52:59 well, you people kick it around, i'm gonna go make a fix. 18:53:05 Well, if I can borrow someone's laptop I will certainly test this 18:53:12 well, honestly, I could have tested it earlier, I just didn't realize it wasn't tested at all during the cycle. and of course there was lots of other stuff to do, as always 18:53:15 of course, we'd need to have anaconda folks on board to do the hero build 18:53:22 kparal: you have nothing to apologize for :-) 18:53:46 and releng to do a compose. 18:53:46 mattdm: how are you feeling about this? 18:53:47 just stating it could have been done earlier by our team. however, all we have is this ancient mac mini, nothing modern 18:54:08 I can see if one of my mac friends let's me test on it today if we block on it 18:54:22 * mcatanzaro suspects that Red Hat can afford to buy one laptop 18:54:30 stickster: dazed :) 18:54:35 * nirik wonders if we should try and have some meta criteria that at least (wild number) 5 people are able to test each test, and if we drop below that revisit the critiera? but that could be difficult to tell 18:54:48 * Southern_Gentlem sets back and eats popcorn watching the show 18:54:52 kparal: fwiw I will see about devoting some test time, assuming that (1) I can order an add'l SSD and successfully install it; (2) I don't destroy my MBP in the process; and (3) cschalle_ is cool with my spending some hours on this at least a few weeks before Alpha/Beta/GA 18:54:55 nirik: we do not get that for spins 18:55:13 nirik: people complained when I wanted two people to test each spin and sign off on it 18:55:37 dgilmore: true. and I am sure theree are other things that get no testing or little... fakeraid, sas drives, 18:55:54 We have 5 minutes left 18:55:57 windows AD joining, etc. 18:55:57 mattdm: I think the question currently under consideration is, "Do we go back on our position and hero this, or not?" 18:56:16 (was afk for a bit - reading scrollback) 18:56:22 dgilmore, feel free to add me to the spins testers I just had to drop off this cycle for work things 18:56:34 linuxmodder: i can not add anyone 18:56:41 mattdm: if no hero-ing, "Do we block, or drop the criterion?" (and it sounds like almost no one wants the latter) 18:56:46 mattdm: so it's really "hero or block" 18:56:55 I don't think we should hero it. Either block or don't 18:56:55 Block 18:57:23 I have to agree with mattdm 18:57:38 I agree, too 18:57:43 not happy about a block but seems the only option that we have 18:57:46 +1 block 18:57:48 +1 block 18:57:51 * stickster has already many times stated his disdain for hero 18:57:53 +1 block 18:57:53 seems that thw Workstation WG wants it so we have to accept it as a blocker 18:57:55 +1 block 18:57:56 er, not the hero, the hero-ing 18:57:59 +1 block 18:58:00 +1 18:58:16 well, this case is less heroing that most 18:58:26 stickster++ 18:58:30 nirik: for sure. 18:58:39 nirik: true 18:58:42 it will not require everyone to retest everything 18:59:34 Whatever else happens, I plan to make a note for the Workstation WG of how this went down, and make an earnest request for testing in run-up to F26 on anything the WG professes to care about :-) 18:59:38 cschalle_: ^ fyi 18:59:52 stickster++ 18:59:53 alrighty 19:00:03 and that gives us time to test more of the spins as well 19:00:19 great, a Thanksgiving week release. :-( 19:00:36 pretty much the worst time to release other than Christmas 19:00:42 proposed #agreed 1393846 - AcceptedBlocker (Final) - by adamw's analysis of the cause, this seems like a clear violation of "The installer must be able to install into free space alongside an existing OS X installation, install and configure a bootloader that will boot Fedora." that will affect all dual-boot OS X installs 19:00:53 +1 block :( 19:00:57 ack 19:01:00 stickster: will it be better to slip for two weeks then ? 19:01:00 ack 19:01:01 ack 19:01:01 ack 19:01:02 ack 19:01:08 ack 19:01:09 stickster: or memorial day, apparently 19:01:09 ack 19:01:56 adamw: Please do update the OS name in the criterion to "macOS" though! 19:01:56 jkurik: that's kind of a USA centric approach, so I would say no, let's get it out of here 19:02:09 stickster: ok 19:02:19 jkurik: but I'm not the sole decider either 19:02:22 * mcatanzaro sees no problem with Thanksgiving week release, since the holiday is at the end of the week after we've already released 19:02:23 mattdm: ^ 19:02:29 mcatanzaro, +1 19:02:31 mcatanzaro: true enough 19:02:34 mattdm should make the call, it's a marketing issue 19:02:42 hey, maybe people will download over the holiday :-D 19:02:54 stickster: I agree, ship it 19:03:01 mcatanzaro, other than the mirror owners maybe gone on vacation 19:03:03 There's people outside US, in case you didn't notice... :-/ 19:03:14 we can release thanksgiving week, but will likely have to slip two weeks if we slip after 19:03:29 give people something to do other than discuss politics with tge relatives 19:03:54 mattdm: good idea :p 19:04:04 Kohane: About half the developers are from the US so the calendar has to account for that.... 19:04:29 sorry folks 19:04:34 #agreed 1393846 - AcceptedBlocker (Final) - by adamw's analysis of the cause, this seems like a clear violation of "The installer must be able to install into free space alongside an existing OS X installation, install and configure a bootloader that will boot Fedora." that will affect all dual-boot OS X installs 19:04:37 * adamw multitasking 19:04:43 jkurik: Where is the release readiness meeting? 19:04:51 As we have a blocker, I would suggest to end the meeting now with result "No-Go". nirik, mboddu, dgilmore, adamw: are you ok with it ? 19:04:54 adamw++ 19:04:58 nah 19:05:00 sorry to prolong 19:05:07 but we need to make a call on the sddm bug too just so we know where we are 19:05:08 stickster: we are already on the readiness meeting for 5 minutes :-) 19:05:09 that shouldn't take too long 19:05:13 jkurik: no, we should go through any other proposed blockers 19:05:14 jkurik: ha, I guess I'm not 19:05:48 * mattdm is eating lunch now ping if needed 19:05:49 #info moving to re-evaluation of one accepted blocker 19:05:50 #topic (1382001) Logging out from second user after switching users kills first user's session (with xorg-x11-drv-qxl at least) 19:05:50 #link https://bugzilla.redhat.com/show_bug.cgi?id=1382001 19:05:50 #info Accepted Blocker, xorg-x11-drv-qxl, ASSIGNED 19:05:52 jkurik: we have to check the status of accepted blockers also 19:06:15 ok, so lets continue with the blocker review 19:06:23 seems only affect VMs, -1 19:06:23 we can do the other proposed blocker easily enough: we already have a verified fix for it that's pending stable push, so fuckit. and the status of all other accepted blockers is 'verified fixed'. 19:06:27 this is the only one that really matters 19:06:51 -1 given that this is qxl only... +1 FE I guess tho. 19:07:23 -1 blocker +1 FE 19:07:25 -1 +1 fe 19:07:31 * potty is here. Sorry for delay 19:07:37 given that I think switching users in virt is a rare use case 19:07:44 o/ potty 19:07:49 +1 FE -1 block 19:07:50 * fale agrees with nirik and dgilmore 19:07:57 -1 +FE 19:08:05 potty: the readiness is delayed as we have not finished the go/no-go yet, sorry 19:08:26 I am +1 FE 19:08:32 jkurik: it is ok. I will be here watching. 19:09:50 adamw: lokos like +1 FE 19:09:56 well FE 19:09:59 bahh 19:10:39 +1 FE -1 blocker 19:11:32 -1 blocker +1 FE 19:11:58 alrightry, looks like a consensus 19:12:49 proposed #agreed 1382001 - RejectedBlocker (Final) AcceptedFreezeException (Final) - this is downgraded to FE as it only affects the qxl driver, and we think user switching is fairly rare in VMs (and also rare in live sessions, so the combination makes a post-release fix acceptable) 19:13:06 ack 19:13:08 ack 19:13:11 ack 19:13:13 ack 19:13:18 ack 19:13:19 ack 19:13:29 ack 19:13:35 ackky 19:14:39 #agreed 1382001 - RejectedBlocker (Final) AcceptedFreezeException (Final) - this is downgraded to FE as it only affects the qxl driver, and we think user switching is fairly rare in VMs (and also rare in live sessions, so the combination makes a post-release fix acceptable) 19:14:44 alright, i'm done, you can all go now 19:14:45 :P 19:14:59 thanks for sticking around, folks, i apologize for the Watch AdamW Debug Blivet Show 19:15:17 thanks adamw for the blocker review and for the extensive analysis 19:15:32 #topic Test Matrices coverage 19:15:50 I would propose to skip this section as we are slipping ... 19:16:50 yes, I think is fine skipping it. 19:17:20 proposed #info Due to a release blocker the Test Matrice coverage is skipped and will be reviewed the next Go/Go-No meeting 19:17:31 releng votes no go 19:17:45 #info Due to a release blocker the Test Matrice coverage is skipped and will be reviewed the next Go/Go-No meeting 19:18:00 #topic Go/No-Go decision 19:18:17 so we have releng no go 19:18:54 fesco (as dgilmore is a member of it) is probably no go as well 19:19:00 nirik, adamw ? 19:19:11 fesco is no go 19:19:13 QA votes no-go due to unaddressed accepted blocker. 19:19:18 infra is no go 19:19:25 yeah, no go for now 19:19:29 who else gets a vote? 19:20:10 #info the Final result of the Go/No-Go meeting for F25 Final is No-Go due to existing blocker (BZ 1393846) 19:20:11 not sure it matters, it's pretty clear. ;) 19:20:36 #action jkurik to publish the Go/No-Go result 19:20:36 nirik: indeed, I was going to vote no for everyone 19:20:45 #topic Open floor 19:20:47 anything else to discuss today on this meeting ? 19:20:47 mattdm says no go 19:20:56 :) 19:21:22 if nothing else, I will continue with the readiness meeting .... 19:21:25 * dgilmore notes that he will be on PTO the new ship date 19:22:06 nothing else 19:22:24 ok, I guess mboddu and nirik can do the release 19:22:38 jkurik: mboddu and pbrobinson 19:22:46 ok 19:22:51 most everything will be done before i go 19:23:03 nirik: is free to help also 19:23:03 * nirik is happy to help. 19:23:03 * pbrobinson waves 19:23:08 thanks all for comming 19:23:10 #endmeeting