16:00:01 #startmeeting Fedora QA meeting 16:00:01 Meeting started Mon Dec 23 16:00:01 2013 UTC. The chair is adamw. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:01 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:00:05 #meetingname fedora-qa 16:00:05 The meeting name has been set to 'fedora-qa' 16:00:09 #topic Roll call 16:00:11 ahoyhoy folks 16:00:41 * satellit listening 16:00:57 * cmurf is alive 16:01:16 * danofsatx present (pun intended) 16:01:25 * brunowolff is here 16:02:09 I will be dissecting an external HD, but I'll be paying attention 16:04:07 * tflink is here 16:04:39 * adamw on laptop, technical difficulties on the big boc 16:04:41 box, too 16:04:48 * akshayvyas is here 16:05:13 #topic Previous meeting follow-up 16:05:58 * satellit anaconda blew awau a failed install to a fat and ntfs ext Hd lost my intrnal HD on main laptop. still reinstalling it 16:06:00 #info "adamw to push the sanity check proposal out, group seems to approve" - did that, https://lists.fedoraproject.org/pipermail/test/2013-December/119558.html 16:06:31 #info "adamw to check F19 and F18 commonbugs for issues that should be copied to F20" - did that too, found a few, cleaned up the f19 page a bit while I was in there 16:07:32 #topic FedUp post-mortem 16:07:57 so in case anyone didn't know, let me #info the story 16:08:34 #info there was a SNAFU with fedup on release day: fedup 0.8 was still in updates-testing for F18 and F19 on the day of release, and it turned out upgrades to F20 with fedup 0.7 and the release upgrade.img do not work 16:09:14 #info upgrades with 0.8 work fine, this was communicated via the commonbugs and various other means: https://fedoraproject.org/wiki/Common_F20_bugs#fedup-07-fail 16:09:41 I just wanted to throw this item on the agenda so we can see if there's anything to learn here 16:10:37 anyone have any thoughts on the whole thing? 16:10:42 so basically 0.8 came late and took us by surprise 16:11:33 well, what surprised me was 0.7 not working 16:11:53 It should be one of our test cases in Beta 16:12:01 i thought 0.8 would be an optional update, we had lots of passes in the matrices and wwoods wasn't yelling about how 0.8 needed to go out or anything 16:12:01 and fedup should be frozen if it works 16:12:02 we didn't exactly have a way to test 0.7 client with an 0.8 fedup-dracut produced image though 16:12:11 the upgrade.img is in the RC tree 16:12:39 hmm 16:13:00 and, it should be a blocker if it doesn't work 16:13:03 the upgrade.img in RC1's tree was built with which version of fedup-dracut? 16:13:43 cmurf: afaik that's the one we released. the RC1.1 tree was just sent out. 16:13:45 dgilmore: right? 16:14:00 danofsatx: it already is tested and blocked on at Beta, but we don't freeze fedup (or anything) at that point. 16:15:03 well then, something in the process is broken. We shouldn't be hit with this issue on release day - we should have known about this with TC4 or 5 16:15:47 the thing that happens on release day is the mirrormanager link moves from pointing to beta to final. 16:16:15 so prior to F20 going out, the fedup test case (and the wiki instructions) said to use the latest fedup from updates-testing, and to pass a --instrepo parameter pointing to the development/20 tree on the mirrors 16:16:17 well similar to how dracut itself cause LVM thinp to explode with RC1, it seems fedup-dracut version changed with RC1 and caused the tested client side fedup to explode 16:17:01 * roshi walks in late 16:17:04 that appears to be the case, yeah, but it wasn't an 'unauthorized' change, we wanted fedup-dracut 0.8 for blocker/FE bugs 16:17:09 morning roshi 16:18:00 Wasn't tracking missed messages and didn't think it started yet... :-/ 16:18:06 ok then fedup-dracut 0.8 arrived too late to have any practical chance of testing with 0.7 fedup? 16:18:08 #info adamw changed the fedup test cases to recommend testing latest stable fedup (instead of updates-testing) against the current TC/RC tree (not development/ tree) 16:18:41 cmurf: no, I'm fairly sure the upgrade.img in each TC/RC tree is the upgrade.img we 'would ship' with that TC/RC. but our test cases did not used to suggest using that tree 16:19:13 i think perhaps when we wrote the instructions, releng was not generating upgrade.img's along with TC/RC compose, or fedup was just moving too fast and we always wanted to test the latest (this stuff was all written for F18 IIRC) 16:19:38 my recollection is that fedup upgrade testing was pretty smooth but also pretty early 16:19:52 the problem was that until F20, you couldn't use the TC/RC trees with fedup 16:20:09 my guess is that by bad luck, everyone who tested a 0.8 upgrade.img tested it with fedup 0.8, and everyone who tested with fedup 0.7 tested with an older upgrade.img somehow or other. but if anyone has figured it out any other way... 16:20:09 * tflink is trying to remember why and what changed 16:20:29 * adamw has tested that you actually *can* run fedup against RC1.1 tree, fwiw. 16:20:31 so i think when fedup-dracut 0.8 landed there just wasn't anyone testing the actual combination of fedup 0.7 and fedup-dracut 0.8 produced updates.img that was going to occur on release day 16:20:40 cmurf: yeah, that's all I can figure. 16:21:09 so I think the more true-to-life test case instructions should help, and be more appropriate now we don't expect fedup to change every day 16:21:24 adamw: yeah, something changed for F20 about how the tree is composed so that you can run fedup against the TC/RC tree but IIRC, it doesn't work against the development/f20 tree 16:22:24 did when I tested, I think 16:22:33 so i guess this might be Nervous Nelly, but I'm getting to the point where any changes to dracut makes me think "oh fuck, red flag, retest anything that involves booting" 16:22:38 an upgrade.img gets built in the development/ tree daily doesn't it? 16:22:48 and by dracut i also mean fedup-dracut 16:23:10 adamw: yes, unless it fails for some reason... the image creation part. 16:23:12 cmurf: dracut and fedup-dracut are not maintained by the same people 16:23:21 yeah i know, small problem 16:23:32 but in any case, it has dracut in the name 16:23:34 fedup-dracut is conceptually a bit of fedup, maintained by wwoods 16:23:36 heh 16:23:54 so, i wouldn't fixate on this being a 'dumb late change' or something like thinp was, the situations were fairly different 16:23:58 i'd say this one is rather more 'on us' 16:24:05 (me) 16:24:11 on the plus side, 0.8 fedup works quite OK with fedup-dracut 0.8 images 16:24:33 so i think it really could have been a lot worse, as in, 0.8 lands in updates testing and likewise has major problems 16:24:36 cmurf: haven't you heard? failing with separate /var is SIMPLY UNACCEPTABLE and we're ALL TERRIBLE INCOMPETENTS 16:24:50 oh yeah, sorry to tell you, tflink, but apparently there is something deeply wrong with our test harnes 16:25:02 yes i have read that but you know, manufacturered anger and drama are part of the assignment 16:25:08 adamw: ? 16:25:20 https://bugzilla.redhat.com/show_bug.cgi?id=1045168#c25 16:25:26 me? bitter? nooo. 16:25:31 (that was sarcasm, btw) 16:25:34 =) 16:26:12 cmurf: oh, i am constantly amazed that things aren't much, much worse, but it's part of the job to pretend we're shocked when everything goes wonky and that we can prevent it in future. =) 16:26:21 adamw: damn, why didn't I think of that ... 16:26:37 tflink: inorite? all you have to do is say it on bugzilla and it'll magically happen the next day! 16:26:54 so, I think the test case changes I made ought to shore us up fairly well against this in future 16:27:11 and, as cmurf says, recognizing the importance of late changes to fedup-dracut (or dracut) and making sure to re-test this kind of thing carefully 16:27:30 i mostly wanted the agenda item to make sure everyone was aware and see if there were any good ideas i'd missed 16:28:20 i think to a great degree some of these things can't be caught by QA 16:28:41 i think in theory we could catch a lot of the stuff we miss...if we had freeze periods longer than two weeks. 16:29:03 ifi there isn't an immutable policy to get certain packages updated by X date before the anticipated release, then there's simply a good chance certain test cases aren't going to find problems related to those packages 16:29:03 and, you know, didn't build the image we sign off on about 16 hours before it happens. as i remarked on-list, this is not how software usually happens. =) 16:29:24 or if we had 4 times as many testers, with 6 times as many hours in a day available to them for testing 16:29:40 right. i was pushing based on TC5 performance, and then a bunch of big changes happened in RC1 that I certainly wasn't expecting were even possible. 16:30:20 that was partly my bad, for never doing a TC6 - always seemed like RC1 was right around the corner 16:30:25 So the reality is, RC1 needs at least as much or more attention than the last TC, even if it doesn't seem like a lot of changes are expected because of how well that last TC tested. 16:30:57 well, there's lots of 'realities' 16:31:03 And somehow a fixed number of hours or days between RCx being available and "go/no-go" 16:31:03 The reality is that _every_little_ change to RC1 from TC5 needs vetted. There *shouldn't* be any.... 16:31:13 it's a 'reality' that we have more danger of borkage if the shipped candidate only exists briefly before 'go' 16:31:25 it's also a 'reality' that fedora people really like shipping stuff. especially before christmas. 16:31:51 so, competing realities! 16:32:00 which will win?! 16:32:04 all of them 16:32:07 right now 16:32:13 I think we know the answer to that question :) 16:32:22 #info noted by multiple parties that short RC lifetime is an invitation to issues like the thinp and fedup bugs 16:32:45 perhaps we can't make it a hard 'RCs must always live for X days' but at least make other stakeholder groups aware of the dangers 16:32:58 sure 16:32:59 i hope they're not expecting that our matrices are somehow 100% foolproof 16:33:33 of course they are - we're QA, after all 16:33:49 hum, idea - we send out a release post-mortem CCed to relevant parties, explaining broadly the stuff we're discussing here? 16:34:07 I like the idea 16:34:16 * satellit_e cannot block all changes to builds? for a period after we test until release...? . 16:34:18 i could do that, or does anyone else want to? 16:34:29 two questions: 1) will anyone read it 2) will anyone do anything with the info (read: care) 16:34:45 1) yes, fedora people also love reading long emails with my name at the top! 16:34:59 2) i think it might filter into their consciousnesses at least to some degree. can't hurt, i guess. 16:36:17 #action adamw to send fedup/thinp post-mortem email 16:36:27 #topic Storage validation 16:36:36 sooo...it's adam's current pet project time! 16:36:47 have people been following the email threads? 16:36:57 I've read all of the emails on this including the long one 16:37:12 i just haven't had time to reply with something meaningfully coherent 16:37:49 The main thing is I agree with categorizing/prioritizing the test matrix: #1 critical must work, #2 in between, #3 bonus 16:38:04 so that testers have some understanding of emphasis, not all tests are equally important 16:38:05 cmurf: ooh, well your mails so far have sure helped me so if you have something better coming i'm all ears =) 16:38:45 cmurf: the thing that's been most interesting to me was going back over the history of previous releases and realizing/remembering that we really didn't used to test custom part hardly at all 16:38:51 we were supposed to reply? 16:38:55 * danofsatx missed the memo 16:39:10 i like the bonus points one showing the insanity of what "needs" testing (or at least the potential for testing) and think it getting big is just fine 16:39:53 the more holes are there due to testers simply not having the time (or interest) to test those bonus items I think shows not only what resources are still needed, but shows what feature maybe aren't all that needed 16:40:28 and also down down down the road, when features are proposed, QA can say "yeah your feature is probably a #3 on our test matrix so good luck testing it yourself" 16:40:50 vs "yeah this is cool, we'd put that in #1 or #2 and hopefully ensure it works as designed" 16:40:57 danofsatx: god yes please 16:41:31 cmurf: yeah, i actually saw the same purpose to the huge, optional matrix idea: point people at it when they moan about how terrible we are for not testing everything 16:41:37 "feel free!" 16:41:42 yes 16:42:08 the other day I counted a MINIMUM of an 80 cell test grid for just Guided partitioning 16:42:14 it's like, really? 16:42:17 so, i feel like you and I are kinda getting on the same train and might be able to come up with something we both liked, i just hope everyone else is on board too :) 16:42:18 right 16:42:19 and then custom? oh god 16:42:23 it's crazy sauce 16:42:35 yes 16:42:46 i keep hoping someone's got a magic matrix that makes it make sense but i'm starting to think it really isn't my drafting skills, just the problem space 16:43:06 no it's the realm of tesseracts and other such things 16:43:16 the reality we have doesn't match up with the reality we require 16:43:25 i guess the other thing that comes into consideration here is the wiki-tooling question, because we kind of are stretching the bounds of wikitables at this point 16:43:39 yes 16:43:47 what about for custom, just considering the options where custom partitioning is most likely required 16:43:55 it's not like it'd make the actual test space any smaller, but it would be useful especially for storage if we had something better for tracking validation testing 16:44:08 danofsatx: yeah, as far as custom goes that was broadly my most recent thought 16:44:09 like coming from a previous install and wishing to keep /home, and God forbid, /var 16:44:31 danofsatx: in my last email the 'priority #2' matrix was the bits of custom partitioning we decided covered really important functions 16:44:36 like the 'poor man's upgrade' etc 16:44:52 really? did thunderbird eat that one? 16:44:58 * danofsatx goes spelunking 16:45:03 either that or you fell asleep after paragraph #46 16:45:35 danofsatx: I think Guided probably ought to have a way to preserve an existing /home on a new install, it's sorta weird we're like "yay easy install Fedora 19" and then for Fedora 20 you either use fedup or custom partitioning to preserve /home. It's a completely different experience. 16:46:16 danofsatx: look for the mail "More on storage validation strategy (was: Re: Criterion revision proposal...)" 16:46:22 cmurf: yes, I wholeheartedly agree. Guided partitioning not asking if you wish to reuse / keep an existing /home partition is broken IMHO 16:46:54 well...it's a bit feature creep-y, because that's quite a sophisticated thing to try and 'help' with 16:46:55 it will let you reuse / by first clicking delete, so that it gets reformatted 16:47:10 adamw: all it needs to do is add that /home to fstab 16:47:12 that's not really re-using 16:47:17 it needs to identify it first 16:47:30 What Could Possibly Go Wrong, right? 16:47:34 reusing / seems like a bad idea and dlehman has been absolutely opposed to it in the past, consistently 16:47:35 you're the one who reads all the storage bugs ;) 16:47:52 cmurf: deleting an existing / and partitioning in the empty space is hardly 're-using'... 16:47:53 use existing /home seems uncontroversial 16:48:06 adamw: yeah do not allow reusing then 16:48:16 it's a common hack, but i'm not sure they'd want to add it to Guided. but you could ask by all means 16:48:16 adamw: reuse that partition OK, reuse that volume, NO 16:48:42 i had the idea of a sort of 'custom-lite' for assigning mount points but not creating new volumes, but that doesn't seem to have gone anywhere 16:48:58 And I'm about an inch away thinking about that for Btrfs which is the one exception to the "must reformat" root policy 16:49:29 Instead, it's allowed that we (must) create a new subvolume for root, on an existing Btrfs volume 16:49:31 god, i'm about an inch away from trying to figure out a way to erase btrfs from existence, it would make so many things simpler 16:49:39 not true 16:49:51 erase everything else and go only with Btrfs and things get simpler 16:50:07 okay, you also have the option of erasing everything *else* instead 16:50:07 just pick one way, universe. you get one way. 16:50:11 you get LVM, RAID, partitioning all built into one thing that basically can't blow up short of kernel regressions 16:50:22 okay, so, let me see if I can #info some semblance of order into this 16:51:07 #info storage validation planning continues on list, there seems to be broad agreement to try and cover guided partitioning extensively and custom with a small(ish) 'must work' matrix and a larger 'bonus credit' matrix 16:51:21 has anyone had any thoughts about wording criteria? 16:51:28 that's the other end you can come at the problem from 16:51:48 if someone can come up with wording for the final release criterion that we'd be happy with, we can then work from the angle of writing matrices that back that 16:52:25 I can help with criteria wording 16:53:01 adamw: think about installing to LVM thinp on RAID6 and making that bootable. is that sane? 16:53:02 * satellit_e will base server and workstation have different anacondas? with different tests? 16:53:05 does anyone do that? 16:53:29 what about making bootable raid4? huh? why? 16:53:45 there's a lot of stuff in custom that's just, it's kooky 16:54:09 well, custom just gives you a bunch of options - right? 16:54:27 there aren't a ton of conditional decision trees for what kinds of configurations you can do 16:54:31 are there? 16:54:38 there aren't trees, no 16:54:40 yeah well i think they should be yanked and just not there 16:54:43 or or or 16:54:48 there, jus tplant some trees 16:54:58 cmurf: the problem with that is we're then maintaining a list of what's 'sane' and what isn't... 16:55:00 new boot parameter 'crazy' which gets you the current Manual Partitioning functionality 16:55:28 and we're tied to this whole "has to boot" thing, right? 16:55:32 but by default, no effin raid4 option, no raid6 option, not LVM on raid option, no LUKS on LVM on RAID6 option 16:55:41 If we're deciding what's sane, who gets to decide if we are? 16:55:51 i mean JESUS, there's a reason why giant ass Microsoft and Apple don't do *anything* like this 16:55:56 we do, danofsatx 16:55:58 they couldn't evne QA this 16:56:02 they wouldn't want to 16:56:16 roshi right - you offer it, it has to work 16:56:21 so i'm like, let's just chop chop chop 16:56:23 well, to be fair - they have a hard time QA'ing the things they *mean* to release 16:56:46 yeah, I'm a personal fan of all M$'s hidden "features" 16:57:09 cmurf: I was being facetious about the boot thing, as in "does Fedora *have* to boot? What if it just partially boots?" 16:57:18 dracut shell? 16:57:19 pass 16:57:30 roshi: it's on my list to take a look at some of the things in the criteria that got...unexpected interpretations in f20 cycle and see if i can tighten them up 16:57:31 technically it booted, which constitutes the kernel and initramfs working 16:57:43 it was clear from some of the blocker meetings that there are reasonable interpretations which i didn't really consider in the wording 16:57:52 and by booting, does that mean with a working display, or headless? 16:58:01 danofsatx: sure 16:58:19 in my view there's boot, startup, gdm and shell phases 16:58:34 cmurf: are you planning to make a proposal to anaconda along these lines? 16:58:40 dont' forget sddm and lightdm 16:59:04 that makes enough sense to me cmurf 16:59:07 well, sddm is the only other blocker 16:59:14 adamw: i was planning on responding to your email to the anaconda list 16:59:14 adamw: I'll pitch in for the wordings 16:59:37 adamw: but when F18 alpha landed and I looked at newui, I said all of these things 16:59:50 i was like "WTF, seriously raid4, fucking get rid of it" and all I heard were crickets 17:00:15 danofsatx: as of f20 we block on gdm and kdm. 17:00:38 cmurf: you may have a more sympathetic audience now, but i dunno 17:00:45 where would Fedora be without a graphical installer? no where. but then also where could we be without an anaconda that doesn't actually try to eat the user for lunch because it doing way way too much. 17:01:13 F21 is sddm. F20 is kdm. 17:01:16 * satellit_e sddm is on KDE live (rawhide) atm 17:01:17 keep up ;) 17:01:47 before I go to anaconda again, i'd like to better understand what QA people think are sane, maybe sane, and not at all sane, layouts 17:02:03 http://fedoraproject.org/wiki/Changes/SDDMinsteadOfKDM 17:02:38 danofsatx: i said 'as of f20'. 17:03:03 I know - but the time for blocking F20 is done. 17:03:06 anyhoo 17:03:11 I think custom partitioning is what it is and all we can really focus on are a few test cases within that subset, and hopefully all guided options working as intended and that's about it. 17:03:22 cmurf: i'm not entirely sure it's QA's call, but i think you're the one with the most experience in that line anyhow 17:03:32 but if you post a mail to the list asking for input i'll contribute 17:03:51 #info cmurf working on a mail to anaconda list proposing a restriction on the breadth of custom partitioning coverage 17:04:58 so i think we're still kinda working on this but at least we're getting somewhere... 17:05:09 we're over time, though, so moving on 17:05:11 we have until september apparently 17:05:11 #topic Open floor 17:05:14 hah. 17:05:26 well, we have until alpha tc1 to make major changes to validation 17:05:34 yes i'm sorta kidding 17:05:42 which is what anywhere from april to august? 17:05:42 it's not usually a good idea to mess with the validation process after that (and it angers the viking) 17:05:50 danofsatx: april to $WHO_KNOWS 17:06:05 I'm working on the wiki with danofsatx to make the onboarding process a bit smoother 17:06:22 the working doc is here: https://fedoraproject.org/wiki/User:Roshi/QA/Join 17:06:24 so, like I said last week - expected ETA of F21 sometime in 2016 17:06:26 If anything I'd like storage validation stuff to be able to affect Rawhide before branch occurs. :-\ 17:06:44 once I get it ironed out a bit more I'll send an email to test@ for feedback :) 17:06:48 roshi: danofsatx: awesome, thank you guys 17:07:02 well, there will be an F21 in 2014, we just don't know if it's F20+1 or if it's Fedora.next. 17:07:09 #info roshi and danofsatx working on a draft for an improved onboarding process at https://fedoraproject.org/wiki/User:Roshi/QA/Join 17:07:25 and then still with the "test maps" which I think can alleviate some issues with testing the different products for the WGs 17:07:35 and I'm trying to figure out a better method of categorizing/finding the test cases. Right now, it's cryptic at best. 17:08:17 I also roped danofsatx into test maps and decrypting testcases 17:08:21 danofsatx: in my mind we're kind of unusual for a QA group in that things are mostly centered around our processes and our test cases are aids to those, rather than the test cases being the centre of everything 17:08:25 er, what he said 17:08:25 * satellit_e I like the #1-#3 prioriitys for test cases 17:08:27 but maybe that doesn't make sense to anyone else... 17:08:40 i rarely wake up and think 'hey, i'll go find a test case', y'know 17:09:34 adamw: what do you think of working with docs team to specifically point out QA recommended installation layouts? 17:09:37 ...but what else is there adamw ? 17:09:43 it's not that, but when one of you senior guys say "this is checked in test case foo_widget partitioning", and newbies like roshi and I can't find said test case to see exactly how it's laid out 17:10:18 cmurf: some kind of guidance in the install guide seems like a good idea, yeah, and docs folks are easy to work with 17:10:25 hah, 'senior' 17:10:48 k, "more experiences in the QA process for Fedora" 17:10:50 adamw: like a concise summary of the storage validation test matrix, that steers people to our #1 list of layouts and at least informs them that QA isn't testing 'crazy' 17:10:50 danofsatx: open the matrix and ctrl-f, jeez! ;) 17:11:10 cmurf: i think it'd need to be a bit more abstracted from strict QA considerations, but i like the general idea 17:11:35 danofsatx: does experience count if we drink it all away? 17:11:55 #info cmurf suggests working with docs folks to highlight 'recommended' custom layouts 17:11:55 that's an xp boost, double xp weekend adamw 17:12:30 * danofsatx done killed those brain cells long, long ago (in a galaxy far, far away....) 17:14:59 okay, any other business? any topic areas I missed that we should be chattin' about right now? 17:15:51 uh, Merry Christmas/Happy ? 17:15:58 +1 17:16:22 meeting next week? 17:16:32 when is that even? 17:17:06 yeah i expect nothing on my emails for docs and anaconda until after the holiday 17:17:07 s 17:17:08 a very genial winter solstice to all indeed 17:17:12 when is what? next week, is well, next week.... 17:17:18 (don't matter who you are, everyone celebrates the solstice) 17:17:30 we could skip next week's meeting possibly 17:17:33 adamw: i did not, i celebrated the day after 17:17:41 I celebrate winter Lagers and Christmas Ales :) 17:17:47 for the more recent folks - there is a provision in the meeting SOP to skip meetings on weeks when there's nothing much that needs discussing 17:17:47 which is "hell yeah, it's no longer getting any darker!" 17:17:53 hehe 17:18:13 yesterday i went snowboarding to celebrate 17:18:24 so if it looks like being a slow one next week i might send out a mail proposing we skip the meeting, and if anyone really wants to have one (ahahahahaha) they get to object 17:18:26 14" in 48 hours 17:18:31 cmurf: rrrrrrr 17:18:36 I'm fine whatever for the meeting next week 17:18:42 only hit 5 rocks! 17:18:51 (and no core shots) 17:19:09 cmurf: still mostly artificial on the vancouver mountains :( 17:19:14 that's why i've been getting more work done than usual :P 17:19:25 snow? 17:19:30 yeah 17:19:34 * danofsatx lives in South Texas. 17:19:37 that sucks work is overrated, 14" of pow is never overrated 17:19:49 indeed 17:19:50 esp on a board 17:19:53 alrighty, thanks for coming folks 17:19:54 * satellit_e bend is too warm - snow is melting : ( 17:20:04 same time next week or not, depending on interest and how drunk adamw is 17:20:17 snow sports discussion to continue in #fedora-qa ;) 17:20:19 #endmeeting