18:01:14 <smooge> #startmeeting Infrastructure (2018-01-11) 18:01:14 <zodbot> Meeting started Thu Jan 11 18:01:14 2018 UTC. The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:01:14 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:01:14 <zodbot> The meeting name has been set to 'infrastructure_(2018-01-11)' 18:01:14 <smooge> #meetingname infrastructure 18:01:14 <zodbot> The meeting name has been set to 'infrastructure' 18:01:14 <smooge> #topic aloha 18:01:14 <smooge> #chair smooge relrod nirik dgilmore threebean pingou puiterwijk pbrobinson maxamillion 18:01:14 <zodbot> Current chairs: dgilmore maxamillion nirik pbrobinson pingou puiterwijk relrod smooge threebean 18:01:21 <nirik> morning everyone. 18:01:21 * pingou here 18:01:34 <puiterwijk> hi 18:01:43 * relrod here 18:01:46 <cverna> morning o/ 18:01:57 <smooge> hello all 18:02:08 <bowlofeggs> .hello2 18:02:09 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <randy@electronsweatshop.com> 18:02:24 <bowlofeggs> it's time 2 partay 18:02:49 * bowlofeggs pops a bottle of infrastructure champagne 18:03:15 * threebean waves 18:03:15 * nirik is still having infrastructure coffee. 18:04:13 <smooge> #topic New folks introductions 18:04:13 <smooge> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves 18:04:25 <smooge> hello any new people this week? 18:04:29 * willo says ello 18:04:46 <ajayvembu> hi all 18:04:55 <ajayvembu> This is Ajay 18:05:09 <smooge> hello ajayvembu 18:05:14 <smooge> how are you? 18:05:41 <bowlofeggs> nirik: is infrastructure coffee highly scalable? 18:05:55 <willo> hey smooge, I'm returning back to the team 18:06:04 <ajayvembu> to give a short background about me I did my Masters in Computer Science and currently working in Informatica mostly related to Linux and Hadoop administration 18:06:11 <willo> so newish, again... :) 18:06:21 <nirik> it's strong and highly available. :) 18:06:23 <ajayvembu> hi Smooge I am good how are you? 18:06:33 <nirik> welcome ajayvembu 18:06:36 <nirik> and welcome back willo 18:06:51 <willo> hey thanks nirik 18:06:56 <smooge> ajayvembu, cool 18:06:57 <ajayvembu> hi Nirik 18:07:23 <smooge> #topic announcements and information 18:07:23 <smooge> #info Lots of reboot been and upcoming for various CVEs - puiterwijk/kevin/smooge/relrod 18:07:23 <smooge> #info taskotron stg and prod redeployed as F27 - tflink 18:07:31 <ajayvembu> I am looking forward to work on some exciting Fedora projects 18:08:09 <smooge> i think we may have a need for someone with knowledge about distributed filesystems in the coming year. 18:08:49 <ajayvembu> sure sounds good. I am eager to work on such projects 18:08:55 <smooge> cool 18:09:22 <smooge> So on announcements, we are still in "Oh what new problem is coming up with Meltdown/Spectre" 18:09:30 <smooge> puiterwijk, has been doing a lot of work on that 18:09:43 <puiterwijk> More reboots are going to be inbound tonight 18:10:04 <nirik> there's some spectre stull landing in fedora kernels soon (at least rawhide) 18:10:07 <pingou> could we stop apache on pkgdb* before rebooting the db? 18:10:19 <pingou> somehow it's the one that spams me the most about db being down :s 18:10:29 <nirik> sure, can try and remember. 18:10:43 <pingou> if you don't no biggy, I'll just know the db was rebooted :D 18:10:54 <puiterwijk> db01 *will* reboot today 18:11:18 <nirik> puiterwijk: hopefully then after today not for a few days at least? 18:11:26 <puiterwijk> nirik: Hopefully... 18:11:36 <nirik> well, I was thinking of migrating it. 18:11:42 <nirik> which likely will take a while... 18:11:46 <puiterwijk> I'm currently preparing some things to hopefully make it not need a lot more reboots 18:11:58 <puiterwijk> nirik: you wanted to move it to vhost06, right? 18:12:08 <nirik> yeah 18:12:49 <puiterwijk> Right. Let's see there. We should not need any more host reboots soon 18:13:08 <nirik> for everyone who doesn't know what we are talking about here: 18:13:39 <nirik> https://spectreattack.com/ and 18:13:49 <puiterwijk> https://meltdownattack.com/ 18:14:17 <nirik> looking for the rh access site... it's got a good overview 18:14:27 <bowlofeggs> https://bowlofattack.com/ 18:14:28 <puiterwijk> https://access.redhat.com/security/vulnerabilities/speculativeexecution 18:15:06 <nirik> yeah, that one. 18:15:13 <nirik> bowlofeggs: don't throw eggs at us. ;) 18:15:25 <bowlofeggs> haha 18:15:28 <tflink> bowlofeggs: wait, you're behind all this? 18:15:44 <bowlofeggs> tflink: i can neither confirm nor disconfirm that suspicion 18:16:03 <bowlofeggs> i was an electrical engineer in a former life… 18:16:17 <nirik> also... if the dry stuff doesn't work for you, I found this one a very good way to explain meltdown: 18:16:23 <nirik> https://twitter.com/securelyfitz/status/949004862968143873 18:16:38 <ajayvembu> sounds good. I will check it out 18:16:40 <ajayvembu> thanks 18:16:42 <puiterwijk> Oh, I liked the XKCD: https://xkcd.com/1938/ 18:17:05 <tflink> "the cloud is full of phantom trolleys armed with hammers" 18:17:16 <nirik> yeah. ;) 18:17:24 <Southern_Gentlem> https://twit.tv/shows/security-now/episodes/645 18:17:26 <nirik> also, the xbox360 one from years ago is cool. 18:17:49 <nirik> https://randomascii.wordpress.com/2018/01/07/finding-a-cpu-design-bug-in-the-xbox-360/ 18:18:13 <nirik> anyhow... we can move along I think 18:20:11 <smooge> ok 18:20:25 <smooge> #topic Ticket cleanup 18:20:25 <smooge> #info none this week. 18:20:25 <smooge> #topic Upgrading/Migrating Jenkins 18:20:25 <smooge> #info: Working with Pagure-CI triggers 18:20:25 <smooge> #info A Pagure SCM Plugin for Jenkins 18:20:35 <smooge> sorry my mouse kept pasting nothing 18:21:30 <smooge> was the Jenkins from last week? 18:21:44 <puiterwijk> I think so, yes 18:21:45 <nirik> yeah, both those were. 18:21:49 <smooge> welp. 18:22:04 <smooge> #topic Apprentice Open office hours 18:22:05 <nirik> jenkins is in progress now... pingou sent out an announcement and bstinson has been working on setting up the centos side 18:22:33 <nirik> I kind of wanted to talk about the apprentice program for a few if we can... 18:22:52 <smooge> #topic Apprentice Program 18:23:12 <smooge> ok nirik your floor 18:23:15 <x3mboy> Hi 18:23:41 <nirik> There were some requests before the holidays to do a more mentor approach... but I still think thats just not going to scale for us. 18:24:12 <nirik> but also, perhaps we should get people thinking that the entire group is their mentor... I think everyone is happy to answer questions as their time permits. 18:24:53 <x3mboy> I was an apprentice before, I want to re apply 18:25:10 <nirik> I have been sending out emails at the first of the month, then removing people who don't respond, but that's been somewhat tedious and also the email is the same everytime... so when someone is in the group for a while they keep answering the same email over and over. 18:25:21 <nirik> x3mboy: cool. welcome back 18:25:41 <x3mboy> nirik, thanks 18:26:16 <nirik> and we continue to have a hard time getting folks to do the next step after joining apprentice group and setting up ssh... ie, working on issues or helping with something. Not sure how we can bridge that gap, but very open to ideas. 18:26:28 <ajayvembu> how this Apprentice program exactly works? 18:26:42 <ajayvembu> we can ping anybody in the infrastucture group? 18:27:01 <pingou> pretty much 18:27:13 <pingou> worst case they will be able to point you to the right person 18:27:14 <nirik> https://fedoraproject.org/wiki/Infrastructure_Apprentice 18:27:24 <nirik> and 18:27:29 <willo> nirik: I've been thinking about that last bit exactly 18:27:30 <nirik> https://fedoraproject.org/wiki/Infrastructure/GettingStarted 18:27:48 <cverna> would it help to have something like office hours ? 18:27:54 <ajayvembu> okay will check thanks 18:28:07 <nirik> well, we tried that... but didn't seem to bring in too many people... 18:28:13 <nirik> but we may have chosen them poorly 18:28:20 <nirik> (the hours) 18:28:57 <cverna> also I think having the ansible repo on pagure would make it a little bit easier 18:29:17 <willo> Returning with basically fresh eyes has been a bit unique this time. 18:29:25 <nirik> perhaps. It would allow PR's... but I mean someone can send a patch about anything anywhere. 18:29:59 <cverna> nirik: that's why I said a little bit :) 18:30:14 <nirik> sure. would be good I agree. :) 18:30:16 * pingou needs to finish the mirror service 18:30:30 <bowlofeggs> one thing difficult about the ansible stuff is that it's hard to try a patch without it being committed 18:30:46 <bowlofeggs> i don't know a solution to that, just an observation :) 18:30:54 <willo> Some of the GettingStarted docs needs some small tweaks, there 18:30:55 <bowlofeggs> might just be how it is 18:31:28 <nirik> yeah, sometimes you can commit just for stg and test whatever there... but yeah, it's hard to test without being able to iterate 18:31:48 <cverna> bowlofeggs: maybe with the PR we could run the detached PR on stg and merge when it works 18:32:21 <bowlofeggs> that might work for some cases - bodhi's code has a lot of "if stg, else" in it though 18:33:07 <cverna> On my side I would like try to share a little bit what I am doing or will be doing, since I am currently trying to get my head around OSBS I was thinking doing an article in the magazine about OSBS. 18:33:21 <willo> I think knowledge on how the systems interact make submitting changes to ansible seem a daunting at first 18:33:22 <nirik> always good... 18:33:29 <pingou> cverna: feel free to document things in the infra-docs repo as well 18:33:31 <willo> of someones lack of knowledge 18:33:42 <cverna> pingou: +1 18:34:04 <nirik> willo: sure. we were all there at one time tho, so asking can usually find someone who knows how things are interacting. 18:34:40 <bowlofeggs> the secret is that patrick knows everything 18:34:51 <pingou> shhht don't tell it 18:34:52 <willo> lol 18:34:53 <ajayvembu> guys any links on Bodhi? 18:34:57 <pingou> now everyone knows our secret! 18:35:06 <cverna> :) 18:35:17 <pingou> http://bodhi.fedoraproject.org/ 18:35:24 <nirik> what do folks think of the administravia stuff? sending emails and removing inactive people? I like not having any inactive folks in there, but wish there was a better way 18:36:10 <ajayvembu> will check thanks 18:36:13 <bowlofeggs> ajayvembu: http://bodhi.fedoraproject.org/docs has more info too 18:36:18 <pingou> outside of coding this somehow :s 18:36:28 * puiterwijk notes that both pingou and bowlofeggs send http links :( 18:36:33 <bowlofeggs> haha 18:36:41 <bowlofeggs> i just copied pingou's link and added docs 18:36:50 <bowlofeggs> i'm innocent i swear! 18:36:50 <pingou> puiterwijk: used browser history w/o pressing enter 18:36:52 <puiterwijk> Please use https://bodhi.fedoraproject.org/ and https://bodhi.fedoraproject.org/docs 18:37:02 <ajayvembu> okay 18:37:06 * bowlofeggs throws pingou under the fedmsg bus 18:37:14 <pingou> thanks man 18:37:24 * smooge gives bowlofeggs of eggs the mop to clean that up 18:37:24 * pingou get messages over by the bodhi messages 18:37:26 <nirik> the wheels on the bus go... 18:37:37 <puiterwijk> pingou: if you hadn't, I'd be really concerned and call you right now, since that would mean you're under attack :) (if you pasted it from a browser that followed all redirects) 18:37:39 <bowlofeggs> nirik: is it possible to see when people last logged in as a way to trim the membership? 18:38:01 <nirik> bowlofeggs: we could yeah... just a last on batcave01 or whatever 18:38:12 <smooge> a last on Jan 1st :) 18:38:19 <x3mboy> ! 18:38:31 <x3mboy> Can I suggest something that may sound a little crazy? 18:38:32 <puiterwijk> x3mboy: you can just speak, no need to ask 18:38:37 <willo> from apprentice POV, the email seem fine, but it did feel a bit. I'm putting the same info in these questions at lot. I wonder how useful it is. 18:38:45 <x3mboy> Why not to interact with apprentice via telegram? 18:38:46 <bowlofeggs> nirik: that might be less overhead and be reasonable (and automatable!) 18:39:05 <smooge> because that would require us to have telegram accounts? 18:39:07 <nirik> bowlofeggs: good thought. 18:39:16 <nirik> x3mboy: I don't think any of us use telegram. ;) 18:39:21 <x3mboy> It's being a great way to work in several teams: Diversity, CommOps, Ambassadors Latam, and others 18:39:43 <x3mboy> nirik, ok, was just a suggestion ;) 18:39:55 <cverna> x3mboy: what is the plus compare to irc ? 18:39:56 <puiterwijk> x3mboy: one problem would be that you'd be entirely out of the loop, since basically all infra stuff happens and gets discussed on IRC 18:40:01 <pingou> bstinson: just got stg.pagure.io to kick a build in ci.centos.org :) 18:40:09 <smooge> x3mboy, this has come up with a lot of different social technologies in the last 10 years... usually we have 1 window to focus on and by the time we get onto whatever technology.. everyone is wanting us to be on something else. 18:40:15 <pingou> cverna: it's where the new kids are :) 18:40:21 <cverna> bstinson++ pingou++ 18:40:21 <zodbot> cverna: Karma for bstinson changed to 1 (for the f27 release cycle): https://badges.fedoraproject.org/tags/cookie/any 18:40:22 <bstinson> pingou: thanks for the debugging session :) 18:40:23 <zodbot> cverna: Karma for pingou changed to 13 (for the f27 release cycle): https://badges.fedoraproject.org/tags/cookie/any 18:40:30 <willo> "automating the email" in a simple web app as mentioned might be good. 18:40:31 <puiterwijk> bstinson++ 18:40:31 <zodbot> puiterwijk: Karma for bstinson changed to 2 (for the f27 release cycle): https://badges.fedoraproject.org/tags/cookie/any 18:40:50 <pingou> bstinson++ 18:40:50 <zodbot> pingou: Karma for bstinson changed to 3 (for the f27 release cycle): https://badges.fedoraproject.org/tags/cookie/any 18:41:04 * pingou amazed he didn't do that sooner 18:41:05 <smooge> x3mboy, so it wasn't a crazy idea.. just one we have had problems getting onto fast enough to ever do. 18:41:30 <willo> it could then pull in activity to providing a summary of what's done, across pagure tickets as well as the login activity as mentioned and others 18:41:54 <nirik> well, I can try this month to just mail everyone and tell them to make sure and login to batcave01, if they don't they will be removed... we can give it a whirl. If they have feedback, start a thread on the list. 18:41:57 <x3mboy> Ok, I'm just telling because even communications to flock happens very fluid, but I understand the POV of don't having enough time to look through several techs 18:42:46 <pingou> bstinson: we weren't that far, just two typos :) 18:43:29 <smooge> x3mboy, understood. 18:43:40 <bstinson> now off to do the rest of the infra work 18:43:42 <x3mboy> puiterwijk, the IRC is not a problem, we use a bridge IRC<->Telegram 18:43:42 <puiterwijk> x3mboy: also, one problem I'd personally have with telegram is that telegram is on my phone, and doing infra stuff on a phone is... painful. And there's no telegram clients that I know of for Fedora in the main repos 18:43:43 <x3mboy> :D 18:43:59 <puiterwijk> x3mboy: yeah, but from the IRC side, that bridge is really painful to use 18:45:02 <x3mboy> Ok, I get it 18:45:22 <puiterwijk> Anyway, not a bad idea, just trying to explain why I'd personally think it's not the best for infra 18:45:25 <nirik> we can probibly move on now unless someone has something further on this? 18:45:33 <x3mboy> Nope 18:45:50 * nirik wanted to get everyone thinking about it... feel free to send ideas later too. 18:45:50 <willo> no, neither do I 18:46:08 <smooge> I think that cleaning out a lot of the sysadmin groups would be a good idea in the coming year. 18:46:24 <cverna> maybe we need to do some screencast :) 18:46:27 <smooge> so that all souns find. 18:46:36 <pingou> smooge: +1 18:46:48 <marc84> smooge +1 18:46:49 <smooge> I need to go take my Geritol(TM) and my prune juice though 18:46:52 * pingou can't imagine puiterwijk disagreeing :D 18:46:57 <pingou> smooge++ 18:47:05 * puiterwijk totally agrees with smooge and has already said so before :) 18:47:06 <nirik> cverna: I thought of that... just never get to it. 18:47:15 <smooge> #topic Open Floor 18:47:19 <nirik> smooge++ 18:47:20 <zodbot> nirik: Karma for smooge changed to 6 (for the f27 release cycle): https://badges.fedoraproject.org/tags/cookie/any 18:47:29 <smooge> ok we have reached the open floor part of this meeting 18:47:32 <puiterwijk> smooge++ 18:47:35 <nirik> woah, I haven't given smooge a cookie yet? weird. 18:47:49 <bstinson> quick update on the Jenkins bits: 18:47:50 <nirik> smooge: you ok to keep running meetings? or should we rotate/trade off more? 18:47:51 <smooge> oh everyone agrees I need to take my geritol and prune juice? Damn I must be cranky 18:48:15 <smooge> Oh yeah.. first bstinson then schedules 18:48:17 <bowlofeggs> worf drinks prune juice too 18:48:37 <smooge> #topic bstinson talks about Jenkins 18:48:38 <bstinson> i kind of like prune juice... 18:48:43 <bstinson> but in any case 18:48:57 <bowlofeggs> bstinson++ for giving us awesome jenkins jenkinsness 18:48:57 <zodbot> bowlofeggs: Karma for bstinson changed to 4 (for the f27 release cycle): https://badges.fedoraproject.org/tags/cookie/any 18:49:06 <bstinson> - the static agents are done, i took the ansible playbooks in infrastructure.fp.o 18:49:22 <bstinson> - i'm fiddling a bit with the custom master image that we'll deploy in openshift 18:49:38 <bstinson> - Deadline: Tomorrow for a trial migration with pingou 18:49:57 <bstinson> then we can talk about what the retirement process looks like 18:50:18 <pingou> \ó/ 18:50:51 <bstinson> Unresolved questions: 18:51:20 <smooge> - Prune juice vs raisin juice 18:51:30 <pingou> vs apple juice/ 18:51:32 <pingou> ? 18:51:34 <bstinson> - The agent setup playbooks are on our end, we'll maintain those in sync with what's in infrastructure.fp.o 18:51:41 <pingou> or are you speaking about fermented raisin juice? 18:51:42 <bstinson> oh apple juice for sure 18:51:52 <bowlofeggs> yeah what is prune juice if a prune is just a dried plum? 18:52:00 <bowlofeggs> if you dry it, how can it have juice 18:52:08 <willo> pineapple juice has my vote 18:52:14 <smooge> ok sorry 18:52:16 <bowlofeggs> pineapple is so good 18:52:19 <smooge> I really derailed this 18:52:22 <bowlofeggs> hahaha 18:52:26 <pingou> ^^ 18:52:34 <willo> smooge++ 18:52:34 <zodbot> willo: Karma for smooge changed to 7 (for the f27 release cycle): https://badges.fedoraproject.org/tags/cookie/any 18:52:35 <bstinson> that cran-apple stuff they have on planes is vile though 18:52:35 <smooge> any other issues bstinson 18:52:37 <pingou> bstinson: keep in sync, so release by release? 18:52:52 <pingou> or is it updated more often? 18:52:55 <smooge> more like how does one run playbooks 18:52:57 <bowlofeggs> on planes i like to ask for a glass of landing gear grease 18:53:14 <bstinson> pingou: as-in any changes will be made in infra.fp.o, and we'll pull those into our internal repo to update the agents 18:53:41 <pingou> bstinson: ok, I'm wondering if we can find a way to make sure we don't forget 18:53:45 <nirik> there is a ticket to add some packages to some of the slaves... 18:53:54 <nirik> (a new ticket filed this morning) 18:53:55 <pingou> potentially to automate it even 18:53:58 <bstinson> nirik: can i take that assignment? 18:54:10 <nirik> sure... let me find it 18:54:16 <pingou> that will no longer be an issue with the dynamic agents no? 18:54:20 <bstinson> pingou: hopefully we don't need to maintain that sync very long 18:54:33 <bstinson> dynamic agents will solve this problem 18:54:38 <pingou> coolio 18:54:42 <nirik> https://pagure.io/fedora-infrastructure/issue/6630 18:55:11 * pingou will try to make some notes on how to do thing 18:55:18 <pingou> might be nice to blog about that 18:55:24 <bstinson> last open question: 18:55:36 <smooge> ok 18:56:02 <bstinson> - i'll push this s2i template for the jenkins master to a personal pagure repo, do we want it to live somewhere 'official'? 18:56:46 <nirik> as long as it's somewhere should be fine.. 18:56:53 <pingou> pagure has no concept of personal project (yet) :) 18:56:58 <nirik> it would at least be in the openshift instance. ;) 18:58:01 <bstinson> ok so 'somewhere' is good enough. i'll be sure to share that with the infra group 18:58:08 <smooge> thanks 18:58:11 * pingou has one question for after smooge 's topic 18:58:20 <smooge> one sec. 18:58:30 <smooge> do we have a meeting here in 2 minutes or can we run over? 18:58:59 <smooge> bstinson, anything else? 18:59:35 <smooge> #topic Schedule for upcoming meetings 18:59:37 <bstinson> that's it for me 18:59:49 <smooge> OK I am on PTO next week 19:00:18 <smooge> the week after that the 25th people will be going to Brno for meetings 19:00:30 <smooge> I can run a meeting that week but expect not a lot of communication 19:00:47 <relrod> I can probably do next week's 19:00:47 <smooge> the 1st may be similar as people who are doing FOSDEM head there 19:00:52 <smooge> thanks relrod 19:01:06 <pingou> February 19:01:15 * pingou read May 1st 19:01:17 <pingou> .... 19:01:20 <smooge> hahahaha 19:01:27 <smooge> I am planning way ahead 19:02:09 <puiterwijk> May 25th? :P 19:02:12 <smooge> ok so February 1st I will run but expect not a lot so it will probably be a "these are the fires and these are all the things who got to go to nice places are now assigned." 19:02:14 <nirik> new years party 2018? 19:02:32 <smooge> and that is it for upcoming meetings 19:02:44 <smooge> #topic Pingou's question (the answer is not 42) 19:03:03 <pingou> any thoughts on upgrading nuancier in prod? 19:03:18 <pingou> the submission phase started 9 days ago, there are been 4 submissions since 19:03:24 <pingou> the outage should be under 5 minutes 19:03:27 <bowlofeggs> 42, 19:03:44 <pingou> and I kinda think having this in before the fedora magazine article comes out would be good 19:03:54 <puiterwijk> pingou: what's the risk? (i.e. what's the change of size) 19:04:27 <bowlofeggs> some people were upset about how the last election round went 19:05:00 <pingou> puiterwijk: py3 compatibility, some UI changes and most importantly: a new date allowing to give some time between the end of the submissions and the start of the voting 19:05:14 <nirik> pingou: FYI, ci-cc-rdu01 needs upgrading/reinstalling with new os. 19:05:37 <puiterwijk> pingou: okay. How big do you guess the chance is that it'll fail and burn, is what I am basically interested in :) 19:05:48 <pingou> to avoid one issue we had last time: someone submitted something just before the deadline, could not be reviewed in time and thus didn't get included in the votes 19:05:53 <pingou> and that made a drama 19:06:02 <pingou> puiterwijk: little I expect 19:06:09 <pingou> the new field is trivial 19:06:10 <puiterwijk> pingou: then I'd say schedule an outage 19:06:18 * nirik nods. agree 19:06:28 <pingou> the py2/py3 change is bigger but unit-tests seem all happy 19:06:32 <pingou> okido, on it :) 19:06:58 <pingou> eof for me 19:07:04 <smooge> ok thanks 19:07:10 <smooge> #topic Open Floor 19:07:24 <smooge> anything left or should I stick a fork in this meeting and call it done? 19:07:26 <bowlofeggs> ok now let's figure out the juice issue 19:07:41 <nirik> apple juice is the best juice. 19:07:50 <pingou> bstinson: was clearly for apple, so is nirik apparently 19:08:03 <bowlofeggs> pomegranite 19:08:10 <willo> pineapple for the win 19:08:15 <pingou> I kinda like fermented raisin juice 19:08:16 <puiterwijk> Did anyone consider orange juice? 19:08:23 <pingou> puiterwijk: way too mainstream :D 19:08:24 <bowlofeggs> it's a pain to eat the fruit, but if someone else extracts the seeds/juice for you it is amazing 19:08:33 <cverna> apple + carrots juice 19:08:34 <puiterwijk> pingou: oh, and apple juice isn't? :O 19:08:42 <willo> oh, bowlofeggs: new contender from left field 19:08:45 <bstinson> oooh pomegranate juice turns into grenadine... 19:08:50 <pingou> puiterwijk: you're comparing orange and apple there man :D 19:08:59 <smooge> #endmeeting