18:01:14 <smooge> #startmeeting Infrastructure (2018-01-11)
18:01:14 <zodbot> Meeting started Thu Jan 11 18:01:14 2018 UTC.  The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:01:14 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:01:14 <zodbot> The meeting name has been set to 'infrastructure_(2018-01-11)'
18:01:14 <smooge> #meetingname infrastructure
18:01:14 <zodbot> The meeting name has been set to 'infrastructure'
18:01:14 <smooge> #topic aloha
18:01:14 <smooge> #chair smooge relrod nirik dgilmore threebean pingou puiterwijk pbrobinson maxamillion
18:01:14 <zodbot> Current chairs: dgilmore maxamillion nirik pbrobinson pingou puiterwijk relrod smooge threebean
18:01:21 <nirik> morning everyone.
18:01:21 * pingou here
18:01:34 <puiterwijk> hi
18:01:43 * relrod here
18:01:46 <cverna> morning o/
18:01:57 <smooge> hello all
18:02:08 <bowlofeggs> .hello2
18:02:09 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <randy@electronsweatshop.com>
18:02:24 <bowlofeggs> it's time 2 partay
18:02:49 * bowlofeggs pops a bottle of infrastructure champagne
18:03:15 * threebean waves
18:03:15 * nirik is still having infrastructure coffee.
18:04:13 <smooge> #topic New folks introductions
18:04:13 <smooge> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
18:04:25 <smooge> hello any new people this week?
18:04:29 * willo says ello
18:04:46 <ajayvembu> hi all
18:04:55 <ajayvembu> This is Ajay
18:05:09 <smooge> hello ajayvembu
18:05:14 <smooge> how are you?
18:05:41 <bowlofeggs> nirik: is infrastructure coffee highly scalable?
18:05:55 <willo> hey smooge, I'm returning back to the team
18:06:04 <ajayvembu> to give a short background about me I did my Masters in Computer Science and currently working in Informatica mostly related to Linux and Hadoop administration
18:06:11 <willo> so newish, again... :)
18:06:21 <nirik> it's strong and highly available. :)
18:06:23 <ajayvembu> hi Smooge I am good how are you?
18:06:33 <nirik> welcome ajayvembu
18:06:36 <nirik> and welcome back willo
18:06:51 <willo> hey thanks nirik
18:06:56 <smooge> ajayvembu, cool
18:06:57 <ajayvembu> hi Nirik
18:07:23 <smooge> #topic announcements and information
18:07:23 <smooge> #info Lots of reboot been and upcoming for various CVEs - puiterwijk/kevin/smooge/relrod
18:07:23 <smooge> #info taskotron stg and prod redeployed as F27 - tflink
18:07:31 <ajayvembu> I am looking forward to work on some exciting Fedora projects
18:08:09 <smooge> i think we may have a need for someone with knowledge about distributed filesystems in the coming year.
18:08:49 <ajayvembu> sure sounds good. I am eager to work on such projects
18:08:55 <smooge> cool
18:09:22 <smooge> So on announcements, we are still in "Oh what new problem is coming up with Meltdown/Spectre"
18:09:30 <smooge> puiterwijk, has been doing a lot of work on that
18:09:43 <puiterwijk> More reboots are going to be inbound tonight
18:10:04 <nirik> there's some spectre stull landing in fedora kernels soon (at least rawhide)
18:10:07 <pingou> could we stop apache on pkgdb* before rebooting the db?
18:10:19 <pingou> somehow it's the one that spams me the most about db being down :s
18:10:29 <nirik> sure, can try and remember.
18:10:43 <pingou> if you don't no biggy, I'll just know the db was rebooted :D
18:10:54 <puiterwijk> db01 *will* reboot today
18:11:18 <nirik> puiterwijk: hopefully then after today not for a few days at least?
18:11:26 <puiterwijk> nirik: Hopefully...
18:11:36 <nirik> well, I was thinking of migrating it.
18:11:42 <nirik> which likely will take a while...
18:11:46 <puiterwijk> I'm currently preparing some things to hopefully make it not need a lot more reboots
18:11:58 <puiterwijk> nirik: you wanted to move it to vhost06, right?
18:12:08 <nirik> yeah
18:12:49 <puiterwijk> Right. Let's see there. We should not need any more host reboots soon
18:13:08 <nirik> for everyone who doesn't know what we are talking about here:
18:13:39 <nirik> https://spectreattack.com/ and
18:13:49 <puiterwijk> https://meltdownattack.com/
18:14:17 <nirik> looking for the rh access site... it's got a good overview
18:14:27 <bowlofeggs> https://bowlofattack.com/
18:14:28 <puiterwijk> https://access.redhat.com/security/vulnerabilities/speculativeexecution
18:15:06 <nirik> yeah, that one.
18:15:13 <nirik> bowlofeggs: don't throw eggs at us. ;)
18:15:25 <bowlofeggs> haha
18:15:28 <tflink> bowlofeggs: wait, you're behind all this?
18:15:44 <bowlofeggs> tflink: i can neither confirm nor disconfirm that suspicion
18:16:03 <bowlofeggs> i was an electrical engineer in a former life…
18:16:17 <nirik> also... if the dry stuff doesn't work for you, I found this one a very good way to explain meltdown:
18:16:23 <nirik> https://twitter.com/securelyfitz/status/949004862968143873
18:16:38 <ajayvembu> sounds good. I will check it out
18:16:40 <ajayvembu> thanks
18:16:42 <puiterwijk> Oh, I liked the XKCD: https://xkcd.com/1938/
18:17:05 <tflink> "the cloud is full of phantom trolleys armed with hammers"
18:17:16 <nirik> yeah. ;)
18:17:24 <Southern_Gentlem> https://twit.tv/shows/security-now/episodes/645
18:17:26 <nirik> also, the xbox360 one from years ago is cool.
18:17:49 <nirik> https://randomascii.wordpress.com/2018/01/07/finding-a-cpu-design-bug-in-the-xbox-360/
18:18:13 <nirik> anyhow... we can move along I think
18:20:11 <smooge> ok
18:20:25 <smooge> #topic Ticket cleanup
18:20:25 <smooge> #info none this week.
18:20:25 <smooge> #topic Upgrading/Migrating Jenkins
18:20:25 <smooge> #info: Working with Pagure-CI triggers
18:20:25 <smooge> #info A Pagure SCM Plugin for Jenkins
18:20:35 <smooge> sorry my mouse kept pasting nothing
18:21:30 <smooge> was the Jenkins from last week?
18:21:44 <puiterwijk> I think so, yes
18:21:45 <nirik> yeah, both those were.
18:21:49 <smooge> welp.
18:22:04 <smooge> #topic Apprentice Open office hours
18:22:05 <nirik> jenkins is in progress now... pingou sent out an announcement and bstinson has been working on setting up the centos side
18:22:33 <nirik> I kind of wanted to talk about the apprentice program for a few if we can...
18:22:52 <smooge> #topic Apprentice Program
18:23:12 <smooge> ok nirik your floor
18:23:15 <x3mboy> Hi
18:23:41 <nirik> There were some requests before the holidays to do a more mentor approach... but I still think thats just not going to scale for us.
18:24:12 <nirik> but also, perhaps we should get people thinking that the entire group is their mentor... I think everyone is happy to answer questions as their time permits.
18:24:53 <x3mboy> I was an apprentice before, I want to re apply
18:25:10 <nirik> I have been sending out emails at the first of the month, then removing people who don't respond, but that's been somewhat tedious and also the email is the same everytime... so when someone is in the group for a while they keep answering the same email over and over.
18:25:21 <nirik> x3mboy: cool. welcome back
18:25:41 <x3mboy> nirik, thanks
18:26:16 <nirik> and we continue to have a hard time getting folks to do the next step after joining apprentice group and setting up ssh... ie, working on issues or helping with something. Not sure how we can bridge that gap, but very open to ideas.
18:26:28 <ajayvembu> how this Apprentice program exactly works?
18:26:42 <ajayvembu> we can ping anybody in the infrastucture group?
18:27:01 <pingou> pretty much
18:27:13 <pingou> worst case they will be able to point you to the right person
18:27:14 <nirik> https://fedoraproject.org/wiki/Infrastructure_Apprentice
18:27:24 <nirik> and
18:27:29 <willo> nirik: I've been thinking about that last bit exactly
18:27:30 <nirik> https://fedoraproject.org/wiki/Infrastructure/GettingStarted
18:27:48 <cverna> would it help to have something like office hours ?
18:27:54 <ajayvembu> okay will check thanks
18:28:07 <nirik> well, we tried that... but didn't seem to bring in too many people...
18:28:13 <nirik> but we may have chosen them poorly
18:28:20 <nirik> (the hours)
18:28:57 <cverna> also I think having the ansible repo on pagure would make it a little bit easier
18:29:17 <willo> Returning with basically fresh eyes has been a bit unique this time.
18:29:25 <nirik> perhaps. It would allow PR's... but I mean someone can send a patch about anything anywhere.
18:29:59 <cverna> nirik: that's why I said a little bit :)
18:30:14 <nirik> sure. would be good I agree. :)
18:30:16 * pingou needs to finish the mirror service
18:30:30 <bowlofeggs> one thing difficult about the ansible stuff is that it's hard to try a patch without it being committed
18:30:46 <bowlofeggs> i don't know a solution to that, just an observation :)
18:30:54 <willo> Some of the GettingStarted docs needs some small tweaks, there
18:30:55 <bowlofeggs> might just be how it is
18:31:28 <nirik> yeah, sometimes you can commit just for stg and test whatever there... but yeah, it's hard to test without being able to iterate
18:31:48 <cverna> bowlofeggs: maybe with the PR we could run the detached PR on stg and merge when it works
18:32:21 <bowlofeggs> that might work for some cases - bodhi's code has a lot of "if stg, else" in it though
18:33:07 <cverna> On my side I would like try to share a little bit what I am doing or will be doing, since I am currently trying to get my head around OSBS I was thinking doing an article in the magazine about OSBS.
18:33:21 <willo> I think knowledge on how the systems interact make submitting changes to ansible seem a daunting at first
18:33:22 <nirik> always good...
18:33:29 <pingou> cverna: feel free to document things in the infra-docs repo as well
18:33:31 <willo> of someones lack of knowledge
18:33:42 <cverna> pingou: +1
18:34:04 <nirik> willo: sure. we were all there at one time tho, so asking can usually find someone who knows how things are interacting.
18:34:40 <bowlofeggs> the secret is that patrick knows everything
18:34:51 <pingou> shhht don't tell it
18:34:52 <willo> lol
18:34:53 <ajayvembu> guys any links on Bodhi?
18:34:57 <pingou> now everyone knows our secret!
18:35:06 <cverna> :)
18:35:17 <pingou> http://bodhi.fedoraproject.org/
18:35:24 <nirik> what do folks think of the administravia stuff? sending emails and removing inactive people? I like not having any inactive folks in there, but wish there was a better way
18:36:10 <ajayvembu> will check thanks
18:36:13 <bowlofeggs> ajayvembu: http://bodhi.fedoraproject.org/docs has more info too
18:36:18 <pingou> outside of coding this somehow :s
18:36:28 * puiterwijk notes that both pingou and bowlofeggs send http links :(
18:36:33 <bowlofeggs> haha
18:36:41 <bowlofeggs> i just copied pingou's link and added docs
18:36:50 <bowlofeggs> i'm innocent i swear!
18:36:50 <pingou> puiterwijk: used browser history w/o pressing enter
18:36:52 <puiterwijk> Please use https://bodhi.fedoraproject.org/ and https://bodhi.fedoraproject.org/docs
18:37:02 <ajayvembu> okay
18:37:06 * bowlofeggs throws pingou under the fedmsg bus
18:37:14 <pingou> thanks man
18:37:24 * smooge gives bowlofeggs of eggs the mop to clean that up
18:37:24 * pingou get messages over by the bodhi messages
18:37:26 <nirik> the wheels on the bus go...
18:37:37 <puiterwijk> pingou: if you hadn't, I'd be really concerned and call you right now, since that would mean you're under attack :)  (if you pasted it from a browser that followed all redirects)
18:37:39 <bowlofeggs> nirik: is it possible to see when people last logged in as a way to trim the membership?
18:38:01 <nirik> bowlofeggs: we could yeah... just a last on batcave01 or whatever
18:38:12 <smooge> a last on Jan 1st :)
18:38:19 <x3mboy> !
18:38:31 <x3mboy> Can I suggest something that may sound a little crazy?
18:38:32 <puiterwijk> x3mboy: you can just speak, no need to ask
18:38:37 <willo> from apprentice POV, the email seem fine, but it did feel a bit.  I'm putting the same info in these questions at lot.  I wonder how useful it is.
18:38:45 <x3mboy> Why not to interact with apprentice via telegram?
18:38:46 <bowlofeggs> nirik: that might be less overhead and be reasonable (and automatable!)
18:39:05 <smooge> because that would require us to have telegram accounts?
18:39:07 <nirik> bowlofeggs: good thought.
18:39:16 <nirik> x3mboy: I don't think any of us use telegram. ;)
18:39:21 <x3mboy> It's being a great way to work in several teams: Diversity, CommOps, Ambassadors Latam, and others
18:39:43 <x3mboy> nirik, ok, was just a suggestion ;)
18:39:55 <cverna> x3mboy: what is the plus compare to irc ?
18:39:56 <puiterwijk> x3mboy: one problem would be that you'd be entirely out of the loop, since basically all infra stuff happens and gets discussed on IRC
18:40:01 <pingou> bstinson: just got stg.pagure.io to kick a build in ci.centos.org :)
18:40:09 <smooge> x3mboy, this has come up with a lot of different social technologies in the last 10 years... usually we have 1 window to focus on and by the time we get onto whatever technology.. everyone is wanting us to be on something else.
18:40:15 <pingou> cverna: it's where the new kids are :)
18:40:21 <cverna> bstinson++ pingou++
18:40:21 <zodbot> cverna: Karma for bstinson changed to 1 (for the f27 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
18:40:22 <bstinson> pingou: thanks for the debugging session :)
18:40:23 <zodbot> cverna: Karma for pingou changed to 13 (for the f27 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
18:40:30 <willo> "automating the email" in a simple web app as mentioned might be good.
18:40:31 <puiterwijk> bstinson++
18:40:31 <zodbot> puiterwijk: Karma for bstinson changed to 2 (for the f27 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
18:40:50 <pingou> bstinson++
18:40:50 <zodbot> pingou: Karma for bstinson changed to 3 (for the f27 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
18:41:04 * pingou amazed he didn't do that sooner
18:41:05 <smooge> x3mboy, so it wasn't a crazy idea.. just one we have had problems getting onto fast enough to ever do.
18:41:30 <willo> it could then pull in activity to providing a summary of what's done, across pagure tickets as well as the login activity as mentioned and others
18:41:54 <nirik> well, I can try this month to just mail everyone and tell them to make sure and login to batcave01, if they don't they will be removed... we can give it a whirl. If they have feedback, start a thread on the list.
18:41:57 <x3mboy> Ok, I'm just telling because even communications to flock happens very fluid, but I understand the POV of don't having enough time to look through several techs
18:42:46 <pingou> bstinson: we weren't that far, just two typos :)
18:43:29 <smooge> x3mboy, understood.
18:43:40 <bstinson> now off to do the rest of the infra work
18:43:42 <x3mboy> puiterwijk, the IRC is not a problem, we use a bridge IRC<->Telegram
18:43:42 <puiterwijk> x3mboy: also, one problem I'd personally have with telegram is that telegram is on my phone, and doing infra stuff on a phone is... painful. And there's no telegram clients that I know of for Fedora in the main repos
18:43:43 <x3mboy> :D
18:43:59 <puiterwijk> x3mboy: yeah, but from the IRC side, that bridge is really painful to use
18:45:02 <x3mboy> Ok, I get it
18:45:22 <puiterwijk> Anyway, not a bad idea, just trying to explain why I'd personally think it's not the best for infra
18:45:25 <nirik> we can probibly move on now unless someone has something further on this?
18:45:33 <x3mboy> Nope
18:45:50 * nirik wanted to get everyone thinking about it... feel free to send ideas later too.
18:45:50 <willo> no, neither do I
18:46:08 <smooge> I think that cleaning out a lot of the sysadmin groups would be a good idea in the coming year.
18:46:24 <cverna> maybe we need to do some screencast :)
18:46:27 <smooge> so that all souns find.
18:46:36 <pingou> smooge: +1
18:46:48 <marc84> smooge +1
18:46:49 <smooge> I need to go take my Geritol(TM) and my prune juice though
18:46:52 * pingou can't imagine puiterwijk disagreeing :D
18:46:57 <pingou> smooge++
18:47:05 * puiterwijk totally agrees with smooge and has already said so before :)
18:47:06 <nirik> cverna: I thought of that... just never get to it.
18:47:15 <smooge> #topic Open Floor
18:47:19 <nirik> smooge++
18:47:20 <zodbot> nirik: Karma for smooge changed to 6 (for the f27 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
18:47:29 <smooge> ok we have reached the open floor part of this meeting
18:47:32 <puiterwijk> smooge++
18:47:35 <nirik> woah, I haven't given smooge a cookie yet? weird.
18:47:49 <bstinson> quick update on the Jenkins bits:
18:47:50 <nirik> smooge: you ok to keep running meetings? or should we rotate/trade off more?
18:47:51 <smooge> oh everyone agrees I need to take my geritol and prune juice? Damn I must be cranky
18:48:15 <smooge> Oh yeah.. first bstinson then schedules
18:48:17 <bowlofeggs> worf drinks prune juice too
18:48:37 <smooge> #topic bstinson talks about Jenkins
18:48:38 <bstinson> i kind of like prune juice...
18:48:43 <bstinson> but in any case
18:48:57 <bowlofeggs> bstinson++ for giving us awesome jenkins jenkinsness
18:48:57 <zodbot> bowlofeggs: Karma for bstinson changed to 4 (for the f27 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
18:49:06 <bstinson> - the static agents are done, i took the ansible playbooks in infrastructure.fp.o
18:49:22 <bstinson> - i'm fiddling a bit with the custom master image that we'll deploy in openshift
18:49:38 <bstinson> - Deadline: Tomorrow for a trial migration with pingou
18:49:57 <bstinson> then we can talk about what the retirement process looks like
18:50:18 <pingou> \ó/
18:50:51 <bstinson> Unresolved questions:
18:51:20 <smooge> - Prune juice vs raisin juice
18:51:30 <pingou> vs apple juice/
18:51:32 <pingou> ?
18:51:34 <bstinson> - The agent setup playbooks are on our end, we'll maintain those in sync with what's in infrastructure.fp.o
18:51:41 <pingou> or are you speaking about fermented raisin juice?
18:51:42 <bstinson> oh apple juice for sure
18:51:52 <bowlofeggs> yeah what is prune juice if a prune is just a dried plum?
18:52:00 <bowlofeggs> if you dry it, how can it have juice
18:52:08 <willo> pineapple juice has my vote
18:52:14 <smooge> ok sorry
18:52:16 <bowlofeggs> pineapple is so good
18:52:19 <smooge> I really derailed this
18:52:22 <bowlofeggs> hahaha
18:52:26 <pingou> ^^
18:52:34 <willo> smooge++
18:52:34 <zodbot> willo: Karma for smooge changed to 7 (for the f27 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
18:52:35 <bstinson> that cran-apple stuff they have on planes is vile though
18:52:35 <smooge> any other issues bstinson
18:52:37 <pingou> bstinson: keep in sync, so release by release?
18:52:52 <pingou> or is it updated more often?
18:52:55 <smooge> more like how does one run playbooks
18:52:57 <bowlofeggs> on planes i like to ask for a glass of landing gear grease
18:53:14 <bstinson> pingou: as-in any changes will be made in infra.fp.o, and we'll pull those into our internal repo to update the agents
18:53:41 <pingou> bstinson: ok, I'm wondering if we can find a way to make sure we don't forget
18:53:45 <nirik> there is a ticket to add some packages to some of the slaves...
18:53:54 <nirik> (a new ticket filed this morning)
18:53:55 <pingou> potentially to automate it even
18:53:58 <bstinson> nirik: can i take that assignment?
18:54:10 <nirik> sure... let me find it
18:54:16 <pingou> that will no longer be an issue with the dynamic agents no?
18:54:20 <bstinson> pingou: hopefully we don't need to maintain that sync very long
18:54:33 <bstinson> dynamic agents will solve this problem
18:54:38 <pingou> coolio
18:54:42 <nirik> https://pagure.io/fedora-infrastructure/issue/6630
18:55:11 * pingou will try to make some notes on how to do thing
18:55:18 <pingou> might be nice to blog about that
18:55:24 <bstinson> last open question:
18:55:36 <smooge> ok
18:56:02 <bstinson> - i'll push this s2i template for the jenkins master to a personal pagure repo, do we want it to live somewhere 'official'?
18:56:46 <nirik> as long as it's somewhere should be fine..
18:56:53 <pingou> pagure has no concept of personal project (yet) :)
18:56:58 <nirik> it would at least be in the openshift instance. ;)
18:58:01 <bstinson> ok so 'somewhere' is good enough. i'll be sure to share that with the infra group
18:58:08 <smooge> thanks
18:58:11 * pingou has one question for after smooge 's topic
18:58:20 <smooge> one sec.
18:58:30 <smooge> do we have a meeting here in 2 minutes or can we run over?
18:58:59 <smooge> bstinson, anything else?
18:59:35 <smooge> #topic Schedule for upcoming meetings
18:59:37 <bstinson> that's it for me
18:59:49 <smooge> OK I am on PTO next week
19:00:18 <smooge> the week after that the 25th people will be going to Brno for meetings
19:00:30 <smooge> I can run a meeting that week but expect not a lot of communication
19:00:47 <relrod> I can probably do next week's
19:00:47 <smooge> the 1st may be similar as people who are doing FOSDEM head there
19:00:52 <smooge> thanks relrod
19:01:06 <pingou> February
19:01:15 * pingou read May 1st
19:01:17 <pingou> ....
19:01:20 <smooge> hahahaha
19:01:27 <smooge> I am planning way ahead
19:02:09 <puiterwijk> May 25th? :P
19:02:12 <smooge> ok so February 1st I will run but expect not a lot so it will probably be a "these are the fires and these are all the things who got to go to nice places are now assigned."
19:02:14 <nirik> new years party 2018?
19:02:32 <smooge> and that is it for upcoming meetings
19:02:44 <smooge> #topic Pingou's question (the answer is not 42)
19:03:03 <pingou> any thoughts on upgrading nuancier in prod?
19:03:18 <pingou> the submission phase started 9 days ago, there are been 4 submissions since
19:03:24 <pingou> the outage should be under 5 minutes
19:03:27 <bowlofeggs> 42,
19:03:44 <pingou> and I kinda think having this in before the fedora magazine article comes out would be good
19:03:54 <puiterwijk> pingou: what's the risk? (i.e. what's the change of size)
19:04:27 <bowlofeggs> some people were upset about how the last election round went
19:05:00 <pingou> puiterwijk: py3 compatibility, some UI changes and most importantly: a new date allowing to give some time between the end of the submissions and the start of the voting
19:05:14 <nirik> pingou: FYI, ci-cc-rdu01 needs upgrading/reinstalling with new os.
19:05:37 <puiterwijk> pingou: okay. How big do you guess the chance is that it'll fail and burn, is what I am basically interested in :)
19:05:48 <pingou> to avoid one issue we had last time: someone submitted something just before the deadline, could not be reviewed in time and thus didn't get included in the votes
19:05:53 <pingou> and that made a drama
19:06:02 <pingou> puiterwijk: little I expect
19:06:09 <pingou> the new field is trivial
19:06:10 <puiterwijk> pingou: then I'd say schedule an outage
19:06:18 * nirik nods. agree
19:06:28 <pingou> the py2/py3 change is bigger but unit-tests seem all happy
19:06:32 <pingou> okido, on it :)
19:06:58 <pingou> eof for me
19:07:04 <smooge> ok thanks
19:07:10 <smooge> #topic Open Floor
19:07:24 <smooge> anything left or should I stick a fork in this meeting and call it done?
19:07:26 <bowlofeggs> ok now let's figure out the juice issue
19:07:41 <nirik> apple juice is the best juice.
19:07:50 <pingou> bstinson: was clearly for apple, so is nirik apparently
19:08:03 <bowlofeggs> pomegranite
19:08:10 <willo> pineapple for the win
19:08:15 <pingou> I kinda like fermented raisin juice
19:08:16 <puiterwijk> Did anyone consider orange juice?
19:08:23 <pingou> puiterwijk: way too mainstream :D
19:08:24 <bowlofeggs> it's a pain to eat the fruit, but if someone else extracts the seeds/juice for you it is amazing
19:08:33 <cverna> apple + carrots juice
19:08:34 <puiterwijk> pingou: oh, and apple juice isn't? :O
19:08:42 <willo> oh, bowlofeggs: new contender from left field
19:08:45 <bstinson> oooh pomegranate juice turns into grenadine...
19:08:50 <pingou> puiterwijk: you're comparing orange and apple there man :D
19:08:59 <smooge> #endmeeting