14:49:46 <dgilmore> #startmeeting RELENG (2016-08-22)
14:49:46 <zodbot> Meeting started Mon Aug 22 14:49:46 2016 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:49:46 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:49:46 <zodbot> The meeting name has been set to 'releng_(2016-08-22)'
14:49:46 <dgilmore> #meetingname releng
14:49:46 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion mboddu
14:49:46 <zodbot> The meeting name has been set to 'releng'
14:49:46 <zodbot> Current chairs: bochecha dgilmore masta maxamillion mboddu nirik pbrobinson pingou sharkcz tyll
14:49:49 <dgilmore> #topic init process
14:50:34 * sharkcz is here
14:50:44 * pbrobinson o/
14:51:11 <nirik> morning
14:51:40 <dgilmore> #topic #6467 Consider splitting the ancient portions of the archive
14:51:48 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6467
14:51:51 <mboddu> morning everyone
14:52:19 <dgilmore> sorry for the late start
14:52:23 <dgilmore> lets get moving
14:53:31 <pbrobinson> I think the above makes sense, can probably archive the secondary stuff into it as well
14:53:44 <dgilmore> I think that some of this may not really be our decision
14:53:44 <tibbs> Hey, folks.
14:53:56 <dgilmore> pbrobinson: well secondary is in archive
14:53:58 <tibbs> Sorry, I'm never in the second meeting channel.  Too many channels.
14:54:05 <dgilmore> at least some of the older releases
14:54:06 <tibbs> I have no scrollback.
14:54:06 <pbrobinson> ah!
14:54:15 <dgilmore> tibbs: nothing has been said yet
14:54:48 <nirik> how about just removing archive from enchlada and buffet... so mirrors that really want it need to add a new seperate module?
14:54:57 <dgilmore> I am against spliting old stuff out unless we just take it down
14:55:47 <dgilmore> I think that perhaps we should engage the council, and possibly legal over older releases to ensure we are in GPL compliance, though GPL wise the sources and binaries will remain in koji
14:55:59 <dgilmore> nirik: that I am okay with
14:56:18 <dgilmore> nirik: I guess tibb's concern was over popular EOL releases
14:56:22 <tibbs> I'm afraid I just don't see the harm in having it online but not in the mirror network.
14:56:32 <dgilmore> say a large number of people still using f22
14:56:55 <tibbs> I recall that smooge had an opinion but I don't see him here.
14:57:14 <tibbs> Since he's the one who maintains the archive stuff.
14:57:17 <dgilmore> tibbs: I think what nirik said and I agree with is just not make archive available in the enchilada module
14:57:34 <nirik> theres still a lot of f21/f22 hits
14:57:38 <dgilmore> tibbs: well I did the secondary archiving
14:57:44 <dgilmore> its not just smooge
14:58:10 <dgilmore> I have done at least one if not two fedora releases also
14:58:34 <tibbs> Anyway, my fundamental point is that archive grows without bound, while the utility of the individual releases within archive diminishes over time.
14:58:51 <dgilmore> tibbs: do not disagree
14:59:03 <dgilmore> tibbs: we expect most mirrors to not mirror archive
14:59:04 <nirik> but it also doesn't change often
14:59:13 <tibbs> It simply does not make sense to keep all of that stuff together _if_ you consider things from the standpoint of the mirrors who are volunteering to carry your (our) stuff.
14:59:30 <tibbs> The rate of change of things in archive isn't really the issue, though.
14:59:51 <nirik> so, perhaps we should be slower to archive...
15:00:16 <tibbs> But then you up the load on everyone who mirrors "fedora" (the module).
15:00:33 <nirik> ie, drop archive from most/all mirrors, but keep eol releases in releases until their usage goes down enough that it won't bother master archive mirrors.
15:00:50 <nirik> "load"?
15:01:33 <tibbs> Load in that context means both the size of things and the number of people connecting to your mirror.
15:01:53 <nirik> sure, but if we just moved all archive to not mirror, master mirrors would get a big hit.
15:01:58 <nirik> which doesn't help anyone
15:02:32 <nirik> https://admin.fedoraproject.org/mirrormanager/statistics/2016-08-22/repositories (~30% of hits were for f21/f22/eol stuff)
15:02:50 <nirik> but much less than that was f20 and older
15:03:04 <tibbs> Well, my point is that if you keep f21 and f22 in the fedora module longer, mirrors of the fedora module now carry 30-40% more content.
15:03:25 <dgilmore> tibbs: how do you figure that?
15:03:26 <nirik> I suppose
15:03:42 <tibbs> Purely by the count of releases that would be in the "fedora" module.
15:03:58 <dgilmore> there is 4 releases in /pub/fedora now
15:04:02 <dgilmore> it would add 2 more
15:04:13 <tibbs> So... 30%?
15:04:17 <dgilmore> 50%
15:05:01 <dgilmore> we would have half as much content again
15:05:12 <tibbs> Well, that only makes my point stronger.
15:05:15 <dgilmore> it is a consideration
15:05:36 <dgilmore> the churn of the content is not an issues as it does not churn at all
15:05:46 <nirik> the main problem with doing anything here is that mirrors are slow to change... so if we dropped archive out and added archive-recent, not many would carry it for a while... or you mean we drop archive out and put archive-recent in buffet?
15:06:07 <tibbs> Yes, churn is not a problem here.
15:06:15 <dgilmore> nirik: I think that he was advocating for it not being in buffet
15:06:32 <dgilmore> tibbs: I think this needs to go to the council or FESCo
15:06:49 <dgilmore> its really much more a political thing than technical
15:07:15 <dgilmore> I am not a fan of splitting the content up and causing churn as it moves
15:07:31 <tibbs> My original proposal was to drop the oldest releases out of archive and move them to "something else".  Whether that is a separate rsync module or some other servers which aren't on the mirror network, I left out of the proposal.
15:07:42 <dgilmore> especially if we do not have a automated way to move content
15:08:09 <dgilmore> tibbs: so it is not Release Engineering's place to decide
15:08:14 <tibbs> That would not cause any additional churn, at least.  Unless you consider deleting things to be churn.
15:08:22 <dgilmore> so lets take it to the council as It is much more political
15:08:30 <dgilmore> they can send it to FESCo if They choose
15:08:33 <tibbs> OK, well, I can handle that.
15:08:54 * nirik isn't sure it's political, but ok
15:08:58 <dgilmore> tibbs: well it is concern if we move it
15:09:13 <dgilmore> as the mirrors have to do "things" to keep it all in sync
15:09:34 <dgilmore> but i guess if they do not mirror the older stuff then it does not matter to them much
15:09:37 <tibbs> True, but no more things than they have to do now to keep archive and fedora in sync.
15:10:10 <dgilmore> nirik: it is political in that some people will get upset that we are doing bad things
15:10:35 <dgilmore> we had someone contact Red Hat legal and complain when we stopped making Source DVD isos
15:10:39 <tibbs> I don't know what those bad things would be.
15:10:50 <dgilmore> because they did not like the way we made the sources available
15:10:51 <tibbs> The stuff could still trivially be accessible through mirrormanager.
15:11:17 <dgilmore> tibbs: well withholding access to the content, not honouring access to sources, etc
15:11:29 <dgilmore> people views on that differ greatly
15:11:36 <tibbs> I never proposed making the content unavailable.
15:11:57 <tibbs> I don't see how anyone could possibly argue that.
15:11:57 <dgilmore> tibbs: but some people will see it as doing that
15:12:08 <tibbs> Again, I cannot imagine how.
15:12:10 <dgilmore> because they can not find it in the way they are used to
15:12:24 <dgilmore> because people
15:12:32 <dgilmore> which is why i say it is political
15:12:33 <tibbs> By that argument we would be unable to change anything ever.
15:12:42 <dgilmore> tibbs: pretty much :)
15:12:49 <dgilmore> everytime we change people complain
15:12:56 <tibbs> I just can't buy into that as a valid reason for doing anything at all.
15:13:09 <tibbs> Or rather, not doing something.
15:13:10 <dgilmore> I am honestly not a fan of the idea
15:13:49 <dgilmore> we have always left it up to mirrors to mirror the content they choose. yes it makes some thing hard on us
15:13:50 <tibbs> Do you have any sympathy for the individual hosts on the mirror network who donate their resources to us?
15:13:58 <dgilmore> it makes mirrormanagers job harder
15:14:23 <dgilmore> but modules and the road we are going down are going to make it much harder going forward
15:14:38 <nirik> tibbs: are those mirrors not able to just exclude archive* if they don't want it?
15:14:39 <tibbs> I think, within the limitations of available resources and manpwer within Fedora, we should take pains to make things as easy as possible for people to mirror our content.
15:14:56 <nirik> or just use enchilada instead of buffet?
15:15:15 <dgilmore> a lot of mirrors only mirror enchilada
15:15:28 <dgilmore> I have seen a lot that exclude updates and rawhide
15:15:28 <tibbs> nirik: Yes, of course, but then that hurts us (Fedora) when we have fewer mirrors of, say, F22 when it moves to archive.
15:16:03 <dgilmore> I think if large number of people continue to use EOL releases then we are failing
15:16:35 <tibbs> We seem to show very little consideration for the mirrors in general.  I'm trying to change that.
15:16:42 <nirik> I guess it would be interesting to see how many mirrors we have using enchilada vs buffet... it's not just archive thats different there... all of secondary, etc
15:16:48 <dgilmore> that 27% of hits https://admin.fedoraproject.org/mirrormanager/statistics/2016-08-22/repositories were for f21 and f22 updates says something
15:17:00 <nirik> tibbs: thats great, I am just not clear on how this is making things nicer for mirrors
15:17:17 <dgilmore> tibbs: I am all for making things better for mirrors
15:17:27 <dgilmore> but like nirik I do not see how this does that
15:17:27 <nirik> if you choose buffet, don't you want all of everything?
15:18:01 <dgilmore> tibbs: perhaps you think more people will opt in to mirroring recent-archive?
15:18:06 <nirik> perhaps we could gather stats on how many are using what so we can optimize better?
15:18:28 <tibbs> Sorry, someone at the door.
15:18:37 <dgilmore> tibbs: np
15:18:56 <dgilmore> nirik: knowing what the mirrors mirror would help
15:19:15 <dgilmore> perhaps engaging in discussion with mirrors over how to make things better for them would also
15:19:27 <dgilmore> we need to talk to them about ostree and modules anyway
15:19:28 <tibbs> Or, to put it another way, to limit the amount of content someone has to mirror if they want to help us and mirror just those things which would actually help us.
15:19:36 <tibbs> Mirroring fc1 doesn't help anyone.
15:19:59 <tibbs> Of course noting that those old releases were absolutely tiny when compared to today's stuff.
15:20:09 <tibbs> But I'm thinking about the future here.
15:20:25 <dgilmore> people using EOL releases does not help either. if people get hacked for security vunerabilities that will never get patched and they blame us for it
15:20:48 <tibbs> Of course.
15:21:15 <nirik> by hits/traffic, the best someone can do is epel. Next would be enchilada (fedora). Then archive, secondary?
15:21:17 <tibbs> A general council question about what we do with "old" releases (and how we define "old") does make sense here regardless.
15:21:56 <tibbs> nirik: I guess that would be a really good question to ask.  "You're willing to mirror?  Here's how you can help the most."
15:22:03 <nirik> if tons of places are just doing buffet though, not sure changing anything matters much
15:22:24 <tibbs> Somehow I doubt too many places are actually mirroring buffet.
15:22:24 <dgilmore> nirik: right
15:22:34 <nirik> we may be able to gather that from rsync logs
15:22:34 <tibbs> But the rsync logs would show that.
15:22:40 <dgilmore> stats on what modules are pulled on rsync would be good
15:23:06 <tibbs> Anyway, I am working from the other end here.  Sadly now the semester has started and I'm buried, but I'll dig out soon.
15:23:38 <nirik> so, lets gather those stats and revisit next week? and/or add a council ticket and close this one?
15:23:42 <dgilmore> though I know when i only rsync parts of /pub i use buffet and use excludes and paths to get the content I need
15:24:13 <dgilmore> perhaps we get more data and revist then send to the council with more than hey what should we do
15:25:01 <nirik> you know I have seen a bunch of people in #fedora surprised that we have any eol releases at all still online. ;)
15:26:30 <dgilmore> nirik: maybe we take them offline
15:26:36 <tibbs> I didn't mean to stir up crap, but I did think that it was a valid question.
15:26:45 <nirik> some people expect us to. ;)
15:26:46 <dgilmore> it may get more people to move to newer releases
15:27:14 <dgilmore> peoples expectations over what we produce and where it goes vary greatly
15:27:29 <dgilmore> I am sure some people are made we move things to archive at all
15:27:39 <dgilmore> are mad even
15:28:05 <nirik> yeah
15:28:10 <nirik> you can't please everyone.
15:28:26 <tibbs> It's a balance.  There's only so much bandwidth, and only so much we can ask of the mirror network.
15:28:39 <dgilmore> fedora-alt     	non-Fedora Alternative Content
15:28:39 <dgilmore> fedora-archive 	Fedora Release Archives
15:28:39 <dgilmore> fedora-enchilada	Fedora - The whole enchilada
15:28:39 <dgilmore> fedora-buffet  	Fedora - The whole buffet. All you can eat.
15:28:39 <dgilmore> fedora-epel    	Extra Packages for Enterprise Linux
15:28:41 <dgilmore> fedora-linux-releases	Fedora Linux Releases
15:28:44 <dgilmore> fedora-linux-development	Fedora Linux Development
15:28:46 <dgilmore> fedora-linux-updates	Fedora Linux Updates
15:28:49 <dgilmore> fedora-secondary	Fedora Secondary Archs
15:28:51 <dgilmore> fedora-stage   	Staging directory
15:28:54 <dgilmore> deltaisos      	Delta isos
15:28:56 <dgilmore> fedora-live-respins	Fedora Live Respins
15:28:59 <dgilmore> so they are the modules we currently have
15:29:19 <dgilmore> deltaisos should go away
15:29:33 <dgilmore> QA have not made them in a few years I think
15:29:38 * masta is here finally
15:29:44 <nirik> we need to decide where secondary lives in the new one koji world too... (I know, they could live anywhere, but should they just be alongside the other arches?)
15:30:03 <nirik> delltaisos only make sense for dvds... and we only have the server dvd left.
15:30:22 <dgilmore> nirik: well the definition of alternative architectures is where the content lives
15:30:38 <dgilmore> fedora-alt kinda makes sense, but it is already uses
15:30:45 <nirik> yeah, but we may want to change the name 's/secondary/alternative' or something, and rearrange it
15:30:46 <dgilmore> maybe fedora-alt-arch
15:30:49 <pbrobinson> I need to head out for an appointment so where others can't answer for my various status updates you can just substitute this https://images.duckduckgo.com/iu/?u=http%3A%2F%2Fwww-static.weddingbee.com%2Fpics%2F342067%2Fthis-is-fine-meme.jpg&f=1
15:31:03 <nirik> pbrobinson: :)
15:31:05 <dgilmore> pbrobinson: :)
15:31:22 <masta> lol
15:31:50 <dgilmore> maybe we should get rid of fedora-linux-releases fedora-linux-development fedora-linux-updates
15:32:18 <dgilmore> I guess knowing how much they are used can help us figure out if it makes sens to add or remove modules
15:32:34 <dgilmore> we will have to look at things with modularity anyway
15:32:43 <dgilmore> and we need to talk to mirrors about ostree
15:32:57 <masta> how are the rsync logs collected?
15:33:01 <dgilmore> #info we need to look at how mirrors are mirroring fedora
15:33:30 <masta> I'd imagine the enchilada would be enough, so yeah... maybe remove fedora-linux-releases fedora-linux-development fedora-linux-updates
15:33:55 <dgilmore> masta: a lot of mirrors may be using them to pick and choose
15:34:02 <dgilmore> we need more data
15:34:04 <nirik> masta: logs to a local file on each download server.
15:34:05 <bowlofeggs> .hello bowlofeggs
15:34:06 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <randy@electronsweatshop.com>
15:34:41 <dgilmore> and we should use the data of how mirrors pull from us to help us engage the council for a policy
15:34:47 <nirik> sure.
15:34:51 <masta> yep
15:35:00 <nirik> someone can file a infra ticket on gathering the rsync info...
15:35:09 <dgilmore> we also need to keep in mine the changing landscape
15:35:19 <bowlofeggs> this all sounds relevant to a question about how we will split docker images into mirror modules
15:35:24 <dgilmore> and what difficulties that may bring
15:35:34 <dgilmore> bowlofeggs: kinda
15:36:01 <masta> does anybody have a favorite rsyncd log analyzer ?
15:36:03 <dgilmore> #info we want to make life easy for mirrors.
15:37:18 <dgilmore> does anyone want to take an action item?
15:37:38 <smooge> hello is this the ticket on my part?
15:37:48 <dgilmore> smooge: perhaps
15:38:25 <smooge> I can add an rsync target which just shows the last N releases.
15:38:38 <dgilmore> smooge: there is a lot of things we can do
15:38:46 <nirik> smooge: we want to gather info on what modules mirrors are using... and how much
15:38:59 <smooge> hahahaha ok.
15:39:02 <nirik> ie, is it mostly buffet? enchilada? something else
15:39:26 <smooge> how many months back do you want?
15:39:27 <masta> I'm keen to review the available logs, and help establish some facts. I guess that is an action item
15:39:52 <dgilmore> #action masta to look at rsync logs and help figure out the modules most often used
15:40:04 <dgilmore> smooge: maybe 1 or 2 months
15:41:25 <smooge> ok good because I can only go back to May it turns out
15:41:43 <masta> smooge, want an infra ticket for gathering those logs?
15:41:49 <bowlofeggs> if it makes sense and there's enough time during this meeting, i wouldn't mind having a chat about how docker stuff should be split into mirror modules. it's one of the open questions on the infra thread i started last week
15:42:06 <bowlofeggs> we could do it another time if that's too off topic here
15:43:19 * nirik notes we are at 1hour 15min of our 1 hour meeting, so perhaps out of band would be better... especially since we only discussed one topic so far
15:43:54 <dgilmore> yeah
15:44:06 <dgilmore> lets wrap up the meeting
15:44:29 <dgilmore> #topic open floor
15:44:43 <dgilmore> does anyone have anything else to bring up?
15:44:44 <nirik> I had one quick item...
15:44:56 <nirik> but we could also do it next week I guess... but will toss it out.
15:45:32 <nirik> I made a bodhi-backend03 thats fedora-24... it works to do pushes (aside from the f24 sigul bug). Once that sigul bug is fixed, should we move to using it instead of 01?
15:45:51 <dgilmore> nirik: probably
15:46:04 <dgilmore> at the least we would find new bugs with weak deps
15:46:05 <nirik> it will give us weak deps in repodata... which would be nice
15:46:32 <masta> sounds good
15:46:51 <dgilmore> #info will move to bodhi-backend03 once sigul is working in f24
15:47:09 <masta> nirik, so long as we keep 01 online a while for fail-back
15:47:21 <masta> seems like very little down-side risk
15:47:25 <smooge> ok I can make a rough guess for stuff for you
15:47:30 <nirik> masta: sure.
15:49:15 <smooge> masta, I can get you the logs in a bit. If you want 'rough' not too detailed guesses on avg number of hits per day per target
15:49:18 <smooge> https://paste.fedoraproject.org/412524/88091914/
15:50:34 <masta> smooge, thank you. Look forward to parsing the data. :-)
15:50:51 <smooge> that is only for one server.. so multiply by 5
15:50:58 <dgilmore> anything else?
15:51:05 <dgilmore> if not will wrap up
15:52:04 <bowlofeggs> nirik: glad that the bodhi build is working for you!
15:52:14 <bowlofeggs> nirik: that's also valuable testing for me, so thanks!
15:53:23 <dgilmore> #endmeeting