20:00:03 <nirik> #startmeeting Infrastructure outage retrospective (2011-03-22)
20:00:03 <zodbot> Meeting started Tue Mar 22 20:00:03 2011 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00:03 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:00:04 <nirik> #meetingname infrastructure-retrospective
20:00:04 <zodbot> The meeting name has been set to 'infrastructure-retrospective'
20:00:27 <nirik> Greeting everyone. We are going to be having a retrospective/lessons learned/brainstorming session here.
20:00:40 <jsmith> w00t!
20:00:57 <nirik> There was a hardware failure friday on our signing server, and we want to figure out better ways to mitigate any risks from such.
20:01:03 <nirik> who all is around? :)
20:01:09 * dgilmore is
20:01:19 <abadger1999> Hola
20:01:31 * skvidal is here
20:01:41 <nirik> Oxf13: you around?
20:01:53 * Oxf13 
20:02:04 * jsmith is here
20:02:16 <Oxf13> funny enough, I wanted to move the meeting to avoid lunch, and I haven't grabbed lunch yet :)
20:02:24 <nirik> smooge: you around?
20:02:51 <jsmith> Oxf13: That's OK... it's 4:00pm here and I still haven't had (a proper) lunch
20:03:24 * nirik had some crackers and cheesh. Lunch of champions. ;)
20:03:31 <nirik> anyhow, I guess lets get started?
20:03:42 <nirik> #topic Timeline/Recap
20:03:53 <smooge> here
20:03:59 <nirik> hey smooge.
20:04:24 <nirik> So, we have a small number of sensitive servers that don't follow our normal updates policy.
20:04:52 <nirik> last friday we determined it might be a good idea to update them and reboot them into new kernels, etc.
20:05:16 <smooge> yes.
20:05:17 <nirik> One of those is the sign-vault01 server. We applied updates to it and rebooted.
20:05:28 <nirik> it had a hardware failure and didn't come back up.
20:05:36 <dgilmore> eventually the plan for the sign-vault is to have it off of the network entirely
20:05:50 <nirik> we then took 2 approaches:
20:06:00 <nirik> a) move drives to spare hardware and bring it back up.
20:06:07 <nirik> b) get a new instance setup as a virtual.
20:06:17 <nirik> a managed to complete first and we were back up.
20:06:32 <nirik> (I would like to thank everyone who worked on getting it back up and working)
20:06:53 <nirik> So, I think thats the recap/timeline... anyone have anything I missed in there?
20:07:01 * dgilmore thinks b is bad
20:07:21 <nirik> it could be. ;)
20:07:33 <Oxf13> I have something
20:07:33 <nirik> #topic setup of sensitive boxes
20:07:36 <smooge> well at the time there wasn't any spare hardware
20:07:37 <nirik> #undo
20:07:37 <zodbot> Removing item from minutes: <MeetBot.items.Topic object at 0x34e5ddd0>
20:07:49 <dgilmore> Oxf13: shoot
20:07:50 <Oxf13> We did actually get B mostly up.  A virt host was created, and I got a copy of the sensitive data onto it
20:07:59 <Oxf13> the data isn't in a production directory though.
20:07:59 <smooge> I would like to thank RHIT for finding the spare hardware that they got out of somewhere for us
20:08:18 <nirik> smooge: good point. RHIT found us spare HW for that.
20:08:29 <nirik> Oxf13: ok.
20:08:51 <smooge> they also dropped another problem and moved that hardware for us.
20:09:15 <nirik> Many Kudos for their assistance.
20:09:39 <nirik> anyone have anything more on background?
20:09:43 <smooge> yes
20:09:52 <nirik> smooge: go ahead
20:10:02 <smooge> or is it a lesson learned as the data was not backed up anywhere
20:10:26 <nirik> yeah... backups is on the agenda. ;)
20:10:35 <Oxf13> it is good background data
20:10:43 <Oxf13> at least we never got a yes/no on whether or not the backup ever completed
20:10:50 <smooge> there is no backup on it
20:11:01 <nirik> :( ok, good to know...
20:11:19 <smooge> it was not put into backups because people were worried about mixing of sensitive data with regular backups
20:11:41 <nirik> ok, shall we move on and discuss backups?
20:11:46 <skvidal> yah
20:11:53 <nirik> #topic Backups of sensitive boxes
20:11:54 <skvidal> and the previous topic tooo
20:12:03 <skvidal> 'setup'
20:12:34 <abadger1999> <nod>
20:12:47 <nirik> why don't we save setup for later and hash out some of the things that might be easier/simpiler? or do we want to look at setup first to know the answer to others?
20:13:10 <Oxf13> maybe I could talk about the original plan for these boxes?
20:13:19 <nirik> ok.
20:13:20 <nirik> #undo
20:13:20 <zodbot> Removing item from minutes: <MeetBot.items.Topic object at 0x28a7d8d0>
20:13:24 <skvidal> I think we need to look at the original assumptions
20:13:27 <nirik> #topic setup of sensitive boxes
20:13:40 <skvidal> which are that these box are being treated in a way which ensures no one pays attention to them
20:13:40 <nirik> I'd like to note: https://fedoraproject.org/wiki/User:Mitr
20:13:51 <Oxf13> My original plan was that bridge would be connected to the network, and allow ssh and sigul connections
20:13:52 <skvidal> I'd like to note that is not in an obvious place :)
20:13:53 <nirik> has background info on the signing servers.
20:14:00 <Oxf13> I wanted to limit the attack vectors though
20:14:05 <dgilmore> originally the box was inteneded to be connected the the bridge via a crossover cable
20:14:05 <nirik> skvidal: agreed.
20:14:08 <Oxf13> so I wanted puppet off, and backups off.
20:14:12 <dgilmore> and have no network connection at all
20:14:23 <Oxf13> the vault, which has the sensitive data, was to be only crossover connected to bridge
20:14:30 <skvidal> which never happened, right?
20:14:37 <Oxf13> any admin work was going to require a serial connection
20:14:51 <nirik> "Non-network connection (USB/serial) does not provide enough infrastructure (packeting, checksumming, retransmissions, debugging tools) that I'd rather not reimplement now; this can be replaced later if necessary."
20:14:56 <Oxf13> and maybe even a on-site call to hook up the serial as to not have it sitting there waiting for somebody to bang on the serial port
20:15:20 <Oxf13> skvidal: that never happened.  The server remained connected to the network and allowed ssh in
20:15:35 <Oxf13> We had some stability issues that required frequent restarts of the vault and bridge processes
20:15:52 <Oxf13> and we were not comfortable severely limiting our access to the machine.
20:16:07 * nirik nods.
20:16:14 <Oxf13> also, these were hand installed and setup systems that got turned into production systems
20:16:17 <Oxf13> which was my fault.
20:16:50 <Oxf13> I believe we were under time pressure to use it for whatever Fedora release needed to be signed
20:17:07 <Oxf13> and did not take the time to use puppet to rebuild the boxes in an automated way
20:17:12 <smooge> 13 beta I think?
20:17:30 <Oxf13> As for backups, the plan was mostly hand-wavy
20:17:56 <Oxf13> "we" felt that too many people had access to the backup storage and could grab the nss dbs and brute force them at their leisure
20:18:04 <Oxf13> so we did not hook it into the backup system
20:18:13 <skvidal> who is 'we'?
20:18:15 <Oxf13> but we also failed to create and follow an alternative backup plan
20:18:34 <Oxf13> skvidal: Mostly me, and I believe mmcgrath and notting were in the conversation
20:18:44 <Oxf13> our goal was to limit the number of ways the nss dbs could be accessed
20:18:55 <Oxf13> number of ways and number of people.
20:19:25 <nirik> the nss dbs are still passphrase protected (but of course could be pounded on with brute force given access to them)
20:20:11 <nirik> so, at the very least we need some backup plan.
20:21:00 <nirik> do we wish to persue the orig plan for serial access only, etc? or something else?
20:21:02 <Oxf13> nirik: yeah, and with Amazon cloud, the amount of time / $$ it takes to brute force got significantly smaller IIRC
20:21:02 <dgilmore> the backups should only need updating when we add a new key
20:21:30 <Oxf13> dgilmore: I'd like to verify with mitr where data about what user has access to which key and the passphrases for the users are stored.
20:21:49 <Oxf13> dgilmore: while we can re-add users and such, it'd be a hassle, so that data should be backed up on change too
20:21:50 * dgilmore would nearly be ok with someone putting an encrypted usb key in and backing up to that
20:22:01 <nirik> the backup01 box is pretty limited / also a sensitive box. Perhaps we could store a backup there?
20:22:12 <dgilmore> Oxf13: true but that doesnt change often
20:22:16 <skvidal> just to be clear
20:22:26 <skvidal> the only confidential data are the keys, correct?
20:22:43 <Oxf13> skvidal: user passphrases should be treated as confidential too IMHO
20:23:01 <Oxf13> each user has their own unique passphrase to access the system, along with an ssl cert
20:23:03 <dgilmore> skvidal: there is the keys and users passwd to access the keys
20:23:09 <Oxf13> (well the ssl cert being the FAS cert)
20:23:12 <skvidal> but we don't need to back up those
20:23:16 <skvidal> they don't matter
20:23:20 <skvidal> they keys for signing matter
20:23:23 <Oxf13> skvidal: no, they can be re-created.
20:23:29 <skvidal> the ssl certs and passphrases can be nuked from orbit
20:23:29 <skvidal> right
20:23:31 <skvidal> so for BACKUPS
20:23:32 <Oxf13> the keys are the critical thing we can't lose.
20:23:36 <skvidal> rightr
20:23:36 <skvidal> so
20:23:44 <skvidal> why don't we just back those up to some place
20:23:47 <skvidal> double-encrtypt
20:23:49 <skvidal> treble, if you'd like
20:23:53 <skvidal> it doesn't really matter
20:24:03 <skvidal> we don't need to automatically restore this info
20:24:17 <jsmith> Encrypt on a USB key, tape the USB key to the back of the server
20:24:19 <skvidal> but EVERYTHING else on the box needs to be automatically re-provisional
20:24:22 <skvidal> jsmith: nah
20:24:28 <skvidal> jsmith: requires someone on sight
20:24:31 <skvidal> onsite, even
20:24:37 <skvidal> my point is just this
20:24:46 <skvidal> the only piece that we need to backup is trivially small
20:24:50 <skvidal> and hell if we PRINT THEM OUT
20:24:53 <skvidal> we can get away with it
20:25:04 <nirik> yes, it's a small amount of data.
20:25:05 <Oxf13> right, it is trivially small
20:25:10 <smooge> personally a gpg encrypted file of the backup files should probably be enough.. make the passphrase 64+ characters and it will still take an IPv6 full amount of computers to crack it.
20:25:15 <Oxf13> the question is how paranoid do we want to be.
20:25:31 <skvidal> well let's put it this way
20:25:32 <Oxf13> mitr may say that the encryption on the nss db is enough
20:25:35 <skvidal> none of the attacks we've faced
20:25:44 <Oxf13> and that we should be "safe" having it out there
20:25:44 <skvidal> have been as a result of someone brute-forcing anything
20:26:02 <skvidal> I've got no issue with keeping these outside of the normal backup routines
20:26:07 <skvidal> but they need to be backed up SOMEWHERE
20:26:10 <skvidal> and that location needs to be:
20:26:12 <skvidal> raptor proof
20:26:14 <skvidal> documented
20:26:16 <skvidal> duplicated
20:26:25 <nirik> right.
20:26:25 <skvidal> and afaict it is NONE of those right now
20:26:27 <skvidal> right?
20:26:34 <nirik> it's not in existance right now. ;)
20:26:39 <skvidal> exactly
20:26:49 <Oxf13> what about this
20:27:09 <Oxf13> what about a cron job on the vault that will check to see if the dbs have changed, and if so, scp them off to some other host
20:27:15 <skvidal> no
20:27:17 <Oxf13> that other host /is/ part of the backup process
20:27:22 <skvidal> the keys change 2 times a year, right?
20:27:39 <Oxf13> skvidal: and at odd times for EPEL
20:27:41 <dgilmore> skvidal: roughly
20:27:45 <nirik> we should be able to add a 'backup the nss dbs' to 'add a new key SOP'
20:27:49 <skvidal> nirik: +1
20:27:51 <skvidal> yes
20:27:53 <skvidal> and moreovr
20:27:54 <dgilmore> epel gets a new key with new rhel
20:27:55 <Oxf13> nirik: ok, fine by me.
20:28:01 <skvidal> if we do not have a releng person who can do this
20:28:03 <skvidal> then we're SCREWED
20:28:33 <nirik> so, proposal: gpg encrypt the needed files for another layer, and back them up on backup01 and/or another non phx2 host?
20:29:00 <skvidal> nirik: back them up to some place inside RHT if we want several more layers of obfuscation
20:29:08 <dgilmore> i would say on a non phx2 box
20:29:15 <dgilmore> maybe our dr box
20:29:17 <Oxf13> I vote non phx2
20:29:42 * nirik is fine with that... passphrase in the private puppet repo?
20:29:52 <Oxf13> I kinda like the idea of somewhere within RHT, but maybe that's too complicated or too politically sensitive.
20:29:52 <skvidal> nirik: no
20:30:12 <smooge> dgilmore, ok 1 we currently do not have a dr box inside a place I would "trust" with them
20:30:12 <Oxf13> I'd like to get mitrs opinion on whether or not we need a second layer of encryption or not
20:30:20 <nirik> ok. but it does us no good if a raptor proof amount of people don't have it. ;)
20:30:27 <skvidal> nirik: sure you can
20:30:42 <skvidal> nirik: give the passphrase to mark cox
20:30:46 <nirik> Oxf13: ok.
20:30:46 <skvidal> or red hat infosec
20:31:07 <skvidal> b/c if rh decides to screw fedora in some way  the gpg keys will be the least of our problems :)
20:31:38 <nirik> sure...
20:31:54 <nirik> that problem goes away if we don't need to do another layer on them.
20:32:37 <nirik> so: Check with mitr on what level of paranoid we need for the nssdb. Either encrypt again or not, and backup to non phx2 host.
20:32:55 <nirik> anything else on backups? or is that good for those?
20:33:17 <nirik> the bridge also has a nssdb, should that get the same treatment?
20:33:22 <Oxf13> we need to also have a periodic test if we can deploy a new host via puppet and restore from backup
20:33:39 <Oxf13> nirik: I'm not sure what all is in the bridge db, and how sensitive it is.
20:33:56 <Oxf13> it may just have the mappings of users to rights and user passphrases.
20:34:00 * nirik gets ready to make with action items if no one objects.
20:34:01 <Oxf13> I don't think it needs to be backed up
20:34:11 <Oxf13> so a mitr question
20:34:19 <nirik> ok
20:34:26 <smooge> There is a problem with the setup assumptions.
20:34:47 <nirik> #agreed backup nssdb to a non phx2 host. (may need another layer of encryption or not)
20:34:51 <smooge> s/is a/are a couple/
20:35:02 <nirik> #agreed will check with mitr on sensativity and what needs to be backed up.
20:35:06 <Oxf13> yeah, somewhat difficult to get the nss dbs off the box if it's off the network.
20:35:13 <Oxf13> well, actually
20:35:24 <nirik> it can't be totally off the net.
20:35:28 <nirik> but it could not allow anything in.
20:35:30 <Oxf13> no, still difficult, but not impossible.
20:35:50 <Oxf13> nirik: right, it at least has to have a network connection to bridge
20:35:51 <nirik> smooge: go ahead.
20:36:25 <smooge> 1) The sign-bridge is a virtual system. We can't do a crossover cable to it. So we need to get another hardware for it if we want that.
20:36:57 <nirik> so, we could say: new setup is no incoming connections allowed to the box. Access is via serial for adding new keys or restarting things or applying updates or doing a backup of nssdb.
20:37:36 <Oxf13> nirik: new key addition is done through the sigul client, no need to log into vault for it
20:37:37 <smooge> 2) New hardware is needed anyway as the box's current warranty is almost over.
20:37:56 <smooge> 3) The box also did not have a good warranty on it. It is next business day RMA only.
20:37:59 <nirik> smooge: well, the serial/crossover is not implemented, so I don't think we need to worry about it now.
20:38:20 <smooge> nirik, however the assumption has been that they could go to it anytime when it was implemented.
20:38:36 <nirik> yeah, not sure if it's on the roadmap or off.
20:38:37 <smooge> they can't.. without more resources.
20:39:19 <dgilmore> nirik: id still like to do it when we are confident that the vault will just run
20:39:21 <nirik> thats another mitr question I guess, but I don't think we can plan for it now without a roadmap.
20:39:39 <nirik> smooge: is there replacement hardware available for the box?
20:40:06 <smooge> I budgeted for one next quarter. But I didn't for second hardware :/
20:40:30 <smooge> so currently if the box has issues it can be 24-72 hours before it is fixed.
20:40:46 <smooge> Since it goes off contract in May I am not sure its worth fixing it.
20:41:28 <smooge> eg by the time I get it through the system...
20:41:34 <nirik> yeah.
20:42:47 <nirik> ok, so should we leave it as setup currently and revisit when new hardware arrives?
20:43:07 <nirik> or should we move to the "no incoming connections allowed to the box. Access is via serial for adding new keys or restarting things or applying updates or doing a backup of nssdb." plan
20:43:18 <smooge> well I think we don't have much choice beyond setting up some sort of gpg encrypt backup of the databases.
20:43:21 <nirik> or does anyone have another plan to toss out there. ;)
20:43:24 <skvidal> I think I am in favor of leaving it as is
20:43:27 <Oxf13> I'd vote for leaving it as is, except for adding the backup SOP
20:43:30 <skvidal> and NOT making those changes when hw arives
20:43:37 <skvidal> meaning - this is how far it goes
20:43:40 <skvidal> and no further
20:43:50 <skvidal> and we all just take the pills which keep us from being this paranoid
20:44:09 <nirik> ok, that goes to the next topic...
20:44:16 <nirik> #topic backups for sensitive boxes
20:44:16 <Oxf13> heh
20:44:21 <nirik> #undo
20:44:21 <zodbot> Removing item from minutes: <MeetBot.items.Topic object at 0x2ac21e10>
20:44:26 <nirik> #topic updates for sensitive boxes
20:44:29 <nirik> updates. ;)
20:44:41 <Oxf13> skvidal: well, I plan on washing my hands of the whole thing in about 9 months so....
20:44:42 <nirik> I think it's a bad idea to have these out of our regular backup cycle.
20:44:56 * nirik sighs
20:44:59 <skvidal> Oxf13: it's like the hotel california
20:45:01 <nirik> s /backup/updates/
20:45:03 <skvidal> Oxf13: :)
20:46:06 <nirik> so, proposal: we apply updates to sensitive boxes at the same time as others (taking into account freezes, etc).
20:46:13 <smooge> nirik "have these out of our".. what is these?
20:46:21 <skvidal> nirik: which means we enable funcd on those boxes?
20:46:52 <nirik> sign-bridge01, sign-vault01, backup1 (I think thats all... are there others that don't run func or the like)
20:46:58 <Oxf13> ugh, paranoia setting in again.
20:47:24 <smooge> backup02 sort of fits into that.
20:47:48 <smooge> though func runs on it.. and puppet so never mind
20:48:12 <nirik> I don't think we should do func unless we also are doing puppet, etc... which I don't think we want.
20:48:22 <skvidal> don't we?
20:48:28 * dgilmore thinks we should only ever install security updates
20:48:57 <nirik> well, I guess the idea is that these are 'off the grid' so compromise in puppet/fas wouldn't also get them.
20:49:20 <Oxf13> right, that was the paranoia
20:49:42 <nirik> dgilmore: we could remove a ton of packages on them too I think... they could be much more minimal. (something to do with new one)
20:49:46 <Oxf13> no attack vectors from automation such as puppet or func (or backups)
20:50:00 <dgilmore> nirik: right
20:50:10 <dgilmore> we should only have installed what we need
20:50:46 <nirik> so, I'd say no func, no fasClient (local users?), no puppet. However, we should apply updates as part of our other updates flow... not put it off.
20:51:09 <dgilmore> nirik: backup01 has local users
20:51:30 <nirik> I think we should have sign* do that too... it's using fas currently I am pretty sure.
20:51:42 <dgilmore> nirik: right
20:51:52 <Oxf13> what is sign* ?
20:51:52 <nirik> does anyone object too stongly to that level of paranoia?
20:52:02 <smooge> well if we are going that route.. we might as well make this a home brewed ARM PCI card. I am not sure where the last stop of the reasonable train is.
20:52:03 <dgilmore> i agree with no fas, no func, no puppet
20:52:06 <nirik> shorthand for sign-bridge01 and sign-vault01
20:52:08 <skvidal> nirik: I think it's a waste of time
20:52:17 <Oxf13> oh ok.
20:52:23 <skvidal> nirik: I think we're chasing down paths that will ultimately get us right back to where we were on friday
20:53:08 <nirik> counterproposal? :) func and puppet and fas like any other machines?
20:53:10 <smooge> I do. I think that at a certain point we might as well go back to hand signing it from a cdrom in Gafton's cubicle because we are now assuming a lot of technical details that can be completely circumvented by physical.
20:54:41 <Oxf13> whatever we do, it has to be better than having the keys just sitting in somebody's homedir on say releng2....
20:54:48 <dgilmore> i think we need to make sure we have the backups so should anything happen, we can rebuild
20:55:00 <Oxf13> "The Incident" is what started some of this paranoia.
20:55:08 <dgilmore> Oxf13: right now the epel key is sitting on releng01
20:55:13 <dgilmore> relepel01
20:55:21 <dgilmore> the one for el4 and 5
20:55:22 <smooge> 1) I think the current level with a set of backup hardware and a set of backups that are encrypted is what we need to deal with.
20:55:29 <dgilmore> i really should import that into sigul
20:56:05 <smooge> after that we are assuming a lot of physical controls that don't exist.
20:56:09 <skvidal> the incident has nothing to do wit hthis
20:56:20 <Oxf13> skvidal: I beg to differ.
20:56:25 <nirik> well, curently we are not doing func or puppet on them, and fasClient only when someone fixes it to run. ;)
20:56:40 * nirik sees we are getting near an hour now.
20:56:45 <Oxf13> here is my input.
20:57:18 <Oxf13> If we're going to turn on puppet/func/fasclient, then I really don't see the point in being extra paranoid about our backups, and we just turn on the backup stuff too and have those dbs sit with the rest of the backup data.
20:57:29 <Oxf13> (provided they aren't unlocked on the filesystem while the daemon is running)
20:58:25 <nirik> so, perhaps input from mitr would be helpfull for us to decide A or B? (since they are kinda wildly different sides. ;)
20:58:54 <smooge> yes.
20:59:16 <nirik> ok, lets gather that and revisit in the regular infra/rel-eng channels for further decision?
20:59:34 <Oxf13> data points for right now
20:59:49 <Oxf13> sign-vault02 exists and has a backup of the vault data as of Friday
21:00:07 <Oxf13> that should remain in place until we have an agreed upon backup solution going
21:00:13 * nirik nods.
21:00:28 <abadger1999> is sign-vault02 fs encrypted?
21:00:47 <Oxf13> I don't believe so
21:00:53 <nirik> also, another datapoint: replacement hardware is available and sign-vault01 drives are going to be moved back to it later today.
21:00:56 <nirik> abadger1999: nope.
21:01:00 <Oxf13> I discussed fs encryption with mitr, and he had the opinion that it was a waste
21:01:01 <abadger1999> k
21:01:07 <Oxf13> given that the dbs were already encrypted
21:01:21 <seanjon> smooge: ping
21:01:25 <abadger1999> A shutdown fs encrypted host would make a decently secure warm backup.
21:01:28 <smooge> hi seanjon
21:01:30 <skvidal> abadger1999: +1
21:01:34 <seanjon> smooge: lets push the hardware swap 5pm
21:01:41 <smooge> 5pm your time?
21:01:43 <seanjon> yea
21:01:47 <skvidal> abadger1999: it would mean getting through the fs encryption and then through the nss db encryption
21:01:49 <nirik> abadger1999: yeah, could be.
21:01:54 <abadger1999> I think skvidal suggested something similar in email.
21:01:57 <nirik> yep.
21:02:13 <smooge> seanjon, I think that will be 00:00 UTC
21:02:25 <Oxf13> abadger1999: it'd make the backup process a bit more complicated, and require yet another passphrase to be shared or stored somewhere.
21:02:31 <smooge> you are UTC-7 I believe
21:02:41 <nirik> ok, I had 2 more small items I wanted to note, shall we do them real quick, then call it a meeting and try and ponder longer term decisions?
21:02:48 <skvidal> Oxf13: backup process MORE complicated?
21:02:49 <smooge> seanjon, ok will move it to 00:00 UTC
21:02:51 <skvidal> you just snapshot the lvm
21:03:19 <skvidal> s/lvm/lv/
21:03:38 <nirik> #topic Misc items
21:04:04 <nirik> I'd like to suggest we not do updates on friday's... especially on hardware that only has next business day response. ;)
21:04:42 <Oxf13> skvidal: er, you'd have to boot the spare system, unlock it, then copy over the changed files, and shut the backup system down again
21:04:50 <skvidal> Oxf13: no
21:04:56 <nirik> Also, on this outage we didn't have any outage announcement, but should we have? it didn't cause any disruption to end users or developers... just need to consider when we notify about outages and how. (revisit that process)
21:04:58 <smooge> yes. sorry I wasn't watching the date when I ok'd doing updates
21:04:59 <skvidal> Oxf13: you run the primary on an lv
21:05:04 <skvidal> snapshot and dd to a file
21:05:16 <skvidal> run the warm-backup off of the dd
21:05:37 <nirik> smooge: no worries. I think it was me that was saying we should just do them then...
21:06:09 <smooge> I think we both did. I have been working so many weekends I didn't think.
21:06:18 <nirik> yeah, easy to do.
21:06:40 <skvidal> to be fair
21:06:48 <skvidal> this haD NOTHING to do with the updates
21:06:51 <skvidal> the updates were fine
21:06:55 <Oxf13> skvidal: I guess I was confused as to how the backed up data would get onto the shut down encrypted other system.
21:06:56 <skvidal> the kernel was fine
21:07:15 <skvidal> the only issue here is that the  hw pooped on itself
21:07:20 <skvidal> and was unrecoverable
21:07:28 <skvidal> let's not make this into something it is not
21:07:28 <nirik> yeah
21:07:44 <skvidal> it has NOTHING to do with updates nor, even, with rebooting
21:07:56 <skvidal> if we had opted to NOT reboot these boxes
21:08:03 <skvidal> it may well have failed in an identical way
21:08:06 <abadger1999> <nod>
21:08:18 <nirik> true.
21:08:27 <smooge> or worse... failed in ways we didn't see
21:08:36 <skvidal> post hoc ergo propter hoc
21:08:37 * abadger1999 thinks we should also consider doing periodic reboots of the box.
21:08:43 <skvidal> abadger1999: +100000
21:08:47 <skvidal> abadger1999: rebooting ALL OF OUR BOXES
21:08:49 <skvidal> every 150 days
21:08:51 <abadger1999> <nod>
21:08:52 <skvidal> no matter way
21:08:54 <smooge> just not on Friday.
21:08:56 <skvidal> err s/way/what/
21:09:09 <nirik> surely. entropymonkey. :)
21:09:19 <Oxf13> drifting off topic
21:09:33 <nirik> anyhow, shall we close up and take our thoughts to the list/infra&rel-eng meetings?
21:09:44 <skvidal> sure
21:09:50 <abadger1999> <nod>
21:10:02 <nirik> thanks for all the info and brainstorming everyone!
21:10:10 <nirik> #endmeeting