Upgrading debian

[Dave Hodgkinson]

Please oh, mighty Linux warriors.

I have a couple of critical, internet-facing Debian 7 servers. Obviously, I
should take them to 9.

Is it better to try upgrading them in-situ, or doing a re-install? Am I
better asking on a Debian mailing list or Stackoverflow?

They’re really badly puppetised (which is part of the epic for getting
these machines up to date).

Thank you,

// Your humble supplicant.

<https://mysignature.io/editor?utm_source=promotion&utm_medium=signature&utm_campaign=create_own_signature>

[Roger Bell_West]

Post by Dave Hodgkinson
Is it better to try upgrading them in-situ, or doing a re-install? Am I
better asking on a Debian mailing list or Stackoverflow?

In my experience (now about twenty years thereof), Debian upgrades in
place are mostly painless. A service may break, but you'll be warned
about it and the fixes are usually easy.

[Dave Hodgkinson]

That's reassuring. Debian always had a reputation of being fairly
bulletproof.

Post by Roger Bell_West

Post by Dave Hodgkinson
Is it better to try upgrading them in-situ, or doing a re-install? Am I
better asking on a Debian mailing list or Stackoverflow?

In my experience (now about twenty years thereof), Debian upgrades in
place are mostly painless. A service may break, but you'll be warned
about it and the fixes are usually easy.

[David Cantrell]

Post by Roger Bell_West

Post by Dave Hodgkinson
Is it better to try upgrading them in-situ, or doing a re-install? Am I
better asking on a Debian mailing list or Stackoverflow?

In my experience (now about twenty years thereof), Debian upgrades in
place are mostly painless. A service may break, but you'll be warned
about it and the fixes are usually easy.

That's the case in my *recent* experience too, although back in the day
I found that dist-upgrade was more likely to turn your machine into
smoking crater than anything else.

So I very strongly suggest taking a backup first, and testing that you
can do a bare-metal recovery.

[David Precious]

On Wed, 10 Oct 2018 11:38:35 +0100

Post by David Cantrell
So I very strongly suggest taking a backup first, and testing that you
can do a bare-metal recovery.

This - whilst IME also, upgrades have gone pretty smoothly, if it's
mission-critical you want the ability to roll back quickly.

If you can get another box, install a fresh Debian 9 install on it then
copy your stuff over, you still have the old box as your rollback plan,
rather than being left without a propulsion method in an
unpleansantly-named river.

Puppet was mentioned earlier - it'd also be an excellent time to check
that you can in fact Puppet your way to a working box from bare metal,
and if not, fix it so that you can :)

For remote boxes that I've done Debian upgrades from old versions on, I
often make sure there's a KVM either already hooked up or quickly
available, so if it breaks hard enough to not boot up, I can poke at it
remotely.

[Roger Bell_West]

Post by David Precious
For remote boxes that I've done Debian upgrades from old versions on, I
often make sure there's a KVM either already hooked up or quickly
available, so if it breaks hard enough to not boot up, I can poke at it
remotely.

The only times that's happened to me during an upgrade (as distinct from
rolling custom kernels), it was because I was stupid enough to reboot
while systemd was installed.

[James Laver]

I haven’t been too impressed with stretch’s package quality to be honest. Several packages I need like Postgres and ‘manpages’ (yes, even manages!) had packaging bugs that were sufficiently difficult to resolve I gave up and used ubuntu.

/j

Post by Dave Hodgkinson
Please oh, mighty Linux warriors.
I have a couple of critical, internet-facing Debian 7 servers. Obviously, I should take them to 9.
Is it better to try upgrading them in-situ, or doing a re-install? Am I better asking on a Debian mailing list or Stackoverflow?
They’re really badly puppetised (which is part of the epic for getting these machines up to date).
Thank you,
// Your humble supplicant.
<https://mysignature.io/editor?utm_source=promotion&utm_medium=signature&utm_campaign=create_own_signature>

[David Hodgkinson]

Neither of those I need. You could always fix and resubmit :)

Post by James Laver
I haven’t been too impressed with stretch’s package quality to be honest. Several packages I need like Postgres and ‘manpages’ (yes, even manages!) had packaging bugs that were sufficiently difficult to resolve I gave up and used ubuntu.
/j

Post by Dave Hodgkinson
Please oh, mighty Linux warriors.
I have a couple of critical, internet-facing Debian 7 servers. Obviously, I should take them to 9.
Is it better to try upgrading them in-situ, or doing a re-install? Am I better asking on a Debian mailing list or Stackoverflow?
They’re really badly puppetised (which is part of the epic for getting these machines up to date).
Thank you,
// Your humble supplicant.

[James Laver]

Actually, Postgres *is* fixed… But apparently only in unstable…

/j

Post by David Hodgkinson
Neither of those I need. You could always fix and resubmit :)

[Peter Corlett]

Post by Dave Hodgkinson
I have a couple of critical, internet-facing Debian 7 servers. Obviously, I
should take them to 9.
Is it better to try upgrading them in-situ, or doing a re-install? Am I
better asking on a Debian mailing list or Stackoverflow?
They’re really badly puppetised (which is part of the epic for getting
these machines up to date).

Ideally, you should put in the effort to Puppetise them properly and thus make
the wipe-and-reinstall trivial. Better still, also do the research to see if
there's something more appropriate than Puppet out there. (Tip: there will be.)

If you can't be bothered with that -- a not unreasonable attitude given that
"laziness" is one of the Perl virtues -- then an in-place upgrade is much less
likely to go wrong than building a new machine and attempting to merge in the
old data and configuration afterwards. That merging is going to be a similar
level of effort to just Puppetising it.

The introduction of systemd is likely to bite you on the arse, whatever upgrade
approach you take. If you aren't going to just immediately blow it away and put
sysvinit back, you will need to learn about this Brave New World if you haven't
done so already. I strongly recommend you do said learning in advance, and not
in a panic when the upgrade broke the server and nothing on it makes sense.

[Kieren Diment]

http://howfuckedismydistro.com/debian/

Post by Dave Hodgkinson
Please oh, mighty Linux warriors.
I have a couple of critical, internet-facing Debian 7 servers. Obviously,
I should take them to 9.
Is it better to try upgrading them in-situ, or doing a re-install? Am I
better asking on a Debian mailing list or Stackoverflow?
They’re really badly puppetised (which is part of the epic for getting
these machines up to date).
Thank you,
// Your humble supplicant.
<https://mysignature.io/editor?utm_source=promotion&utm_medium=signature&utm_campaign=create_own_signature>

[Dave Cross]

Post by Dave Hodgkinson
Please oh, mighty Linux warriors.
I have a couple of critical, internet-facing Debian 7 servers. Obviously, I should take them to 9.
Is it better to try upgrading them in-situ, or doing a re-install? Am I better asking on a Debian mailing list or Stackoverflow?
They’re really badly puppetised (which is part of the epic for getting these machines up to date).

I assume there's a good reason why you're not Dockerising them and
sticking them on AWS[1]?

Dave...

[1] Other cloud providers are available

[Peter Corlett]

On Wed, Oct 10, 2018 at 12:13:00PM +0100, Dave Cross wrote:
[...]

I assume there's a good reason why you're not Dockerising them and sticking
them on AWS[1]?

Not all servers are cattle, and some cattle may not gain useful benefit from
Dockerisation. Docker is a rather Special bit of software which adds an extra
layer of risk and complexity, so why bother where there's no benefit?

As to AWS, it is rather expensive. Partly because reliability is expensive to
engineer the more nines you add, and partly because they can charge what the
market will bear and the main market for that kind of service are people
spending other people's money. And just like Docker, AWS adds a layer of risk
and complexity that may not be justified.

Also, sometimes you just can't beat an old PC running Linux shoved under a desk
somewhere: Low-latency gigabit connectivity, as much storage as you can wedge
into the case, and no credit card required to sign up.

[1] Other cloud providers are available

I quite like Hetzner when I'm spending my own money. There are many other
discount hosting operations, but Hetzner hits the sweet spot for me between
having a reasonable price and not looking like a fly-by-night pink provider.

I'm still mostly using dedicated servers, but have given the Hetzner Cloud
offering a spin and rather liked what I saw. The feature that sold it to me was
the ability to connect to the virtual display and keyboard and poke at a cloud
instance that's not responding to the network, which I don't even get on my
much more expensive dedicated servers. This can also be used to enter the
passphrase on servers with fully-encrypted disks.

AWS doesn't seem to have anything like this, only the ability to inspect the
last few lines logged to the serial console and being able to turn the server
off and on again to try and bring it back to life.

[Gareth Harper]

In my experience it's pretty painless, I spent 8 years upgrading, however
it very much depends if you are using a couple of system packages (apache
and perl). If you are then you will probably have to redo your apache
config (more modern versions have substantially changed config options.

Perl has also been upgraded a couple of times so you'll need to reinstall
all your perl modules unless you've used a seperate perl installation
(something I now do specifically because of this problem).

Post by Dave Hodgkinson
Please oh, mighty Linux warriors.
I have a couple of critical, internet-facing Debian 7 servers. Obviously,
I should take them to 9.
Is it better to try upgrading them in-situ, or doing a re-install? Am I
better asking on a Debian mailing list or Stackoverflow?
They’re really badly puppetised (which is part of the epic for getting
these machines up to date).
Thank you,
// Your humble supplicant.
<https://mysignature.io/editor?utm_source=promotion&utm_medium=signature&utm_campaign=create_own_signature>

[Dirk Koopman]

Over the years I have done both things. In general an in place upgrade
works just fine, but recent experience suggests that, particularly in
the networking area (possibly to do with going onto systemd), it has
been less happy (NIC  bonding in my case). Now I don't want to start a
systemd flame war - especially as I am using a systemd Ubuntu box to
type this on - but we "fixed" the problem we had with Devuan.

Personally, depending on how hard it is to achieve for you, I would be
inclined to backup and start afresh - particularly as you will be going
to a systemd based system. Especially if you are going from 7 -> 9 it
will almost certainly be quicker as well. People forget just how long
the upgrade process can take, especially if you intend to go through 8
on the way.

There is also a competition to be had here of course, once you have the
backup in place (do keep a list of installed packages as well - very
useful), do both and report back here. Your findings would be useful to
us all.

Regards

Dirk

Post by Dave Hodgkinson
Please oh, mighty Linux warriors.
I have a couple of critical, internet-facing Debian 7 servers.
Obviously, I should take them to 9.
Is it better to try upgrading them in-situ, or doing a re-install? Am
I better asking on a Debian mailing list or Stackoverflow?
They’re really badly puppetised (which is part of the epic for getting
these machines up to date).
Thank you,
// Your humble supplicant.
<https://mysignature.io/editor?utm_source=promotion&utm_medium=signature&utm_campaign=create_own_signature>

[Dave Hodgkinson]

In other news, if/when Debian 9 goes into circulation, it'll bring up a
load of "experimental" perl which may have been deprecated in perl 5.24.1.

Did I say ARGH? Still, not my problem. Only my responsibility.