Tag Archives: crypto

Curious SMART failure

Uncategorized

My main computer has 5 physical hard drives installed.  There’s my primary drive, and the live backup that is synchronized to it every night.  There are two large drives concatenated to make a backup volume for the MythTV box.  And there’s a separate drive that holds work-related code and files.

I have had many hard drive failures over the years, fortunately my backup routine has prevented me from losing anything important.

In light of the number of hard drives in the machine, I recently decided to start using SMART monitoring software on the main drive, its backup, and the work drive.  A daemon running on Linux periodically performs short and long tests of the hard drives.  Every Sunday, a report is generated and emailed to me to notify me of any developing issues with the hard drives.  The assumption is that, before outright failure, hard drives are likely to show degradation that manifests in these tests, allowing the user to prepare for the imminent loss of the drive.

In my latest Sunday report, all three drives reported normal conditions.  No signs of impending failure were noted.  Later that afternoon, the SMART daemon started issuing non-routine email messages indicating that it was unable to perform SMART queries on two drives, the work drive and my main backup drive.

It’s unreasonable to expect that two drives failed without warning, simultaneously, mere hours after getting a clean bill of health from SMART, so something unusual must have been happening.  Both the work drive and the backup drive were delivering SATA errors on any activity, they couldn’t be mounted, and files could not be read from them.  The emails indicated that the work drive failed first, and the primary backup drive about thirty minutes later.

My first thought was that the SATA bus was confused.  Maybe a BIOS error, a kernel bug, the infamous “cosmic ray”, or something else.  With the SATA bus unreliable, all activities, including SMART lookups, could fail.  So the first thing to try was a power cycling of the box.  After that, both drives were again mountable, and files could be read.

Less than an hour later, the SMART daemon started sending its messages again.  Work drive failure, followed half an hour later by backup drive failure.  The new theory was that the work drive was failing in a curious way.  When it received SMART queries, issued periodically by the daemon, the drive responded in such a way as to confuse the SATA bus or the kernel module responsible for handling it.

So, I bought a new hard drive to replace the failing work drive.  I modified my startup system to avoid activating the SMART daemon while I ran a new backup.  The work drive isn’t fully backed up every night, as many of the files are mirrored on other work computers, but just to be sure, I did a full backup of that drive.  This turned out to be straight-forward, without the SMART daemon issuing queries the drive and the SATA bus remained stable throughout the process.  Once the backup was completed, I powered down again and replaced the drive.  Set up the encryption on the new drive, formatted it, and recovered from the backup.

I’ve reactivated the SMART daemon, and the system has remained stable.  This leads me to believe that I had an unexpected SMART failure, where the only visible problem with the drive was that SMART queries messed up the state, leading to the inability to use not only that drive, but another unrelated drive in the computer.

When distributions patch wrongly

Uncategorized

Events of recent weeks have provided another reason one might be inclined to avoid the use of distributions. Let’s call this the “debian SSL bug”. A patch applied by a well-meaning Debian coder made cryptographic keys generated by numerous applications on that distribution entirely useless. Details can be found here.

The Debian patch affected derived distributions as well, such as Ubuntu. For almost two years, many cryptographic transactions were severely compromised. The biggest problem was that the patch was not correctly passed back to development team of the OpenSSL project. Had it been, they would have pointed out its fatal security implications, and this entire headache would have been avoided.

I always feel uncomfortable when I see distributions applying patches against the original sources. There can be several reasons for these patches.

  1. They may be back-porting selected bugfixes to an earlier version of a library rather than including the latest version of the library with all of its new, and possibly untested features.
  2. They may be modifying a logo or informational string to include something specific to the distribution.
  3. They may be changing some default pathnames or other resources to mesh more well with the idiosyncracies of their own distribution.
  4. They may be changing the appearance of the interface to make it more consistent with other applications.
  5. They may be applying changes that the original maintainers of the package do not consider necessary, but which the distribution maintainers find desirable.
  6. Other…

None of these motivations will usually convince me to apply foreign patches. Your opinion may differ.

Web browsing behind the great firewall of China

Uncategorized

I sometimes spend time in China, and while there, I work remotely to my office and to my home computer. I do somewhat technical work that sometimes requires online research, and it’s annoying that a significant fraction of non-Chinese sites are unreachable from China.

The thing to remember is that the firewall isn’t there to keep me from working. I’m a Canadian passport holder, and they really don’t care what I read while in China. That explains certain curious omissions, such as the fact that TCP port 22 (ssh) is not blocked.

So, here I am, in China, with a Linux laptop, and I’d like to browse the web. Rather than take my chances with the firewall, I proxy the connection through my home computer’s apache daemon.

So, first I set up the proxy service on my apache. Make sure you’ve built the httpd with these configuration options:

--enable-mods-shared="proxy proxy-http proxy-connect"

These settings turn on the proxy service and set it to proxy HTTP traffic. The “proxy-connect” flag allows the httpd to be used as a reflector for SSL connections. If you want to visit a banking website, the data still travels as SSL between your laptop and the home machine, but the home machine just reflects the traffic to the bank without knowing what’s in the data stream (the home machine cannot decode that data, if it could, it would count as a man-in-the-middle compromise of the SSL stream).

Next, add some lines to the httpd configuration file. Mine’s in /etc/apache/httpd.conf.

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so


ProxyRequests On


 Order deny,allow
 Deny from all
 Allow from 127.0.0.1


What this does is to enable proxying, but only on connections from localhost. I don’t want my httpd to be a proxy for any random person in the outside world.

Next, I set up my ssh on connections to my home computer. You can either add a switch like this to the invocation:

-L 8080:127.0.0.1:80

or you can add a line to your ~/.ssh/config entry for the connection to the home computer:

LocalForward 8080 127.0.0.1:80

Now, you ssh into your home computer.

Finally, you start up firefox, and select the menu item:
Edit->Preferences->Advanced->Network->Settings
Select “Manual proxy configuration”, and point your HTTP and SSL proxies at “localhost” with the port number 8080.

That’s it, now when you browse websites, the HTTP-related data stream appears simply as a pile of encrypted bits over your ssh connection. The firewall cannot know what websites you’re visiting, it can’t even tell that you’re visiting a website at all.

Important note: this system proxies the HTTP data. That means web pages, frames, images in the page, RSS feeds, and so on. It does not proxy UDP or post-connection traffic, like youtube videos. If your web browser has a plugin that downloads data from an external site, that plugin may not be using your proxy.

If you want to know what data is not passing through your proxy, you can run tcpdump in another window. Something like this:

tcpdump 'host  and not port 22'

where is the IP number of your external interface (not 127.0.0.1). You may have to add a “-i” switch if your laptop has more than one network interface. This command will show you all traffic that is not going over the ssh connection.

Selective sendmail relaying based on self-signed keys

Uncategorized

Back in the early days of the Internet, people trusted one another not to abuse email. Sure, there were accidents. A badly configured mailing list could fill up with traffic as two vacation programs talked to one another, each informing the other that his latest message would not be read until some later date, because the recipient was out of the office.

In those days, you set up your sendmail to relay messages for others. Many people had email addresses that weren’t on a full-time connection to the network, they might be on a BBS that did a nightly download of email, or down some Bitnet rabbit hole. Email was relayed from one intermediate post to another, rather than being simply sent directly from the sender to the receiver. A sendmail daemon that relayed messages for others was helpful to the community, everybody pitched in to get everyone’s email where it was ultimately intended.

Then came new developments. Canter & Siegel, the September that never ended, and the presence of people who would buy things they saw in an unsolicited email message. Spam started to appear in mailboxes. Suddenly, being a helpful person and relaying messages was no longer beneficial to the community, as commercial email senders used relays to hide the origins of their messages. People started turning off open relays on their boxes as a defensive move.

So, now you’ve got a domain set up with a sendmail daemon at home, and you’re traveling with a laptop. To make this a bit more complicated, let’s say your laptop is a work computer, and you send email from its sendmail, but with a different domain than your home computer. Everything’s working fine, until you find that the coffee shop in Beijing where you’re using your laptop has made it onto a list of spamming IP numbers. Some recipients of your messages may not receive them because their sendmail is set up to refuse messages from computers on these bad IP numbers. You know that your home computer is not on a banned IP number, so it would be nice if you could forward your laptop-generated work-related messages through your home computer. It would be even nicer if people selling generic pharmaceuticals could not do the same thing, otherwise your home computer’s IP number will very quickly find itself on one of those banned lists. So, you want to allow relaying from your laptop, but only from your laptop, and do it easily even if you move to another coffee shop.

What you want, then, is a way for your home computer to recognize your laptop, and permit only that computer to relay messages through the home sendmail. This will be done with sendmail’s TLS facility. You will create a private certificate authority, one you don’t have to pay to sign your keys. You’ll then use a signed certificate to verify the identity of the laptop. The following procedure will be performed on the home computer, only at the end of this process will the laptop be involved.

We’ll start by creating two directories on your home computer, one for the certificate authority, and the other for the signed certificates. I’ll use the directory locations that are found in the default OpenSSL configuration file, so that you don’t have to edit too many files.

mkdir /etc/mail/CA /etc/mail/certs /etc/mail/CA/demoCA /etc/mail/CA/demoCA/private

Copy the OpenSSL openssl.cnf file into /etc/mail/CA.

Next, we will create the signing certificate.

$ cd /etc/mail/CA
$ openssl req -new -x509 -keyout demoCA/private/cakey.pem -out demoCA/cacert.pem -days 1000 -config openssl.cnf

You will be prompted for several fields, such as country code, location, name. Here’s a sample dialogue:

$ openssl req -new -x509 -keyout demoCA/private/cakey.pem -out demoCA/cacert.pem -days 1000 -config openssl.cnf
Generating a 1024 bit RSA private key
.............++++++
.........++++++
writing new private key to 'demoCA/private/cakey.pem'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:CA
State or Province Name (full name) [Some-State]:Ontario
Locality Name (eg, city) []:Toronto
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Example
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []:Bert Ificate
Email Address []:bertificate@example.com

When prompted, you will have to enter a pass phrase twice. Remember this phrase, you will need it if you ever want to sign certificates with this signing certificate.

This command creates new files: /etc/mail/CA/demoCA/cacert.pem and /etc/mail/CA/demoCA/private/cakey.pem. The file contains encoded information related to a certificate signing authority that will be valid for 1000 days.

Next, you must create the certificate that you will use to validate your laptop. You enter the commands:

$ cd /etc/mail/CA
$ openssl req -nodes -new -x509 -keyout laptopcert.pem -out laptopcert.pem -days 365 -config openssl.cnf

Again, you will have to answer some questions. Here is a sample dialogue:

$ openssl req -nodes -new -x509 -keyout laptopcert.pem -out laptopcert.pem -days 365 -config openssl.cnf
Generating a 1024 bit RSA private key
....++++++
............................................++++++
writing new private key to 'laptopcert.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:CA
State or Province Name (full name) [Some-State]:Alberta
Locality Name (eg, city) []:Calgary
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Example
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []:Rhoda Warrior
Email Address []:rhoda-warrior@example.com

Now, you have a certificate for your laptop, but it hasn’t yet been signed. You use the signing certificate to vouch for the laptop certificate. First, we have to set up a bit more information for the signing process:

$ mkdir /etc/mail/CA/demoCA/newcerts
$ touch /etc/mail/CA/demoCA/index.txt
$ echo 01 > /etc/mail/CA/demoCA/serial

You’ll only have to do this the first time you set up a signing authority.

Now, we issue two commands to sign the laptop certificate:

$ openssl x509 -x509toreq -in laptopcert.pem -signkey laptopcert.pem -out tmp.pem
$ /usr/local/ssl/bin/openssl ca -config openssl.cnf -policy policy_anything -out signed-laptopcert.pem -infiles tmp.pem

Once again, there will be a brief dialogue when the second command is run, something like this:

$ openssl ca -config openssl.cnf -policy policy_anything -out signed-laptopcert.pem -infiles tmp.pem
Using configuration from openssl.cnf
Enter pass phrase for ./demoCA/private/cakey.pem:
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 1 (0x1)
        Validity
            Not Before: Mar 12 00:46:43 2008 GMT
            Not After : Mar 12 00:46:43 2009 GMT
        Subject:
            countryName               = CA
            stateOrProvinceName       = Alberta
            localityName              = Calgary
            organizationName          = Example
            commonName                = Rhoda Warrior
            emailAddress              = rhoda-warrior@example.com
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            Netscape Comment:
                OpenSSL Generated Certificate
            X509v3 Subject Key Identifier:
                67:11:5A:25:6F:5C:70:36:03:14:3B:04:4A:8C:30:C9:CF:60:51:AE
            X509v3 Authority Key Identifier:
                keyid:F4:CE:58:BD:82:8A:E3:EC:0F:89:C6:60:E2:45:58:A4:CA:79:C8:89

Certificate is to be certified until Mar 12 00:46:43 2009 GMT (365 days)
Sign the certificate? [y/n]:y


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated

Now, it’s time to tell the home machine’s sendmail that it should relay messages received from this key. Add a line to the /etc/mail/access.src file that looks like this:

CertIssuer:/C=CA/ST=Ontario/L=Toronto/O=Example/CN=Bert+20Ificate/emailAd
dress=bertificate@example.com RELAY

You’ll have to make that file readable by sendmail:

makemap hash access.db < access.src

And now we have to make sure that the home machine’s sendmail knows where to find its certificates and access file. Build a new sendmail.cf using a sendmail.mc something like this:

divert(0)dnl
VERSIONID(`sendmail.mc for example.com version 01')
OSTYPE(linux)dnl
DOMAIN(example.com)dnl
FEATURE(`nouucp', `reject')
FEATURE(`virtusertable', `hash /etc/sendmail/virtusertable')dnl
FEATURE(`genericstable', `hash /etc/sendmail/genericstable')dnl
FEATURE(`local_procmail', `/usr/local/bin/procmail')
FEATURE(`access_db', `hash -T /etc/mail/access')
FEATURE(`mailertable')
MAILER(local)
MAILER(smtp)
define(`CERT_DIR', `MAIL_SETTINGS_DIR`'certs')dnl
define(`confCACERT_PATH', `CERT_DIR')dnl
define(`confCACERT', `CERT_DIR/CAcert.pem')dnl
define(`confSERVER_CERT', `CERT_DIR/MYcert.pem')dnl
define(`confSERVER_KEY', `CERT_DIR/MYkey.pem')dnl
define(`confCLIENT_CERT', `CERT_DIR/MYcert.pem')dnl
define(`confCLIENT_KEY', `CERT_DIR/MYkey.pem')dnl

Now, we move some things around a bit. We copy the signing certificate and laptop signed certificate like this:

$ cd /etc/mail/CA
$ /bin/cp signed-laptopcert.pem /etc/mail/certs
$ /bin/cp demoCA/cacert.pem /etc/mail/certs/CAcert.pem
$ cd /etc/mail/certs
$ ln -s signed-laptopcert.pem `openssl x509 -noout -hash < signed-laptopcert.pem`.0

The three files, demoCA/cacert.pem, laptopcert.pem and signed-laptopcert.pem get copied onto the laptop, in its /etc/mail/certs directory. Now, you must tell the laptop’s sendmail that these are its certificates. This is done by building (on the laptop) the sendmail.cf file from a sendmail.mc file that looks roughly like this:

divert(0)dnl
VERSIONID(`$Id: generic-linux.mc,v 8.1 1999/09/24 22:48:05 gshapiro Exp $')
OSTYPE(linux)dnl
DOMAIN(example.net)dnl
define(`confCACERT_PATH', `/etc/mail/certs/')
define(`confCACERT', `/etc/mail/certs/cacert.pem')
define(`confCLIENT_CERT', `/etc/mail/certs/laptopcert.pem')
define(`confCLIENT_KEY', `/etc/mail/certs/signed-laptopcert.pem')
define(`confSERVER_CERT', `/etc/mail/certs/laptopcert.pem')
define(`confSERVER_KEY', `/etc/mail/certs/signed-laptopcert.pem')
FEATURE(`genericstable')
FEATURE(`virtusertable')
FEATURE(`local_procmail', `/usr/local/bin/procmail')
MAILER(local)dnl
MAILER(smtp)dnl

Finally, you’ll have to decide when you want to relay through the home computer. You really have two choices. You could set it up so that all messages are always relayed through the home computer, by setting a smart relay in your sendmail.cf, or you could relay them explicitly. There are other places that identify the technique for setting up a smart relay, so I’ll just describe the second, on-demand version.

If you are trying to send email from your laptop to the user somebody@example.net, but want to relay it through your home computer at example.com, you would send the message to this email address:

somebody%example.net@example.com

And there you go, on-demand secure relaying of messages through your home computer.

A Followup On Cryptographic Mounts, The Bad News

Uncategorized

Previously, I discussed cryptographic mounts to hold sensitive data. It’s worth pointing out an article that is making the rounds today by 9 authors from Princeton, in which the researchers describe an attack on cryptographic techniques, including the one I’ve described.

The technique relies on the fact that modern memory can retain its information for several minutes after the computer stops sending it refresh signals. What this means is that a person with physical access to the computer can pull the power connector from the computer and then remove the memory chips, insert them in another computer, and read the cryptographic keys out of the memory. I don’t know of a good way to avoid this attack. If the cryptographic volumes are mounted when the computer falls into the hands of the attacker, the data will be, in theory, recoverable.

So, what can be done to prevent the key from being resident in the computer’s memory at the instant that the attacker unplugs it? The key has to be available to the operating system so that it can read and write that data in normal operation. Sure, you could get specially modified hardware that deliberately overwrites the main memory from batteries when the power connector is removed, but maybe there’s a way to store 128 bits somewhere other than in main memory?

A cache line on a modern CPU is 64 bytes, big enough to hold two 128-bit keys. Could the operating system subvert the hardware’s L1 caching mechanism sufficiently to pin a value in the cache and remove it from L2 and main memory? This attack won’t recover data from the L1 cache, so if that’s the only place the key is kept, maybe that would be enough. You sacrifice a cache line, but maybe it’s worth it?

How about the TLB? That’s another part of the CPU that holds data, and that one is explicitly designed to interact with the operating system. Could we find a way to store 128 bits in parts of the TLB, and then deliberately avoid overwriting them? Can the operating system read those numbers back out of the TLB?

Are there any registers that could be used? Probably not on 32-bits, there aren’t many registers there, and on 64-bits you’d probably have to use a special-purpose compiler to avoid these registers being touched by a context switch, and avoid them being saved to memory when an interrupt handler runs.

What if you have fifteen keys, all of 128 bits? Well, I believe we could handle that if we had 256 bits of volatile storage space. The first 128 bits of volatile space holds an XOR key, that decodes all of the fifteen keys. The second 128 bits of volatile space holds the decoded key in active use.

Those are my thoughts, anyway.