Implementation of root-re-sign operation on March 12, 2016 ---------------------------------------------------------- @(#)(CAcert) $Id$ This document is based on the procedure.txt maintained in https://github.com/BenBE/cacert-procedures/blob/root-resign-sha256/rootResignSHA256/procedure.txt and is maintained itself in https://svn.cacert.org/CAcert/SystemAdministration/signer/re-sign-2016/implementation.txt Attendees: ========== * Bas van den Dikkenberg (secure-u access engineer) * Mendel Mobach (CAcert critical system administrator) * Martin Simons (CAcert critical system administrator) * Benedikt Heintel (CAcert internal auditor) Materials brought in: ===================== 1. Ubuntu Live DVD: ubuntu-14.04.4-desktop-amd64.iso, with sha256 checksum: 7d9337e7b19d74b197100035ffa4318999e2d6a875bba020cfe197faeb98e904 (prepared by Wytze van der Raay, CAcert critical system administrator teamleader). 2. USB stick with ext2 file system containing the following files from https://svn.cacert.org/CAcert/SystemAdministration/signer/re-sign-2016/inputs: execute.sh main.c libssl1.0.0_1.0.1f-1ubuntu2.18_amd64.deb zlib1g-dev_1.2.8.dfsg-1ubuntu1_amd64.deb libssl-dev_1.0.1f-1ubuntu2.18_amd64.deb The sha256 checksum for execute.sh is: d9c28d34df10f62175205e5b41ec3fda96de2e7676f33dbd45d0899c2969546e Execution of the execute.sh script will verify the sha256 checksums of the other four files. (prepared by Wytze van der Raay, CAcert critical system administrator teamleader). 3. USB stick to be formatted and filled in the process (brought by Bas van den Dikkenberg, secure-u access engineer). 4. PC without connected hard disk or network, for the preparation phase (brought by Bas van den Dikkenberg, secure-u access engineer). 5. This document (prepared by Wytze van der Raay, CAcert critical system administrator teamleader). Preparation phase ================= The preparation phase will be conducted in a guest room at the BIT-2 data center. After installing the PC brought in, it should be inspected to make sure that no hard disk(s) or network(s) are connected to it. Preferably some pictures should be made. 1. Boot the PC with the Ubuntu live DVD, start a terminal and change user to root with "sudo -s" 2. Verify the integrity of the boot DVD by running the following command: # dd if=/dev/sr0 ibs=2048 | sha256sum and compare the checksum displayed with the one mentioned above for this DVD. 3. Insert the USB stick to be formatted in the PC and determine its name from dmesg output (ignore/click away any popup screens). Assuming it is /dev/sda, issue the following commands to format: # fdisk /dev/sda delete all existing partition(s) create a new primary partition 1 write (save) the partitioning # mke2fs /dev/sda1 # eject /dev/sda 4. Shutdown the live DVD annd power off the PC. 5. Remove the formatted USB stick and save it. 6. Boot the PC with the Ubuntu live DVD, start a terminal and change user to root with "sudo -s" 7. Create a working directory (in ram since there is nothing else): # mkdir /ramdisk; cd /ramdisk 8. Start logging the rest of the operations: # script -ttimelog.prep script.prep 9. Insert the prepared USB stick with the 5 files in the PC, and determine its name from the dmesg output (ignore/click away any popup screens). Assuming it is /dev/sda, issue the following commands: # mount -r /dev/sda1 /mnt # cp -p /mnt/*.deb /mnt/execute.sh . # mkdir compilation # cp -p /mnt/main.c compilation # umount /mnt # eject /dev/sda # ls -laR 10. Verify the checksum of the copied execute.sh script: # sha256sum execute.sh against the one listed above. 11. Insert the formatted USB stick (still empty) in the PC, and determine its name from the dmesg output (ignore/click away any popup screens). Assuming it is /dev/sdb, mount it with the following command: # mount /dev/sdb1 /mnt 12. Start the execute.sh script to verify the remaining checksums, compile the utility and generate key pair and signatures for verification afterwards. # ./execute.sh This will produce output that roughly looks like: --------------------------------------------------------------------------------------------- Are the 3 deb-packages in /ramdisk? Is only the 'main.c' in /ramdisk/compilation? Is only the empty usb stick connected and mounted in /mnt? Did you verify my fingerprint? Checking checksums of own files ./compilation/main.c: OK libssl1.0.0_1.0.1f-1ubuntu2.18_amd64.deb: OK libssl-dev_1.0.1f-1ubuntu2.18_amd64.deb: OK zlib1g-dev_1.2.8.dfsg-1ubuntu1_amd64.deb: OK (Reading database ... 100%^ (Reading database ... 171422 files and directories currently installed.) Preparing to unpack libssl1.0.0_1.0.1f-1ubuntu2.18_amd64.deb ... Unpacking libssl1.0.0:amd64 (1.0.1f-1ubuntu2.18) over (1.0.1f-1ubuntu2.16) ... Selecting previously unselected package libssl-dev:amd64. Preparing to unpack libssl-dev_1.0.1f-1ubuntu2.18_amd64.deb ... Unpacking libssl-dev:amd64 (1.0.1f-1ubuntu2.18) ... Selecting previously unselected package zlib1g-dev:amd64. Preparing to unpack zlib1g-dev_1.2.8.dfsg-1ubuntu1_amd64.deb ... Unpacking zlib1g-dev:amd64 (1:1.2.8.dfsg-1ubuntu1) ... Setting up libssl1.0.0:amd64 (1.0.1f-1ubuntu2.18) ... Setting up zlib1g-dev:amd64 (1:1.2.8.dfsg-1ubuntu1) ... Setting up libssl-dev:amd64 (1.0.1f-1ubuntu2.18) ... Processing triggers for man-db (2.6.7.1-1ubuntu1) ... Processing triggers for libc-bin (2.19-0ubuntu6.7) ... Generating RSA private key, 2048 bit long modulus ...........................................................................................................................+++ ...............................+++ e is 65537 (0x10001) writing RSA key Write down the pubkey fingerprint: writing RSA key (stdin)= 75453f4a82fa6b57c460fb1ece67a8d392093d9c605dfb8428900dcdd5ca852c /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcrypto.a(dso_dlfcn.o): In function `dlfcn_globallookup': (.text+0x11): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking ================================ things left to do: cp typescript timelog /mnt ls -Al /mnt umount /mnt --------------------------------------------------------------------------------------------- After the "Did you verify my fingerprint?" question, a return should be entered. After the "Write down the pubkey fingerprint:", the value after (stdin)= should be recorded on paper by all attendees. After that a return can be entered. Note that the compilation of the utility produces a warning which can be ignored safely. 13. Terminate the logging session with: # exit 14. Perform the final steps to save and inspect all results: # cp -p typescript.prep timelog.prep /mnt # ls -Al /mnt # umount /mnt # eject /dev/sdb and remove the USB stick. 15. Ensure destruction of the private key generated in the process by shutting down the PC and turning its power off. If you can spare the time, perform a full run of Memtest86+ (contained on the Ubuntu Live DVD). The just generated USB stick should be kept by the secure-u access engineer, and carried to the location of the signing server for the second part of this procedure. Re-signing phase ================ The re-signing phase will be performed in the machine room of the BIT-2 data center where the CAcert servers are located. The only materials needed from the previous phase are the USB stick created in that phase, the fingerprint written down in the preparation phase and this text document. The CAcert critical system administrators will bring their private copy of the system passwords required to gain access to the server via its console. 1. Connect a terminal to the signing server and login as root. 2. Create a ramdisk for performing the re-sign and enter it: # mkdir /tmp/ram-re-sign # mount -t tmpfs none /tmp/ram-re-sign # chmod 700 /tmp/ram-re-sign # cd /tmp/ram-re-sign 3. Start logging the rest of the operations: # script -t script.re-sign 2>timelog.re-sign 4. Plug in the USB stick with the results from the preparation phase and use dmesg to determine which disk drive to use for mounting it. Assuming it is /dev/sdc, issue the following command: # mount /dev/sdc1 /mnt # cp /mnt/main . 5. Validate the signatures of the files on the USB thumb drive like this: # cp /mnt/checksums* /mnt/signature /mnt/signkey.pub . # openssl dgst -sha256 checksums > checksums.verifyhash # openssl rsautl -verify -keyform PEM -inkey signkey.pub -pubin -in signature > signature.verifysig # cat checksums.verifyhash signature.verifysig # cmp checksums.verifyhash signature.verifysig On success, no output should result from the cmp. # sha256sum -c checksums Should display main: OK, and failures to open for main.o and main.c. 6. Setup for re-signing and do it: # cp -p /etc/ssl/CA/cacert.crt root.crt # openssl rsa root.key Note: this will ask for the pass phrase, which is well-known. # cp -p /etc/ssl/class3/cacert.crt class3.crt # ./main 7. Inspect the results: # openssl x509 -in root.crt -noout -text >old1.txt # openssl x509 -in root_256.crt -noout -text >new1.txt # diff old1.txt new1.txt Observe the changed fields. # openssl x509 -in root_256.crt -outform der | openssl dgst -sha256 Write down the fingerprint of root_256.crt # openssl x509 -in class3.crt -noout -text >old3.txt # openssl x509 -in class3_256.crt -noout -text >new3.txt # diff old3.txt new3.txt Observe the changed fields. # openssl x509 -in class3_256.crt -outform der | openssl dgst -sha256 Write down the fingerprint of class3_256.crt 8. Copy the results back to the USB stick: # cp -p root_256.crt class3_256.crt /mnt # cp -p old1.txt new1.txt old3.txt new3.txt /mnt 9. Copy the results to the actual signer: # cp -p root_256.crt /etc/ssl/CA # cp -p class3_256.crt /etc/ssl/class3 10. Terminate the logging session with: # exit 11. Perform the final steps to save and inspect all results: # cp -p typescript.re-sign timelog.re-sign /mnt # ls -Al /mnt # umount /mnt # eject /dev/sdc and remove the USB stick. 12. Destroy the ram disk: # cd / # umount /tmp/ram-re-sign 13. Logout from the signing server, and disconnect the console. Post-execution Steps ==================== 1. Connect a terminal to the webdb server and login as mendel or martin. 2. Insert the USB stick from the previous step in the webdb server, and use dmesg to figure out as which disk it shows up. Assuming /dev/sdd in the following. 3. Mount the stick and copy the entire contents to a safe directory for later perusal by CAcert critical system administrators: $ sudo -s # mount /dev/sdd1 /mnt # mkdir /home/signer/re-sign-20160312 # cp -p -r /mnt /home/signer/re-sign-20160312 # ls -Al /home/signer/re-sign-20160312 # umount /mnt # eject /dev/sdd 4. Logout from the webdb server and disconnect the console terminal. Final Steps =========== 1. Pass all materials used to internal auditor for later inspection. This includes: 1. the Ubuntu Live CD 2. the USB stick with program input 3. the USB stick with all results 2. Report results on cacert-systemlog@lists.cacert.org 3. Any other steps as indicated in the procedure.txt document referred in the beginning (not time-critical). -- end