Jekyll2024-03-14T09:59:55-07:00https://insufficient.coffee/feed.xmlInsufficient.CoffeeAttending Real World Crypto and the Open Source Cryptography Workshop 20242024-03-14T00:00:00-07:002024-03-14T00:00:00-07:00https://insufficient.coffee/2024/03/14/rwc-and-oscw-2024<p>I’ll be attending the <a href="https://rwc.iacr.org/2024/">Real World Crypto Symposium in Toronto</a> in two weeks time, and after that, I’m once again co-organizing the <a href="https://opensourcecryptowork.shop/2024/">Open Source Cryptography Workshop</a>.<!-- more --></p>
<figure>
<a href="/content/2024/03/oscw2024.560.png">
<picture>
<source type="image/webp" srcset="/content/2024/03/oscw2024.560.webp" />
<img src="/content/2024/03/oscw2024.560.png" />
</picture>
</a>
</figure>
<p>I’ll also be real happy to talk about the new developments with the <a href="https://sunlight.dev/">Sunlight Certificate Transparency log design</a>, Let’s Encrypt’s new <a href="https://datatracker.ietf.org/doc/draft-ietf-acme-ari/">ACME Renewal Information</a> draft specification, <a href="/tag/crlite">CRLite</a>, Rustls… all that stuff.</p>
<p>And of course, I’ll have my stickers collection. And Minisquish. Just ask.</p>I’ll be attending the Real World Crypto Symposium in Toronto in two weeks time, and after that, I’m once again co-organizing the Open Source Cryptography Workshop.Make It Memory Safe: Adapting curl to use Rustls2023-03-30T00:00:00-07:002023-03-30T00:00:00-07:00https://insufficient.coffee/2023/03/30/opensource-crypto-workshop-rustls-ffi<p>As I mentioned in my post about attending <a href="/2023/03/21/rwc-and-opensource-crypto-workshop-2023/">Real World Crypto 2023 and the Open Source Cryptography Workshop</a>, I’ve given a talk discussing <a href="https://github.com/rustls/rustls-ffi/">Rustls-FFI</a> and the work to allow <code class="language-plaintext highlighter-rouge">curl</code> and <code class="language-plaintext highlighter-rouge">libcurl</code> to use the Rust-based, memory-safe <a href="https://github.com/rustls/rustls">Rustls TLS library</a> in a talk called <em>Make It Memory Safe: Adapting Curl to use Rustls</em>.<!-- more --></p>
<p>Pulling Rustls into Curl was a <a href="https://www.memorysafety.org/initiative/rustls/">Prossimo initiative</a> at <a href="https://www.isrg.org/about/">my day job</a>.</p>
<p>This was supposed to be an in-person talk, but <a href="/2023/03/25/nevermind-about-rwc-and-oscw/">I got sick instead</a>. I recorded this talk a few times, the first ones while still quite sick and, uh, oof. But in this one, I only have a cough (that I avoided triggering, hurray)!</p>
<p>Mass kudos to <a href="https://jacob.hoffman-andrews.com/">Jacob Hoffman-Andrews</a>, the primary author of <a href="https://github.com/rustls/rustls-ffi/">Rustls-FFI</a>, for both the excellent work on the library but also for answering all my questions in preparation. All kudos to him, all blame for mistakes and errors in the talk to me!</p>
<video controls="" loop="1" poster="/content/2023/03/2023-03-ocsw-Rustls-FFI-poster.jpg">
<source type="video/webm; codecs=vp9" src="https://i.have.insufficient.coffee/2023-03-ocsw-Rustls-FFI.webm" />
<source type="video/m4v" src="https://i.have.insufficient.coffee/2023-03-ocsw-Rustls-FFI.mp4" />
Your browser does not support HTML5 video. You can download it directly in one of these formats:
<ul>
<li><a href="https://i.have.insufficient.coffee/2023-03-ocsw-Rustls-FFI.mp4">maximum quality</a></li>
<li><a href="https://i.have.insufficient.coffee/2023-03-ocsw-Rustls-FFI.webm">webm vp9 256Kbit quality</a></li>
</ul>
</video>
<p>In the talk, I mention that <a href="https://www.memorysafety.org/initiative/rustls/rustls-work-plan/">one of the items in Prossimo’s Rustls work plan</a> is an <strong>OpenSSL API Compatibility Layer</strong>; I’ve been told that <a href="https://github.com/ymjing/tabbyssl">TabbySSL</a> is such a compatibility layer, but that it isn’t under active development.</p>
<p>The <a href="https://i.have.insufficient.coffee/2023-03-30-oscw-rustls-slides.pdf">slides for this talk</a> are also available.</p>As I mentioned in my post about attending Real World Crypto 2023 and the Open Source Cryptography Workshop, I’ve given a talk discussing Rustls-FFI and the work to allow curl and libcurl to use the Rust-based, memory-safe Rustls TLS library in a talk called Make It Memory Safe: Adapting Curl to use Rustls.Nevermind about RWC and OSCW: COVID instead2023-03-25T00:00:00-07:002023-03-25T00:00:00-07:00https://insufficient.coffee/2023/03/25/nevermind-about-rwc-and-oscw<p>At this point I’m supposed to be in Tokyo, attending the <a href="https://rwc.iacr.org/2023/">Real World Crypto Symposium in Tokyo</a> next week, and after that, I’m co-organizing and <a href="/speaking-engagements/">speaking at</a> the <a href="https://opensourcecryptowork.shop/2023/">Open Source Cryptography Workshop</a>. But I’ve gotten COVID-19 again, instead.<!-- more --></p>
<p>Long story short, my household got a positive PCR test result just a couple of hours before I was to get on a plane to Tokyo, and while I wasn’t sick yet, I’d been significantly exposed.</p>
<p>Since then I’ve gotten sick. Not as sick as I was last August (yet), but clearly glad to have not gotten on the plane, even if I’m just crushed that I’m once again not at RWC, let alone not going to be at the workshop I’ve helped organize.</p>
<p>I got some time yesterday to pre-record the talk about
<a href="https://github.com/rustls/rustls-ffi/">Rustls-FFI</a> and the work to allow <code class="language-plaintext highlighter-rouge">curl</code> and <code class="language-plaintext highlighter-rouge">libcurl</code> to use <a href="https://github.com/rustls/rustls">Rustls</a> for TLS. Watching through the couple of recordings I made, to me it’s painfully obvious how I’m struggling to avoid coughing while talking. I am my own worst critic, though.</p>
<p>Maybe I’ll feel well enough to give it another try before the workshop.</p>
<p>Since I’m not there to tell folk in person, let me make sure the record shows that Thyla van der Merwe is absolutely the star organizer for the workshop: She had the idea and pulled together the funding and the venue. I helped out where I could, and now I’m not going to be there to help on the ground.</p>
<p>Blasted virus.</p>At this point I’m supposed to be in Tokyo, attending the Real World Crypto Symposium in Tokyo next week, and after that, I’m co-organizing and speaking at the Open Source Cryptography Workshop. But I’ve gotten COVID-19 again, instead.Attending Real World Crypto and the Open Source Cryptography Workshop 20232023-03-21T00:00:00-07:002023-03-21T00:00:00-07:00https://insufficient.coffee/2023/03/21/rwc-and-opensource-crypto-workshop-2023<p>I’ll be attending the <a href="https://rwc.iacr.org/2023/">Real World Crypto Symposium in Tokyo</a> next week, and after that, I’m co-organizing and <a href="/speaking-engagements/">speaking at</a> the <a href="https://opensourcecryptowork.shop/2023/">Open Source Cryptography Workshop</a>.<!-- more --></p>
<p>I’ll be discussing <a href="https://github.com/rustls/rustls-ffi/">Rustls-FFI</a> and the work to allow <code class="language-plaintext highlighter-rouge">curl</code> and <code class="language-plaintext highlighter-rouge">libcurl</code> to use the Rust-based, memory-safe <a href="https://github.com/rustls/rustls">Rustls TLS library</a> in a talk called <em>Adapting Curl to use Rustls</em>. Pulling Rustls into Curl was a <a href="https://www.memorysafety.org/initiative/rustls/">Prossimo initiative</a> at <a href="https://www.abetterinternet.org/about/">my day job</a>.</p>
<figure>
<a href="/content/2023/03/oscw2023.png">
<picture>
<source type="image/webp" srcset="/content/2023/03/oscw2023.1024.webp" />
<img src="/content/2023/03/oscw2023.1024.png" />
</picture>
</a>
</figure>
<p>I have links to my prior talks <a href="/speaking-engagements/">under About: Speaking Engagements</a>.</p>I’ll be attending the Real World Crypto Symposium in Tokyo next week, and after that, I’m co-organizing and speaking at the Open Source Cryptography Workshop.Moving Human Blood Around Arizona2021-08-06T00:00:00-07:002021-08-06T00:00:00-07:00https://insufficient.coffee/2021/08/06/moving-blood<p>I’d never thought about the logistics of moving donated blood around. The actual donation part is known to be a universally good thing to do, but I’d never searched out what happens next until I heard a fellow pilot talk about volunteering to move platelets around Arizona.<!-- more --></p>
<p>Arizona is a fairly mountainous state, in addition to simply being large. It’s at best a three hour drive from Phoenix to Yuma (each way) and closer to four hours to points northeast in the mountains, like St. Johns or Holbrook. Yet folks everywhere <a href="https://en.wikipedia.org/wiki/Blood_donation">donate blood</a>, and folks everywhere need <a href="https://en.wikipedia.org/wiki/Plateletpheresis">platelets</a>.</p>
<p>It turns out that freshness for blood matters quite a bit for processing. Since a lot of <em>mobile</em> blood drives collect <a href="https://en.wikipedia.org/wiki/Whole_blood">whole blood</a>, the longer a donation waits, the more the platelets inside clot, and the fewer can be processed out. Also, while certainly one wants the whole blood to drop to cold temperatures before and during transport, there’s a balancing act of <a href="https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1751-2824.2008.00196.x">avoiding freezing the blood prematurely</a>, as that, too, limits what can be processed out of it.</p>
<p>Clearly it’s advantageous to process the blood as soon as possible after a donation — however, in Arizona, all the processing happens centrally here in Phoenix.</p>
<p>Mountain towns here in Arizona donate blood, too, but with four hours of cross-country driving added to the processing time — sometimes in the desert heat — the obvious answer is to simply freeze their donations and accept the limitations that sets on the usefulness of the blood.</p>
<p>However, some <a href="http://www.flightsforlife.org/about_us.php">pilots in 1984 partnered with an Arizona blood bank</a> to provide rapid air transportation for these communities’ blood donations. Using aircraft — over the mountains, especially — changes the equation dramatically. St. Johns, AZ to Deer Valley, AZ is <a href="https://skyvector.com/?ll=34.10437576207119,-110.73065185119395&chart=301&zoom=4&fpl=N0135%20KSJN%20WEKUM%20V190%20GRINE%20KDVT">a 65 minute flight</a>, versus a more-than <a href="https://www.google.com/maps/dir/St+Johns,+AZ/Phoenix,+Arizona/">4 hour drive</a>.</p>
<p>And so I realized that this idea of pilots volunteering to move blood (and platelets) around the state wasn’t only an excuse to write off some flying time on their taxes.</p>
<p>I signed up.</p>
<h2 id="my-first-missions">My First Missions</h2>
<p>The Flights for Life organization had very thorough online (textual) training, combined with <a href="https://www.aopa.org/lms/courses/public-benefit-flying/">AOPA’s Public Benefit Flying course</a>, but there’s no mentorship for that first mission, just the rendezvous times and locations, a few phone numbers, and your experience as a pilot to make the right choices.</p>
<p>Late in the afternoon on a Wednesday in October 2020 I fueled up my Skylane and flew to Springerville, AZ, <a href="https://skyvector.com/?ll=33.70027411783179,-110.56091308165199&chart=301&zoom=4&fpl=N0135%20KJTC%20WEKUM%20V190%20GRINE%20KCHD">just about a 1 hour flight</a>. I arrived as the shadows got long, and within 2 minutes of opening the door of the plane, the blood drive’s van pulled into view.</p>
<p>The crew for the blood drive, 7 people in scrubs, wheeled a cart with nine boxes straight across the ramp to me. The mission was to transport seven boxes, but the blood drive was more successful than anticipated. I’d done the math that I could easily transport more than six hundred pounds of cargo with the fuel load I had, but the problem wasn’t the weight.</p>
<figure>
<a href="/content/2021/08/planeside.jpg">
<picture>
<source type="image/webp" srcset="/content/2021/08/planeside.1024.webp" />
<img src="/content/2021/08/planeside.1024.png" alt="First Load of Blood" />
</picture>
</a>
<figcaption>First Load of Blood</figcaption>
</figure>
<p>The blood drive crew watched, curiously, as I played a form of Tetris to figure out how to finangle so many boxes into the aircraft. To their eyes, I got it figured out — there were no more boxes on their cart. We wished each other safe travels, and they left. I got out my weight-and-balance calculator and made sure the configuration of boxes I came up with was within a safe one.</p>
<figure>
<a href="/content/2021/08/cargo_hold.jpg">
<picture>
<source type="image/webp" srcset="/content/2021/08/cargo_hold.1024.webp" />
<img src="/content/2021/08/cargo_hold.1024.png" alt="Packing the cargo hold. Tetris FTW!" />
</picture>
</a>
<figcaption>Packing the cargo hold. Tetris FTW!</figcaption>
</figure>
<p>It was. Cessna Skylanes are like sky-trucks.</p>
<figure>
<a href="/content/2021/08/back_seat.jpg">
<picture>
<source type="image/webp" srcset="/content/2021/08/back_seat.1024.webp" />
<img src="/content/2021/08/back_seat.1024.png" alt="Can definitely fit a few on the back seat and the footwell, too." />
</picture>
</a>
<figcaption>Can definitely fit a few on the back seat and the footwell, too.</figcaption>
</figure>
<p>The trip back to Phoenix was a quiet flight into the sunset, timed quite well for the fun times over the mountains to still be well-lit, but give the full effect of the city lights at night while coming home.</p>
<p>On this first mission, I took a selfie with the driver that the blood processing center sent to meet me. He was happy to do it, and answer/confirm some of my questions about the blood processing, as he was actually starting his shift and would be personally processing all the blood that evening.</p>
<figure>
<a href="/content/2021/08/first_mission_delivery.jpg">
<picture>
<source type="image/webp" srcset="/content/2021/08/first_mission_delivery.1024.webp" />
<img src="/content/2021/08/first_mission_delivery.1024.png" alt="Successful first delivery. I'm on the left." />
</picture>
</a>
<figcaption>Successful first delivery. I'm on the left.</figcaption>
</figure>
<p>A few weeks later, I did it again. This time I flew to Show Low, AZ: a 50-minute flight that can easily be a four hour drive if there’s traffic along the small roads outside of Payson, AZ. Which there always is.</p>
<figure>
<a href="/content/2021/08/show_low_airport.jpg">
<picture>
<source type="image/webp" srcset="/content/2021/08/show_low_airport.1024.webp" />
<img src="/content/2021/08/show_low_airport.1024.png" alt="Show Low Airport" />
</picture>
</a>
<figcaption>Show Low Airport, on final approach.</figcaption>
</figure>
<p>This was a mission that called for two aircraft. I arrived a bit early and refueled. And waited. Neither the second aircraft nor the blood drive crew were on time.</p>
<p>Eventually the blood drive crew arrived with sixteen boxes of blood to transport, and still no second aircraft. The leader of the blood drive and I both made phone calls, and we found out: second aircraft never left, it bowed out for a mechanical issue, but the texts never made it through to let us know.</p>
<p>Okay, so now for extra fun: How many of the sixteen boxes could I take and be safe? The folk for the blood drive encouraged me to take all the time I needed to figure it out. There was no way to get the blood to Phoenix that night except me — any blood I didn’t take would have to go out sometime the following day, the crew leader explained, most likely the following night, due to the way agreements with the trucking companies work.</p>
<p>Keeping in mind I had just refueled, it still turns out that weight-and-balance wasn’t the limiting factor, it was physical dimensions. I fit eleven boxes of blood (330 lbs), and two boxes of paperwork (a few pounds), and refused five more boxes because I couldn’t find places to fit them that would be safe. (If I had straps to hold boxes down, I could have put two more in the front passenger seat, but I didn’t want to risk a tip-over!)</p>
<figure>
<a href="/content/2021/08/load_of_blood.jpg">
<picture>
<source type="image/webp" srcset="/content/2021/08/load_of_blood.1024.webp" />
<img src="/content/2021/08/load_of_blood.1024.png" alt="I guess I could have put more in the passenger seat" />
</picture>
</a>
<figcaption>I guess I could have put more in the passenger seat</figcaption>
</figure>
<p>I got a lot of thanks from that blood drive crew; they waved at me as I taxied to depart. When I met the driver in Phoenix an hour later, he likened my plane to “a clown car” because boxes kept coming out.</p>
<figure>
<a href="/content/2021/08/blood_delivered.jpg">
<picture>
<source type="image/webp" srcset="/content/2021/08/blood_delivered.1024.webp" />
<img src="/content/2021/08/blood_delivered.1024.png" alt="All that blood, delivered" />
</picture>
</a>
<figcaption>All that blood, delivered</figcaption>
</figure>
<p>I’ve had more flights since then. Sometimes everything is right-on-time, often it’s a little bit off, but no matter what, the blood gets where it’s going much faster than it otherwise could, and the donations these towns make — which often far outstrip the anticipated volume — are utilized to their fullest potential.</p>
<h2 id="about-platelets">About Platelets</h2>
<p>I have yet to fly platelets anywhere, but the gist is that the shelf life of platelets is very short, except for trauma care where refrigeration is permitted. After platelets are processed in Phoenix, every hour before they’re transfused counts, so Flights for Life moves boxes of platelets to outlying hospitals about weekly, though demand varies.</p>
<p>The platelet missions are uniformly first-thing-in-the-morning departures from Phoenix with the cargo, and right now first-thing-in-the-morning isn’t a place of honor in my life. Maybe later.</p>
<h2 id="commercial">Commercial?</h2>
<p>It might seem like flying blood around would be a commercial pilot activity — and if I were compensated for it, it would be. But here in the US at least, it’s considered to still be private pilot activity to donate my time and aircraft this way.</p>
<h2 id="public-benefit-flying">Public Benefit Flying</h2>
<p>I’ve been part of <a href="https://www.angelflightwest.org/about-us/">Angel Flight West</a> for a number of years as a <a href="http://www.angelflightwest.org/earth-angels/">driver</a> (think Lyft/Uber, but just for Angel Flight passengers), but I hadn’t <em>flown</em> any charitable missions before. Specifically for Angel Flight, I’ve been watching my logbook climb to the number of pilot-in-command (PIC) hours needed to apply to fly.</p>
<figure>
<a href="/content/2021/08/night_delivery.jpg">
<picture>
<source type="image/webp" srcset="/content/2021/08/night_delivery.1024.webp" />
<img src="/content/2021/08/night_delivery.1024.png" alt="A late night delivery from upstate" />
</picture>
</a>
<figcaption>A late night delivery from upstate</figcaption>
</figure>
<p>Moving blood around turned out to be a great way to accumulate more PIC time. I’m happy to say that I’m now a full-fledged Angel Flight West Command Pilot too, but that’s for a different post.</p>
<h2 id="a-final-note">A Final Note</h2>
<p>Everyone wants to help when you’re pulling out boxes labeled <strong>HUMAN BLOOD</strong>, whether they’re other pilots, line technicians, or anyone else on the ramp. I guess it’s just admirable enough, without the potential awkwardness of the passengers themselves on Angel Flights, where I swear everyone goes out of their way to pretend there’s nothing amazingly cool about those. But they are super cool.</p>
<figure>
<a href="/content/2021/08/boxes_on_cart.jpg">
<picture>
<source type="image/webp" srcset="/content/2021/08/boxes_on_cart.1024.webp" />
<img src="/content/2021/08/boxes_on_cart.1024.png" alt="Anyone looking for some HUMAN BLOOD?" />
</picture>
</a>
<figcaption>Anyone looking for some HUMAN BLOOD?</figcaption>
</figure>
<h3 id="links">Links</h3>
<ul>
<li>
<p><a href="http://www.flightsforlife.org/about_us.php">Flights for Life</a> is the Arizona organization that flies blood and blood products around the desert mountains. <a href="http://www.flightsforlife.org">http://www.flightsforlife.org</a></p>
</li>
<li>
<p><a href="https://www.angelflightwest.org/about-us/">Angel Flight West</a> is the west-coast chapter of pilots that fly non-emergency medical relief flights for in-need passengers. <a href="https://www.angelflightwest.org">https://www.angelflightwest.org</a></p>
</li>
</ul>I’d never thought about the logistics of moving donated blood around. The actual donation part is known to be a universally good thing to do, but I’d never searched out what happens next until I heard a fellow pilot talk about volunteering to move platelets around Arizona.Design of the CRLite Infrastructure2020-12-01T00:00:00-07:002020-12-01T00:00:00-07:00https://insufficient.coffee/2020/12/01/crlite-part-4-infrastructure-design<p>Firefox is the only major browser that still evaluates every website it connects to whether the certificate used has been reported as revoked. Firefox users are notified of all connections involving untrustworthy certificates, regardless the popularity of the site. Inconveniently, checking certificate status sometimes slows down the connection to websites. Worse, the check reveals cleartext information about the website you’re visiting to network observers.</p>
<p>We’re now testing a technology named CRLite which provides Firefox users with the confidence that the revocations in the Web PKI are enforced by the browser without this privacy compromise. This is a part of our goal to use encryption everywhere. (See also: <a href="https://blog.mozilla.org/security/2018/10/18/encrypted-sni-comes-to-firefox-nightly/">Encrypted SNI</a> and <a href="https://blog.mozilla.org/futurereleases/2019/07/31/dns-over-https-doh-update-detecting-managed-networks-and-user-choice/">DNS-over-HTTPS</a>)</p>
<p>The first three posts in this series are about the newly-added CRLite technology and provide background that will be useful for following along with this post:</p>
<ul>
<li><a href="/2020/01/09/crlite-part-1-all-web-pki-revocations-compressed/">Introducing CRLite: All of the Web PKI’s revocations, compressed</a>,</li>
<li><a href="/2020/01/21/crlite-part-3-speeding-up-secure-browsing/">CRLite: Speeding Up Secure Browsing</a>, and specifically useful is</li>
<li><a href="/2020/01/09/crlite-part-2-end-to-end-design/">The End-to-End Design of CRLite</a>.</li>
</ul>
<p>This blog post discusses the back-end infrastructure that produces the data which Firefox uses for CRLite. To begin with, we’ll trace that data in reverse, starting from what Firefox needs to use for CRLite’s algorithms, back to the inputs derived from monitoring the whole Web PKI via Certificate Transparency.</p>
<h2 id="tracing-the-flow-of-data">Tracing the Flow of Data</h2>
<p>Individual copies of Firefox maintain in their profiles a CRLite database which is periodically updated via Firefox’s <a href="https://remote-settings.readthedocs.io">Remote Settings</a>. Those updates come in the form of CRLite filters and “stashes”.</p>
<h3 id="filters-and-stashes">Filters and Stashes</h3>
<p>The general mechanism for how the filters work is explained in Figure 3 of <a href="/2020/01/09/crlite-part-2-end-to-end-design/">The End-to-End Design of CRLite</a>.</p>
<p>Introduced in this post is the concept of CRLite <a href="https://github.com/mozilla/crlite/wiki#what-are-the-stashes">stashes</a>. These are lists of certificate issuers and the certificate serial numbers that those issuers revoked, which the CRLite infrastructure distributes to Firefox users in lieu of a whole new filter. If a certificate’s identity is contained within any of the issued stashes, then that certificate is invalid.</p>
<p>Combining stashes with the CRLite filters produces an algorithm which, in simplified terms, proceeds like this:</p>
<figure>
<a href="/content/2020/12/crlite-decision-flow.png">
<picture>
<source type="image/svg+xml" srcset="/content/2020/12/crlite-decision-flow.svg" />
<source type="image/webp" srcset="/content/2020/12/crlite-decision-flow.1024.webp" />
<img src="/content/2020/12/crlite-decision-flow.png" alt="A representation of the CRLite Algorithm: Is this website’s Certificate Authority enrolled in CRLite? If not, use the online status check, OCSP. If it is enrolled, should I expect this website’s certificate to be a part of the CRLite filter available in my local profile? If it should be in the CRLite filter, does this website’s certificate appear in the filter as having been revoked by its issuer? If it’s not in the filter as having been revoked, does this website’s certificate appear in any of my local profile’s CRLite stashes as being revoked? If either the CRLite filter or the stashes indicate the website’s certificate is revoked, don’t trust it and show an error page. If it’s not in the CRLite filter nor in the CRLite stashes as revoked, then proceed to run the rest of the Web PKI validity and trust checks. If for any reason we can’t tell from the above, for example the local filters or stashes are too old or we encounter an error, then we go back to using the online status check, OCSP." />
</picture>
</a>
<figcaption>Figure 1: Simplified CRLite Decision Tree</figcaption>
</figure>
<p>Every time the CRLite infrastructure updates its dataset, it produces both a new filter and a stash containing all of the new revocations (compared with the previous run). Firefox’s CRLite is up-to-date if it has <a href="https://github.com/mozilla/crlite/wiki#what-determines-whether-a-new-filter-gets-distributed-or-a-new-stash-distributed">a filter and all issued stashes for that filter</a>.</p>
<h3 id="enrolled-valid-and-revoked">Enrolled, Valid and Revoked</h3>
<p>To produce the filters and stashes, CRLite needs as input:</p>
<ol>
<li>The list of trusted certificate authority issuers which are <a href="https://github.com/mozilla/crlite/wiki#how-do-you-pick-what-cas-are-included-in-crlite">enrolled in CRLite</a>,</li>
<li>The list of all currently-valid certificates issued by each of those enrolled certificate authorities, e.g. information from Certificate Transparency,</li>
<li>The list of all unexpired-but-revoked certificates issued by each of those enrolled certificate authorities, e.g. from Certificate Revocation Lists.</li>
</ol>
<p>These bits of data are the basis of the CRLite decision-making.</p>
<p>The enrolled issuers are communicated to Firefox clients as updates within the existing <a href="https://blog.mozilla.org/security/content/2020/11/13/preloading-intermediate-ca-certificates-into-firefox/">Intermediate Preloading</a> feature, while the certificate sets are compressed into the CRLite filters and stashes. Whether a certificate issuer is enrolled or not is directly related to obtaining the list of their revoked certificates.</p>
<h3 id="collecting-revocations">Collecting Revocations</h3>
<p>To obtain all the revoked certificates for a given issuer, the CRLite infrastructure reads the Certificate Revocation List (CRL) Distribution Point extension out of all that issuer’s unexpired certificates and filters the list down to those CRLs which are available over HTTP/HTTPS. Then, every URL in that list is downloaded and verified: Does it have a valid, trusted signature? Is it up-to-date? If any could not be downloaded, do we have a cached copy which is still both valid and up-to-date?</p>
<p>For issuers which are considered enrolled, all of the entries in the CRLs are collected and saved as a complete list of all revoked certificates for that issuer.</p>
<h3 id="lists-of-unexpired-certificates">Lists of Unexpired Certificates</h3>
<p>The lists of currently-valid certificates and unexpired-but-revoked certificates have to be calculated, as the data sources that CRLite uses consist of:</p>
<ol>
<li>Certificate Transparency’s list of all certificates in the WebPKI, and</li>
<li>All the published certificate revocations from the previous step.</li>
</ol>
<p>By policy now, <a href="https://www.certificate-transparency.org/">Certificate Transparency (CT) Logs</a>, in aggregate, are assumed to provide a complete list of all certificates in the public Web PKI. CRLite then filters the complete CT dataset down to certificates which haven’t yet reached their expiration date, but which have been issued by certificate authorities trusted by Firefox.</p>
<p>Filtering CT data down to a list of unexpired certificates allows CRLite to derive the needed data sets using set math:</p>
<ul>
<li>The currently-valid certificates are those which are unexpired and not included in any revocation list,</li>
<li>The unexpired-but-revoked certificates are those which are unexpired and are included in a revocation list.</li>
</ul>
<p>The CT data simply comes from a continual monitoring of the Certificate Transparency ecosystem. <a href="https://github.com/mozilla/crlite/wiki#what-ct-logs-are-monitored">Every known CT log</a> is monitored by Mozilla’s infrastructure, and every certificate added to the ecosystem is processed.</p>
<h2 id="the-kubernetes-pods">The Kubernetes Pods</h2>
<p>All these functions are orchestrated as four <a href="https://en.wikipedia.org/wiki/Kubernetes">Kubernetes</a> pods with the descriptive names Fetch, Generate, Publish, and Sign-off.</p>
<h3 id="fetch">Fetch</h3>
<p>Fetch is a Kubernetes deployment, or always-on task, which constantly monitors Certificate Transparency data from all Certificate Transparency logs. Certificates that aren’t expired are inserted into a <a href="http://redis.io/">Redis database</a>, configured so that certificates are expunged automatically when they reach their expiration time. This way, whenever the CRLite infrastructure requires a list of all unexpired certificates known to Certificate Transparency, it can iterate through all of the certificates in the Redis database. The actual data stored in Redis is <a href="https://github.com/mozilla/crlite/wiki#what-gets-stored-in-redis">described in our FAQ</a>.</p>
<figure>
<a href="/content/2020/12/crlite-fetch.png">
<picture>
<source type="image/svg+xml" srcset="/content/2020/12/crlite-fetch.svg" />
<source type="image/webp" srcset="/content/2020/12/crlite-fetch.1024.webp" />
<img src="/content/2020/12/crlite-fetch.png" alt="Figure 2: The Fetch task reads
from Certificate Transparency and stores data in a Redis database" />
</picture>
</a>
<figcaption>Figure 2: The Fetch task reads from Certificate Transparency and
stores data in a Redis database</figcaption>
</figure>
<h3 id="generate">Generate</h3>
<p>The Generate pod is a periodic task, which currently runs four times a day. This task reads all known unexpired certificates from the Redis database, downloads and validates all CRLs from the issuing certificate authorities, and synthesizes a filter and a stash from those data sources. The resulting filters and stashes are uploaded into a <a href="https://github.com/mozilla/crlite/wiki#where-can-i-get-the-crlite-data-that-is-used-to-make-filters">Google Cloud Storage bucket, along with all the source input data</a>, for both public audit and distribution.</p>
<figure>
<a href="/content/2020/12/crlite-generate.png">
<picture>
<source type="image/svg+xml" srcset="/content/2020/12/crlite-generate.svg" />
<source type="image/webp" srcset="/content/2020/12/crlite-generate.1024.webp" />
<img src="/content/2020/12/crlite-generate.png" alt="Figure 3: The Generate task
reads from a Redis database and the Internet, and writes its results to Google
Cloud Storage" />
</picture>
</a>
<figcaption>Figure 3: The Generate task reads from a Redis database and the
Internet, and writes its results to Google Cloud Storage</figcaption>
</figure>
<h3 id="publish">Publish</h3>
<p>The Publish task is also a periodic task, running often. It looks for new filters and stashes in the Google Cloud Storage bucket, and stages <a href="https://github.com/mozilla/crlite/wiki#what-determines-whether-a-new-filter-gets-distributed-or-a-new-stash-distributed">either a new filter or a stash</a> to Firefox’s Remote Settings when the Generate task finishes producing one.</p>
<figure>
<a href="/content/2020/12/crlite-publish.png">
<picture>
<source type="image/svg+xml" srcset="/content/2020/12/crlite-publish.svg" />
<source type="image/webp" srcset="/content/2020/12/crlite-publish.1024.webp" />
<img src="/content/2020/12/crlite-publish.png" alt="Figure 4: The Publish job
reads from Google Cloud Storage and writes to Remote Settings" />
</picture>
</a>
<figcaption>Figure 4: The Publish job reads from Google Cloud Storage and
writes to Remote Settings</figcaption>
</figure>
<h3 id="sign-off">Sign-Off</h3>
<p>Finally, a separate Sign-Off task runs periodically, also often. When there is an updated filter or stash staged at Firefox’s Remote Settings, the sign-off task downloads the staged data and tests it, looking for coherency and to make sure that CRLite does not accidentally include revocations that could break Firefox. If all the tests pass, the Sign-Off task approves the new CRLite data for distribution, which triggers <a href="https://mozilla-push-service.readthedocs.io/en/latest/">Megaphone</a> to push the update to Firefox users that are online.</p>
<figure>
<a href="/content/2020/12/crlite-signoff.png">
<picture>
<source type="image/svg+xml" srcset="/content/2020/12/crlite-signoff.svg" />
<source type="image/webp" srcset="/content/2020/12/crlite-signoff.1024.webp" />
<img src="/content/2020/12/crlite-signoff.png" alt="Figure 5: The Sign-Off task
interacts with both Remote Settings and the public Internet" />
</picture>
</a>
<figcaption>Figure 5: The Sign-Off task interacts with both Remote Settings
and the public Internet</figcaption>
</figure>
<h2 id="using-crlite">Using CRLite</h2>
<p>We recently announced in the <a href="https://groups.google.com/g/mozilla.dev.platform/c/mbYKUEGE5oQ">mozilla.dev.platform mailing list that Firefox Nightly users on Desktop are relying on CRLite</a>, after collecting encouraging performance measurements for most of 2020. We’re working on plans to begin tests for Firefox Beta users soon. If you want to try using CRLite, you can use Firefox Nightly, or for the more adventurous reader, <a href="https://github.com/mozilla/crlite/wiki#how-can-i-query-my-crlite-filter">interact with the CRLite data directly</a>.</p>
<p>Our final blog post in this series, Part 5, will reflect on the collaboration between Mozilla Security Engineering and the several research teams that designed and have analyzed CRLite to produce this impressive system.</p>Firefox is the only major browser that still evaluates every website it connects to whether the certificate used has been reported as revoked. Firefox users are notified of all connections involving untrustworthy certificates, regardless the popularity of the site. Inconveniently, checking certificate status sometimes slows down the connection to websites. Worse, the check reveals cleartext information about the website you’re visiting to network observers. We’re now testing a technology named CRLite which provides Firefox users with the confidence that the revocations in the Web PKI are enforced by the browser without this privacy compromise. This is a part of our goal to use encryption everywhere. (See also: Encrypted SNI and DNS-over-HTTPS) The first three posts in this series are about the newly-added CRLite technology and provide background that will be useful for following along with this post: Introducing CRLite: All of the Web PKI’s revocations, compressed, CRLite: Speeding Up Secure Browsing, and specifically useful is The End-to-End Design of CRLite. This blog post discusses the back-end infrastructure that produces the data which Firefox uses for CRLite. To begin with, we’ll trace that data in reverse, starting from what Firefox needs to use for CRLite’s algorithms, back to the inputs derived from monitoring the whole Web PKI via Certificate Transparency. Tracing the Flow of Data Individual copies of Firefox maintain in their profiles a CRLite database which is periodically updated via Firefox’s Remote Settings. Those updates come in the form of CRLite filters and “stashes”. Filters and Stashes The general mechanism for how the filters work is explained in Figure 3 of The End-to-End Design of CRLite. Introduced in this post is the concept of CRLite stashes. These are lists of certificate issuers and the certificate serial numbers that those issuers revoked, which the CRLite infrastructure distributes to Firefox users in lieu of a whole new filter. If a certificate’s identity is contained within any of the issued stashes, then that certificate is invalid. Combining stashes with the CRLite filters produces an algorithm which, in simplified terms, proceeds like this: Figure 1: Simplified CRLite Decision Tree Every time the CRLite infrastructure updates its dataset, it produces both a new filter and a stash containing all of the new revocations (compared with the previous run). Firefox’s CRLite is up-to-date if it has a filter and all issued stashes for that filter. Enrolled, Valid and Revoked To produce the filters and stashes, CRLite needs as input: The list of trusted certificate authority issuers which are enrolled in CRLite, The list of all currently-valid certificates issued by each of those enrolled certificate authorities, e.g. information from Certificate Transparency, The list of all unexpired-but-revoked certificates issued by each of those enrolled certificate authorities, e.g. from Certificate Revocation Lists. These bits of data are the basis of the CRLite decision-making. The enrolled issuers are communicated to Firefox clients as updates within the existing Intermediate Preloading feature, while the certificate sets are compressed into the CRLite filters and stashes. Whether a certificate issuer is enrolled or not is directly related to obtaining the list of their revoked certificates. Collecting Revocations To obtain all the revoked certificates for a given issuer, the CRLite infrastructure reads the Certificate Revocation List (CRL) Distribution Point extension out of all that issuer’s unexpired certificates and filters the list down to those CRLs which are available over HTTP/HTTPS. Then, every URL in that list is downloaded and verified: Does it have a valid, trusted signature? Is it up-to-date? If any could not be downloaded, do we have a cached copy which is still both valid and up-to-date? For issuers which are considered enrolled, all of the entries in the CRLs are collected and saved as a complete list of all revoked certificates for that issuer. Lists of Unexpired Certificates The lists of currently-valid certificates and unexpired-but-revoked certificates have to be calculated, as the data sources that CRLite uses consist of: Certificate Transparency’s list of all certificates in the WebPKI, and All the published certificate revocations from the previous step. By policy now, Certificate Transparency (CT) Logs, in aggregate, are assumed to provide a complete list of all certificates in the public Web PKI. CRLite then filters the complete CT dataset down to certificates which haven’t yet reached their expiration date, but which have been issued by certificate authorities trusted by Firefox. Filtering CT data down to a list of unexpired certificates allows CRLite to derive the needed data sets using set math: The currently-valid certificates are those which are unexpired and not included in any revocation list, The unexpired-but-revoked certificates are those which are unexpired and are included in a revocation list. The CT data simply comes from a continual monitoring of the Certificate Transparency ecosystem. Every known CT log is monitored by Mozilla’s infrastructure, and every certificate added to the ecosystem is processed. The Kubernetes Pods All these functions are orchestrated as four Kubernetes pods with the descriptive names Fetch, Generate, Publish, and Sign-off. Fetch Fetch is a Kubernetes deployment, or always-on task, which constantly monitors Certificate Transparency data from all Certificate Transparency logs. Certificates that aren’t expired are inserted into a Redis database, configured so that certificates are expunged automatically when they reach their expiration time. This way, whenever the CRLite infrastructure requires a list of all unexpired certificates known to Certificate Transparency, it can iterate through all of the certificates in the Redis database. The actual data stored in Redis is described in our FAQ. Figure 2: The Fetch task reads from Certificate Transparency and stores data in a Redis database Generate The Generate pod is a periodic task, which currently runs four times a day. This task reads all known unexpired certificates from the Redis database, downloads and validates all CRLs from the issuing certificate authorities, and synthesizes a filter and a stash from those data sources. The resulting filters and stashes are uploaded into a Google Cloud Storage bucket, along with all the source input data, for both public audit and distribution. Figure 3: The Generate task reads from a Redis database and the Internet, and writes its results to Google Cloud Storage Publish The Publish task is also a periodic task, running often. It looks for new filters and stashes in the Google Cloud Storage bucket, and stages either a new filter or a stash to Firefox’s Remote Settings when the Generate task finishes producing one. Figure 4: The Publish job reads from Google Cloud Storage and writes to Remote Settings Sign-Off Finally, a separate Sign-Off task runs periodically, also often. When there is an updated filter or stash staged at Firefox’s Remote Settings, the sign-off task downloads the staged data and tests it, looking for coherency and to make sure that CRLite does not accidentally include revocations that could break Firefox. If all the tests pass, the Sign-Off task approves the new CRLite data for distribution, which triggers Megaphone to push the update to Firefox users that are online. Figure 5: The Sign-Off task interacts with both Remote Settings and the public Internet Using CRLite We recently announced in the mozilla.dev.platform mailing list that Firefox Nightly users on Desktop are relying on CRLite, after collecting encouraging performance measurements for most of 2020. We’re working on plans to begin tests for Firefox Beta users soon. If you want to try using CRLite, you can use Firefox Nightly, or for the more adventurous reader, interact with the CRLite data directly. Our final blog post in this series, Part 5, will reflect on the collaboration between Mozilla Security Engineering and the several research teams that designed and have analyzed CRLite to produce this impressive system.Auditing the CRLs in CRLite2020-11-27T00:00:00-07:002020-11-27T00:00:00-07:00https://insufficient.coffee/2020/11/27/auditing-crls-of-crlite<p>Since Firefox Nightly is now using <a href="/tag/crlite">CRLite</a> to determine if enrolled websites’ certificates are revoked, it’s useful to dig into the data to answer why a given certificate issuer gets enrolled or not.</p>
<p>Ultimately this is a matter of whether the CRLs for a given issuer are available to CRLite, and are valid, but the Internet is a messy place, and sometimes things don’t work as planned. If an issuing CA is not enrolled in CRLite, the Mozilla infrastructure emits enough information to figure out what went wrong.</p>
<h2 id="digging-into-crlites-data-sets">Digging into CRLite’s Data Sets</h2>
<p>Mozilla’s infrastructure publishes not only the filters and stashes that define the <a href="/tag/crlite">CRLite</a> state-of-the-WebPKI, it also publishes all the intermediate datasets that can be used to both reproduce the filters, and validate them.</p>
<p>I have a tool, <a href="https://github.com/jcjones/crlite-status">crlite-status</a>, which barely sticks a toe into the water of these datasets.</p>
<p><code class="language-plaintext highlighter-rouge">crlite-status</code> is distributed as a Python package that can be installed via a simple invocation of <code class="language-plaintext highlighter-rouge">pip</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip install crlite-status
</code></pre></div></div>
<p>It downloads data directly from the Google Cloud Storage buckets for the production and staging environments, and parses out some of the useful statistics. For a simple example, asking for the last four runs out of the default production environment:</p>
<figure>
<a href="/content/2020/11/crlite-status.png">
<picture>
<source type="image/webp" srcset="/content/2020/11/crlite-status.1024.webp" />
<img src="/content/2020/11/crlite-status.png" alt="Four recent CRLite runs" />
</picture>
</a>
<figcaption>Four recent CRLite runs</figcaption>
</figure>
<p>It gets more exciting when enabling CRL auditing, as CRLs dermine whether an issuer is “enrolled” in CRLite or not:</p>
<figure>
<a href="/content/2020/11/crlite-status-crls.png">
<picture>
<source type="image/webp" srcset="/content/2020/11/crlite-status-crls.1024.webp" />
<img src="/content/2020/11/crlite-status-crls.png" alt="crlite-status showing not-enrolled issuers" />
</picture>
</a>
<figcaption>crlite-status showing not-enrolled issuers</figcaption>
</figure>
<p>For just the most recent run, we can see which issuers were excluded from CRLite due to CRL issues, marked with a red X. There are actually six of them.</p>
<p>Even more details to trace this information is available with the <code class="language-plaintext highlighter-rouge">--crl-details <path></code> command line option. I’ve posted <a href="/content/2020/11/crlite-status-crl-details.html">one output from <code class="language-plaintext highlighter-rouge">20201126-0</code> as an example</a>. Using that <code class="language-plaintext highlighter-rouge">20201126-0</code>, one can see that all three CertCloud issuers are not enrolled as their CRLs are returning a status <code class="language-plaintext highlighter-rouge">403 Forbidden</code>, while the Actalis URL is giving a <code class="language-plaintext highlighter-rouge">404 Not Found</code>. TeleSec’s CRL is hosted at a domain that is down entirely. Starfield and Secom both had recoverable warnings, though the Starfield root failed to recover and details would have to be harvested from the <code class="language-plaintext highlighter-rouge">crls</code> and <code class="language-plaintext highlighter-rouge">logs</code> directory, while the Secom issuer remained enrolled on the basis of the already-cached CRL still being valid.</p>
<h3 id="more-to-do">More to do</h3>
<p>Lots of the CRLite CRL auditing is manual, still. Since the whole state of the generation task gets uploaded to the Google Cloud Filestore, it’s always possible to determine what state existed for CRLite at the time it made its enrollment decisions, certainly better tools would make it a simpler process.</p>Since Firefox Nightly is now using CRLite to determine if enrolled websites’ certificates are revoked, it’s useful to dig into the data to answer why a given certificate issuer gets enrolled or not. Ultimately this is a matter of whether the CRLs for a given issuer are available to CRLite, and are valid, but the Internet is a messy place, and sometimes things don’t work as planned. If an issuing CA is not enrolled in CRLite, the Mozilla infrastructure emits enough information to figure out what went wrong. Digging into CRLite’s Data Sets Mozilla’s infrastructure publishes not only the filters and stashes that define the CRLite state-of-the-WebPKI, it also publishes all the intermediate datasets that can be used to both reproduce the filters, and validate them. I have a tool, crlite-status, which barely sticks a toe into the water of these datasets. crlite-status is distributed as a Python package that can be installed via a simple invocation of pip: pip install crlite-status It downloads data directly from the Google Cloud Storage buckets for the production and staging environments, and parses out some of the useful statistics. For a simple example, asking for the last four runs out of the default production environment: Four recent CRLite runs It gets more exciting when enabling CRL auditing, as CRLs dermine whether an issuer is “enrolled” in CRLite or not: crlite-status showing not-enrolled issuers For just the most recent run, we can see which issuers were excluded from CRLite due to CRL issues, marked with a red X. There are actually six of them. Even more details to trace this information is available with the --crl-details <path> command line option. I’ve posted one output from 20201126-0 as an example. Using that 20201126-0, one can see that all three CertCloud issuers are not enrolled as their CRLs are returning a status 403 Forbidden, while the Actalis URL is giving a 404 Not Found. TeleSec’s CRL is hosted at a domain that is down entirely. Starfield and Secom both had recoverable warnings, though the Starfield root failed to recover and details would have to be harvested from the crls and logs directory, while the Secom issuer remained enrolled on the basis of the already-cached CRL still being valid. More to do Lots of the CRLite CRL auditing is manual, still. Since the whole state of the generation task gets uploaded to the Google Cloud Filestore, it’s always possible to determine what state existed for CRLite at the time it made its enrollment decisions, certainly better tools would make it a simpler process.Querying CRLite for WebPKI Revocations2020-11-26T00:00:00-07:002020-11-26T00:00:00-07:00https://insufficient.coffee/2020/11/26/querying-crlite<p>Firefox Nightly is now using <a href="/tag/crlite">CRLite</a> to determine if websites’ certificates are revoked — e.g., if the Certificate Authority published that web browsers shouldn’t trust that website certificate. Telemetry shows that querying the local CRLite dataset is much faster than making a network connection for <a href="https://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol">OCSP</a>, which makes intuitive sense. It also avoids sending the website’s certificate information in cleartext over the network to check the revocation status: solving one of the remaining cleartext browsing data leakages in Firefox.</p>
<p>Mozilla is currently publishing CRLite data to <a href="https://remote-settings.readthedocs.io">Remote Settings</a> four times per day, keeping a very fresh set of revocation information for the public Web. I’ve provided some direct details on how to get at that data from the <a href="https://github.com/mozilla/crlite/wiki">CRLite FAQ</a>, and I want to introduce one of my command-line tools I’ve used to analyze and play with the dataset: <a href="https://github.com/mozilla/moz_crlite_query/"><code class="language-plaintext highlighter-rouge">moz_crlite_query</code></a>. I’ll introduce <a href="https://github.com/jcjones/crlite_status"><code class="language-plaintext highlighter-rouge">crlite_status</code></a> in a later post.</p>
<h2 id="querying-crlite-status-at-the-cli">Querying CRLite Status at the CLI</h2>
<p><code class="language-plaintext highlighter-rouge">moz_crlite_query</code> is a Python tool that more-or-less implements the same algorithm Firefox does to check a website’s certificate against CRLite. It maintains a local database of the CRLite dataset in your <code class="language-plaintext highlighter-rouge">~/.crlite_db/</code> directory, which for me right now looks like:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> 0 Nov 25 09:24 .last_updated
5381898 Nov 25 09:24 2020-11-24T00:08:12+00:00Z-full
27363 Nov 25 09:24 2020-11-24T06:08:12+00:00Z-diff
59196 Nov 25 09:24 2020-11-24T12:08:14+00:00Z-diff
56121 Nov 25 09:24 2020-11-24T18:08:15+00:00Z-diff
204863 Nov 25 09:24 2020-11-25T00:08:23+00:00Z-diff
25826 Nov 25 09:24 2020-11-25T06:08:05+00:00Z-diff
66626 Nov 25 09:24 2020-11-25T12:08:22+00:00Z-diff
25826 Nov 25 09:24 2020-11-25T06:08:05+00:00Z-diff
66626 Nov 25 09:24 2020-11-25T12:08:22+00:00Z-diff
53783 Nov 25 22:19 2020-11-25T18:08:11+00:00Z-diff
190353 Nov 25 22:19 2020-11-26T00:08:11+00:00Z-diff
1118208 Nov 25 22:19 intermediates.sqlite
</code></pre></div></div>
<p>You can see here that the last full CRLite filter update happened at the midnight-UTC update on 24 November 2020, and since then there have been eight delta updates (e.g., “stashes”).</p>
<p><code class="language-plaintext highlighter-rouge">moz_crlite_query</code> is distributed as a Python package that can be installed via an invocation of <code class="language-plaintext highlighter-rouge">pip</code> (Note that this tool requires Python 3.7 or newer because of SQLite features.):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip install moz_crlite_query
</code></pre></div></div>
<p>On invocation, the tool will try and update to the latest CRLite data, unless it already updated in the last several hours.</p>
<p>You can query the CRLite status for either a certificate you already have, or the tool can connect to one or more websites and examine the certificates presented there.</p>
<h3 id="experimenting-with-the-query-tool">Experimenting with the Query Tool</h3>
<p>Let’s download a few certificates into <code class="language-plaintext highlighter-rouge">/tmp</code> from crt.sh: <a href="https://crt.sh/?q=77575263">77575263</a>, <a href="https://crt.sh/?q=1988442812">1988442812</a>, <a href="https://crt.sh/?q=1485147627">1485147627</a>, and <a href="https://crt.sh/?q=2680822568">2680822568</a>. These simply are representative – right now – of various states in the Web PKI.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>○ → for id in 77575263 1988442812 1485147627 2680822568; do
> curl --silent https://crt.sh/?d=${id} > /tmp/${id}.pem
> done
</code></pre></div></div>
<p>Now, run them all through CRLite:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>○ → moz_crlite_query /tmp/*.pem
INFO:crlite_query:CRLite Update: Syncing CRLite filters.
100% (9 of 9) |######################################################| Elapsed Time: 0:00:00 Time: 0:00:00
INFO:query_cli:Status: 2457 Intermediates, Current filter: 2020-11-24T00:08:12+00:00Z-full
with 27 layers and 43053008 bit-count, 8 stash files with 43022 stashed revocations,
up-to-date as of 2020-11-26 00:08:11+00:00 (5:11:15.467645 ago).
/tmp/1485147627.pem Issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US
Enrolled in CRLite: ❌
Result: ❔ Not Enrolled ❔
/tmp/1988442812.pem Issuer: CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US
Enrolled in CRLite: ✅
Revoked via CRLite filter: 2020-11-24T00:08:12+00:00Z-full
Result: ⛔️ Revoked ⛔️
/tmp/2680822568.pem Issuer: CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US
Enrolled in CRLite: ✅
Result: 👍 Valid 👍
/tmp/77575263.pem Issuer: CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US
Enrolled in CRLite: ✅
Result: ⏰ Expired ⏰
</code></pre></div></div>
<p>We can see <a href="https://crt.sh/?q=1485147627">1485147627</a> is issued by a CA not participating in CRLite: Let’s Encrypt doesn’t publish CRLs. <a href="https://crt.sh/?q=1988442812">1988442812</a> is for <code class="language-plaintext highlighter-rouge">revoked.badssl.com</code>, and is revoked. <a href="https://crt.sh/?q=2680822568">2680822568</a> is currently a certificate used for many Mozilla properties, like <code class="language-plaintext highlighter-rouge">firefox.com</code> and <code class="language-plaintext highlighter-rouge">taskcluster.net</code>. Finally, <a href="https://crt.sh/?q=77575263">77575263</a> is an expired certificate for <code class="language-plaintext highlighter-rouge">blog.mozilla.com</code> (and some others).</p>
<p>We can also supply hosts, either on the command line, as an input file, or both:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>○ → cat >/tmp/top4.txt <<EOF
apple.com
youtube.com
www.google.com:443
# This is definitely half of my top 8 spaces
www.blogger.com
EOF
○ → moz_crlite_query --hosts mozilla.com firefox.com --hosts getfirefox.net --hosts-file /tmp/top4.txt
INFO:query_cli:Database was updated at 2020-11-25 22:19:26.467085, skipping.
INFO:query_cli:Status: 2457 Intermediates, Current filter: 2020-11-24T00:08:12+00:00Z-full
with 27 layers and 43053008 bit-count, 8 stash files with 43022 stashed revocations,
up-to-date as of 2020-11-26 00:08:11+00:00 (5:21:25.126138 ago).
mozilla.com:443 Issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US
Enrolled in CRLite: ❌
Result: ❔ Not Enrolled ❔
firefox.com:443 Issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US
Enrolled in CRLite: ❌
Result: ❔ Not Enrolled ❔
getfirefox.net:443 Issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US
Enrolled in CRLite: ❌
Result: ❔ Not Enrolled ❔
apple.com:443 Issuer: CN=DigiCert SHA2 Extended Validation Server CA-3,OU=www.digicert.com,O=DigiCert\, Inc.,C=US
Enrolled in CRLite: ✅
Result: 👍 Valid 👍
youtube.com:443 Issuer: CN=GTS CA 1O1,O=Google Trust Services,C=US
Enrolled in CRLite: ✅
Result: 👍 Valid 👍
www.google.com:443 Issuer: CN=GTS CA 1O1,O=Google Trust Services,C=US
Enrolled in CRLite: ✅
Result: 👍 Valid 👍
www.blogger.com:443 Issuer: CN=GTS CA 1O1,O=Google Trust Services,C=US
Enrolled in CRLite: ✅
Result: 👍 Valid 👍
</code></pre></div></div>
<p>This is all in good fun, but there’s more possibility here than just a command-line gimmick.</p>
<h3 id="embedding-crlite-into-other-tools">Embedding CRLite into Other Tools?</h3>
<p>Many tools use the Mozilla Root CA program’s list of trust anchors for their trust decisions, particularly on Linux or BSD operating systems. For many such tools, certificate revocations are sufficiently unlikely and the consequences sufficiently low that checking revocation status is simply unimportant. For those tools that do check, leaking the status check in cleartext to network observers is likely to simply be evidence of doing a software update. However, I can imagine that some tools might benefit from CRLite’s infrastructure-as-a-service, and make good use of the privacy benefit.</p>
<p>As it’s not particularly difficult to <a href="https://github.com/mozilla/moz_crlite_query/">develop a CRLite client</a>, perhaps if this tooling makes it to Firefox Release, we’ll see some libraries for non-Firefox users of CRLite.</p>
<p>While Mozilla certainly isn’t intending to have numerous users of the data beyond Firefox — at least they aren’t accounted for in the budget — the data is there, and there to be useful.</p>Firefox Nightly is now using CRLite to determine if websites’ certificates are revoked — e.g., if the Certificate Authority published that web browsers shouldn’t trust that website certificate. Telemetry shows that querying the local CRLite dataset is much faster than making a network connection for OCSP, which makes intuitive sense. It also avoids sending the website’s certificate information in cleartext over the network to check the revocation status: solving one of the remaining cleartext browsing data leakages in Firefox. Mozilla is currently publishing CRLite data to Remote Settings four times per day, keeping a very fresh set of revocation information for the public Web. I’ve provided some direct details on how to get at that data from the CRLite FAQ, and I want to introduce one of my command-line tools I’ve used to analyze and play with the dataset: moz_crlite_query. I’ll introduce crlite_status in a later post. Querying CRLite Status at the CLI moz_crlite_query is a Python tool that more-or-less implements the same algorithm Firefox does to check a website’s certificate against CRLite. It maintains a local database of the CRLite dataset in your ~/.crlite_db/ directory, which for me right now looks like: 0 Nov 25 09:24 .last_updated 5381898 Nov 25 09:24 2020-11-24T00:08:12+00:00Z-full 27363 Nov 25 09:24 2020-11-24T06:08:12+00:00Z-diff 59196 Nov 25 09:24 2020-11-24T12:08:14+00:00Z-diff 56121 Nov 25 09:24 2020-11-24T18:08:15+00:00Z-diff 204863 Nov 25 09:24 2020-11-25T00:08:23+00:00Z-diff 25826 Nov 25 09:24 2020-11-25T06:08:05+00:00Z-diff 66626 Nov 25 09:24 2020-11-25T12:08:22+00:00Z-diff 25826 Nov 25 09:24 2020-11-25T06:08:05+00:00Z-diff 66626 Nov 25 09:24 2020-11-25T12:08:22+00:00Z-diff 53783 Nov 25 22:19 2020-11-25T18:08:11+00:00Z-diff 190353 Nov 25 22:19 2020-11-26T00:08:11+00:00Z-diff 1118208 Nov 25 22:19 intermediates.sqlite You can see here that the last full CRLite filter update happened at the midnight-UTC update on 24 November 2020, and since then there have been eight delta updates (e.g., “stashes”). moz_crlite_query is distributed as a Python package that can be installed via an invocation of pip (Note that this tool requires Python 3.7 or newer because of SQLite features.): pip install moz_crlite_query On invocation, the tool will try and update to the latest CRLite data, unless it already updated in the last several hours. You can query the CRLite status for either a certificate you already have, or the tool can connect to one or more websites and examine the certificates presented there. Experimenting with the Query Tool Let’s download a few certificates into /tmp from crt.sh: 77575263, 1988442812, 1485147627, and 2680822568. These simply are representative – right now – of various states in the Web PKI. ○ → for id in 77575263 1988442812 1485147627 2680822568; do > curl --silent https://crt.sh/?d=${id} > /tmp/${id}.pem > done Now, run them all through CRLite: ○ → moz_crlite_query /tmp/*.pem INFO:crlite_query:CRLite Update: Syncing CRLite filters. 100% (9 of 9) |######################################################| Elapsed Time: 0:00:00 Time: 0:00:00 INFO:query_cli:Status: 2457 Intermediates, Current filter: 2020-11-24T00:08:12+00:00Z-full with 27 layers and 43053008 bit-count, 8 stash files with 43022 stashed revocations, up-to-date as of 2020-11-26 00:08:11+00:00 (5:11:15.467645 ago). /tmp/1485147627.pem Issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US Enrolled in CRLite: ❌ Result: ❔ Not Enrolled ❔ /tmp/1988442812.pem Issuer: CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US Enrolled in CRLite: ✅ Revoked via CRLite filter: 2020-11-24T00:08:12+00:00Z-full Result: ⛔️ Revoked ⛔️ /tmp/2680822568.pem Issuer: CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US Enrolled in CRLite: ✅ Result: 👍 Valid 👍 /tmp/77575263.pem Issuer: CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US Enrolled in CRLite: ✅ Result: ⏰ Expired ⏰ We can see 1485147627 is issued by a CA not participating in CRLite: Let’s Encrypt doesn’t publish CRLs. 1988442812 is for revoked.badssl.com, and is revoked. 2680822568 is currently a certificate used for many Mozilla properties, like firefox.com and taskcluster.net. Finally, 77575263 is an expired certificate for blog.mozilla.com (and some others). We can also supply hosts, either on the command line, as an input file, or both: ○ → cat >/tmp/top4.txt <<EOF apple.com youtube.com www.google.com:443 # This is definitely half of my top 8 spaces www.blogger.com EOF ○ → moz_crlite_query --hosts mozilla.com firefox.com --hosts getfirefox.net --hosts-file /tmp/top4.txt INFO:query_cli:Database was updated at 2020-11-25 22:19:26.467085, skipping. INFO:query_cli:Status: 2457 Intermediates, Current filter: 2020-11-24T00:08:12+00:00Z-full with 27 layers and 43053008 bit-count, 8 stash files with 43022 stashed revocations, up-to-date as of 2020-11-26 00:08:11+00:00 (5:21:25.126138 ago). mozilla.com:443 Issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US Enrolled in CRLite: ❌ Result: ❔ Not Enrolled ❔ firefox.com:443 Issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US Enrolled in CRLite: ❌ Result: ❔ Not Enrolled ❔ getfirefox.net:443 Issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US Enrolled in CRLite: ❌ Result: ❔ Not Enrolled ❔ apple.com:443 Issuer: CN=DigiCert SHA2 Extended Validation Server CA-3,OU=www.digicert.com,O=DigiCert\, Inc.,C=US Enrolled in CRLite: ✅ Result: 👍 Valid 👍 youtube.com:443 Issuer: CN=GTS CA 1O1,O=Google Trust Services,C=US Enrolled in CRLite: ✅ Result: 👍 Valid 👍 www.google.com:443 Issuer: CN=GTS CA 1O1,O=Google Trust Services,C=US Enrolled in CRLite: ✅ Result: 👍 Valid 👍 www.blogger.com:443 Issuer: CN=GTS CA 1O1,O=Google Trust Services,C=US Enrolled in CRLite: ✅ Result: 👍 Valid 👍 This is all in good fun, but there’s more possibility here than just a command-line gimmick. Embedding CRLite into Other Tools? Many tools use the Mozilla Root CA program’s list of trust anchors for their trust decisions, particularly on Linux or BSD operating systems. For many such tools, certificate revocations are sufficiently unlikely and the consequences sufficiently low that checking revocation status is simply unimportant. For those tools that do check, leaking the status check in cleartext to network observers is likely to simply be evidence of doing a software update. However, I can imagine that some tools might benefit from CRLite’s infrastructure-as-a-service, and make good use of the privacy benefit. As it’s not particularly difficult to develop a CRLite client, perhaps if this tooling makes it to Firefox Release, we’ll see some libraries for non-Firefox users of CRLite. While Mozilla certainly isn’t intending to have numerous users of the data beyond Firefox — at least they aren’t accounted for in the budget — the data is there, and there to be useful.Austin or Bust: A Post-Y’allhands Adventure2020-11-01T00:00:00-07:002020-11-01T00:00:00-07:00https://insufficient.coffee/2020/11/01/austin-or-bust<p>Due to a variety of factors, Mozilla changed venues for our 2017 winter all-hands meeting to Austin, Texas, and Austin being a city with relatively poor commercial airline connectivity, ticket prices were fairly high even for me to fly there from Phoenix.<!-- more --> High enough, actually, that I’d already flown myself to Austin earlier that year for a graduation — two round-trip economy tickets cost more than the aviation fuel it’d take to fly ourselves. The same general logic held for the Mozilla trip, but there was only one of me. But if I brought two colleagues back with me and they could get much cheaper airfare out of Phoenix Sky Harbor… well, a spreadsheet proved the math, but we all know the real idea here.</p>
<p>It gave us the Great Post Ya’ll-Hands Adventure!</p>
<p>But getting ahead of myself just a little - weather between Arizona and eastern Texas in December tends to be reasonably predictable: there’s unpredictable fog there in coastal Texas, but generally bad weather comes as fronts. As long as you can be flexible on your trip dates if there’s going to be a frontal passage soon, it’s not a bad time of year to take to the skies in the southern US.</p>
<p>I set myself up so that I had options to arrive in Austin up to several days in advance of the week of meetings, had side-trips available for the return, leeway before the international flights home for my colleagues, and the definite option to leave the conference early should we need to. It looked good, so I proceeded to confuse the hell out of the travel folks for the two months before the conference.</p>
<p>Watching the weather closely the week ahead of the event, the route out looked like it was going to be smooth sailing on the official arrival day, so I didn’t take off early. I planned for an 8:30am departure out of <a href="airnav.com/airport/KCHD">KCHD</a>. When I flew to Austin earlier that year, I landed at Austin Executive. This time, since Mozilla was meeting at Austin International, I checked prices and found them to be a wash for a Cessna 182, so I filed <abbr title="Instrument Flight Rules, meaning I'm safe to fly into clouds">IFR</abbr> to <a href="airnav.com/airport/KAUS">KAUS</a>.</p>
<p>Fully loaded with 88 gallons of fuel - about 8 hours of cruising time - I had plenty of range to make it the 780 nautical miles with IFR reserves to spare.</p>
<p>At this time I had a cute battery-electric car which I could pull into the hangar and plug in so it’d be fully charged for the return home.</p>
<figure>
<a href="/content/2020/11/yallhands-i3-departing.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-i3-departing.1024.webp" />
<img src="/content/2020/11/yallhands-i3-departing.1024.png" alt="My i3, plugged in to trickle-charge in the hangar for the trip" />
</picture>
</a>
<figcaption>My i3, plugged in to trickle-charge in the hangar for the trip</figcaption>
</figure>
<p>Chandler Tower had me waiting almost 20 minutes for an IFR release that morning; plenty of time to plug in the full IFR clearance, check Twitter, you know. But the engine was running, and just as I was getting ready to take off <abbr title="Visual Flight Rules">VFR</abbr> and activate en-route, Tower called, something like, “N625LS, I have your IFR release, line up and wait runway 4L.”</p>
<p>Of course, there was a taildragger that had pulled in front of me. “Unable, I have an aircraft ahead of me on the taxiway.”</p>
<figure>
<a href="/content/2020/11/yallhands-kchd-taildragger.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-kchd-taildragger.1024.webp" />
<img src="/content/2020/11/yallhands-kchd-taildragger.1024.png" alt="I don't think it was this particular aircraft that delayed me, but I sat so long, this was just one of many." />
</picture>
</a>
<figcaption>I don't think it was this particular aircraft that delayed me, but I sat so long, this was just one of many.</figcaption>
</figure>
<p>Then the tower controller had fun trying to raise the other plane, which hadn’t yet switched to the tower’s radio frequency, to get them out of the way. Phoenix Departure was presumably holding airspace open for me, so they wanted me in the air ASAP, which hey, suited me after that wait. Chandler isn’t known to be busy enough to clear multiple aircraft in one instruction, but here it was in one breath something like, “(taildragger) cleared for takeoff runway 4 left no delay Cessna 625LS line up and wait runway 4 left no delay!”</p>
<p>Yes, sir. Of course, all that and Phoenix Departure has me fly 20 miles straight South (a bit out of the way) before letting me turn on-course, but it was smooth sailing the rest of the way.</p>
<figure>
<a href="/content/2020/11/yallhands-kchd-kaus-map.png">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-kchd-kaus-map.1024.webp" />
<img src="/content/2020/11/yallhands-kchd-kaus-map.1024.png" alt="Actual flight path in red" />
</picture>
</a>
<figcaption>Actual flight path in red</figcaption>
</figure>
<p>The flight was about 5 hours and 45 minutes, though the delay at the front made the time to be 6.1 hours in my logbook. I took a few photos, but nothing much remarkable. Mostly I tuned to an XM satellite radio station while I listened to long stretches of radio silence, watched the engine instruments, and kept tabs of the “Nearest Airports” list.</p>
<figure>
<a href="/content/2020/11/yallhands-kchd-kaus-enroute-hat.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-kchd-kaus-enroute-hat.1024.webp" />
<img src="/content/2020/11/yallhands-kchd-kaus-enroute-hat.1024.png" alt="Crusing at 10,000 feet doing 144 knots true (165 mph) near Fort Stockton, TX" />
</picture>
</a>
<figcaption>Crusing at 10,000 feet doing 144 knots true (165 mph) near Fort Stockton, TX</figcaption>
</figure>
<p>The Austin area had high clouds but unlimited visibility, and I arrived with the computer showing over two hours remaining fuel endurance left, plus another 45 minute IFR reserve.</p>
<figure>
<a href="/content/2020/11/yallhands-kaus-fuel-endurance.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-kaus-fuel-endurance.1024.webp" />
<img src="/content/2020/11/yallhands-kaus-fuel-endurance.1024.png" alt="2 hours endurance, plus then 45 minutes of reserves remained" />
</picture>
</a>
<figcaption>2 hours endurance, plus then 45 minutes of reserves remained</figcaption>
</figure>
<p>Austin Approach asked me two times to confirm I was landing at Austin International and not Austin Executive, and then asked in advance where I was parking - that’s a little odd for the approach controller to care about parking details, but I guess I was enough of a curiosity that they checked carefully.</p>
<p>Even though the day was totally clear, Approach gave me radar vectors to the <code class="language-plaintext highlighter-rouge">SCALI</code> intersection for runway 17 Left before letting me turn inbound on the final approach course. It was pretty clear what was happening, as I’d studied all the approaches, so I plugged in the localizer and the <abbr title="Instrument Landing System">ILS</abbr> procedure into the navigation system and flew the 6 mile final approach via <code class="language-plaintext highlighter-rouge">RRTOO</code> and <code class="language-plaintext highlighter-rouge">DDTOO</code>.</p>
<figure>
<a href="/content/2020/11/yallhands-austin-approach-path.png">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-austin-approach-path.1024.webp" />
<img src="/content/2020/11/yallhands-austin-approach-path.png" />
</picture>
</a>
</figure>
<p>I’m honestly not sure why I was given such a wide path that day; there was other inbound airliner traffic that was sequenced behind me, and I never heard anyone ahead of me. Anyway, there was a reason, and it was an easy approach - since it was an ILS path anyway, some of it I let the autopilot fly.</p>
<p>As Approach had asked where I was parking in advance, Austin Tower and Ground already knew, and the ground controller gave me a hearty “Welcome to Texas!” followed by progressive taxi instructions without me even having to ask, which I greatly appreciated.</p>
<figure>
<a href="/content/2020/11/yallhands-kaus-parked.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-kaus-parked.1024.webp" />
<img src="/content/2020/11/yallhands-kaus-parked.1024.png" alt="Parked where the FBO marshalled me." />
</picture>
</a>
<figcaption>Parked where the FBO marshalled me.</figcaption>
</figure>
<p>The <abbr title="Fixed Base Operator; the folks who sell fuel and charge parking fees">FBO</abbr>’s “follow me” cart had me park kind of in the middle of nowhere, but they gave me a ride back to their building so hey, no problem. I left the aircraft in their hands with a request for a couple dozen gallons of fuel to be added; I was playing the game of “enough fuel for them to waive fees” but not “pay their inflated prices for the whole tank.”</p>
<p>The work week went by quite quickly. <a href="/2017/12/04/countering-phishing-with-cryptography/">There’s even a recording of my lightning talk about Web Authentication</a> for people who want a break from all this fun aviation stuff. I kept a close eye on the weather, as it wasn’t going to be as easy a trip home as it was to come out.</p>
<p>My return-trip passengers and I had several check-in points about whether we’d need to leave early on Friday, or stay through the end of the work week and depart Saturday, but it ultimately looked like Friday wouldn’t help us, we really needed to wait until Sunday to cross Texas again.</p>
<p>I had a contingency planned. Saturday morning came with the promise of bad weather from a cold front moving across Texas, but while it was raining already, the bad stuff hadn’t arrived yet. Still, we were definitely going to wait it out — but we didn’t have to wait it out in Austin! I had family 150 miles south in Corpus Christi, Texas, on the coast of the Gulf of Mexico, and they were happy to spend their Saturday with us.</p>
<p>The weather was coming though, and Corpus was varying back and forth between instrument conditions with fog and marginal visibility. The forecast had more than 5 hours before Austin or San Antonio were going to have real weather issues, though, so we had a variety of options if we couldn’t make it into Corpus Christi. I filed an IFR flight plan with an alternate of San Antonio.</p>
<p>So, we three boarded a shuttle bus to the Austin airport, and again puzzled a poor helper since we were on the list, but had no flight information. Oh, don’t worry about it…</p>
<p>The FBO sent a car around to pick us up from the commercial terminal and had pulled out and fueled the plane. The preflight went well, bill paid, and we tried to speed ahead of the incoming serious weather, but not too fast for the fog to burn off!</p>
<p>Austin was very efficient; Clearance Delivery had our clearance available immediately, and Ground gave us excellent taxi instructions without asking.</p>
<figure>
<a href="/content/2020/11/yallhands-kaus-making-a-southwest-flight-wait.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-kaus-making-a-southwest-flight-wait.1024.webp" />
<img src="/content/2020/11/yallhands-kaus-making-a-southwest-flight-wait.1024.png" />
</picture>
</a>
<figcaption>photo credit <a href="https://creativecommons.org/licenses/by/4.0/">cc-by 4.0</a> by <a href="https://www.franziskuskiefer.de/">Franziskus Kiefer</a></figcaption>
</figure>
<p>My colleagues loved that Ground made a Southwest 737 wait for us. We’re sure we knew people on that flight, since the airport would have been saturated with Mozillians that morning.</p>
<figure>
<video controls="" loop="1" poster="/content/2020/11/yallhands-departing-austin.jpg">
<source type="video/webm; codecs=vp9" src="https://i.have.insufficient.coffee/2020-11-yallhands-departing-austin.webm" />
<source type="video/m4v" src="https://i.have.insufficient.coffee/2020-11-yallhands-departing-austin.mp4" />
Your browser does not support HTML5 video. You can download it directly in one of these formats:
<ul>
<li><a href="https://i.have.insufficient.coffee/2020-11-yallhands-departing-austin.mp4">maximum quality</a></li>
<li><a href="https://i.have.insufficient.coffee/2020-11-yallhands-departing-austin.webm">webm vp9 high quality</a></li>
</ul>
</video>
<figcaption>Departing Austin International Airport, TX; video credit <a href="https://creativecommons.org/licenses/by/4.0/">cc-by 4.0</a> by <a href="https://www.franziskuskiefer.de/">Franziskus Kiefer</a></figcaption>
</figure>
<p>We climbed out of Austin and almost immediately went into a cloud layer. This was the first time my passengers had been up in a light aircraft, so they were looking forward to the views from the air, and of course we stayed in that cloud layer the rest of the flight.</p>
<figure>
<a href="/content/2020/11/yallhands-kaus-kcrp-enroute-weather.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-kaus-kcrp-enroute-weather.1024.webp" />
<img src="/content/2020/11/yallhands-kaus-kcrp-enroute-weather.1024.png" alt="The datalink weather implied we were going to get out of the rain as we headed to Corpus Christi. We didn't." />
</picture>
</a>
<figcaption>The datalink weather implied we were going to get out of the rain as we headed to Corpus Christi. We didn't. photo credit <a href="https://creativecommons.org/licenses/by/4.0/">cc-by 4.0</a> by <a href="https://www.franziskuskiefer.de/">Franziskus Kiefer</a></figcaption>
</figure>
<p>Flight time was about one hour from Austin down to the coast. I was hoping we wouldn’t find turbulence in those clouds, luckily it stayed pretty smooth.</p>
<figure>
<a href="/content/2020/11/yallhands-kaus-kcrp-enroute-pilot.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-kaus-kcrp-enroute-pilot.1024.webp" />
<img src="/content/2020/11/yallhands-kaus-kcrp-enroute-pilot.1024.png" alt="Not much to see outside for this hour-long flight" />
</picture>
</a>
<figcaption>Not much to see outside for this hour-long flight. photo credit <a href="https://creativecommons.org/licenses/by/4.0/">cc-by 4.0</a> by <a href="https://www.franziskuskiefer.de/">Franziskus Kiefer</a></figcaption>
</figure>
<p>I was also hoping any minute now there’d be news from Corpus that they were in visual conditions again, but as we passed the halfway point Houston Approach advised me to expect the <abbr title="Instrument Landing System">ILS</abbr> for runway 36.</p>
<figure>
<a href="/content/2020/11/yallhands-kaus-kcrp-map.png">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-kaus-kcrp-map.1024.webp" />
<img src="/content/2020/11/yallhands-kaus-kcrp-map.1024.png" alt="Austin to Corpus Christi is 145 nautical miles; our path was 178 nm entirely because of the instrument approach at the end." />
</picture>
</a>
<figcaption>Austin to Corpus Christi is 145 nautical miles; our path was 178 nm entirely because of the instrument approach at the end.</figcaption>
</figure>
<p>When I checked in with Corpus tower on their ILS approach, they were nice enough to ask me to remain at my best possible speed because of “other traffic.” Certainly that’s what piston engine drivers want to hear when they’re in zero visibility at an unfamiliar airport, but I replied that I was already at my best speed – and, looking back at the data, I absolutely was! I put the other traffic out of my mind and flew the approach to the best of my ability; it worked out great, breaking clear at about 500 ft above minimums, though the controller wasn’t hiding their annoyance at asking the flight of aircraft behind me on the ILS to fly the missed approach.</p>
<figure>
<video controls="" loop="1" poster="/content/2020/11/yallhands-kcrp-ils-36-approach.jpg">
<source type="video/webm; codecs=vp9" src="https://i.have.insufficient.coffee/2020-11-yallhands-kcrp-ils-36-approach.webm" />
<source type="video/m4v" src="https://i.have.insufficient.coffee/2020-11-yallhands-kcrp-ils-36-approach-2mbit.mp4" />
<source type="video/m4v" src="https://i.have.insufficient.coffee/2020-11-yallhands-kcrp-ils-36-approach.mp4" />
Your browser does not support HTML5 video. You can download it directly in one of these formats:
<ul>
<li><a href="https://i.have.insufficient.coffee/2020-11-yallhands-kcrp-ils-36-approach.mp4">maximum quality H.264</a></li>
<li><a href="https://i.have.insufficient.coffee/2020-11-yallhands-kcrp-ils-36-approach-2mbit.mp4">2Mbit H.264</a></li>
<li><a href="https://i.have.insufficient.coffee/2020-11-yallhands-kcrp-ils-36-approach.webm">webm vp9 256Kbit quality</a></li>
</ul>
</video>
<figcaption>Instrument Landing System approach into Corpus Christi Airport, TX; video credit <a href="https://creativecommons.org/licenses/by/4.0/">cc-by 4.0</a> by <a href="https://twitter.com/mr_goodwin">Mark Goodwin</a></figcaption>
</figure>
<p>When we’d parked at the FBO and shut down, my colleagues pointed out that the flight of aircraft behind us were <a href="https://en.wikipedia.org/wiki/McDonnell_Douglas_T-45_Goshawk">US Navy T-45 Goshawks</a>, coming our way!</p>
<figure>
<a href="/content/2020/11/yallhands-kcrp-goshawks.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-kcrp-goshawks.1024.webp" />
<img src="/content/2020/11/yallhands-kcrp-goshawks.1024.png" alt="US Navy McDonnell Douglas T-45 Goshawks that followed us in" />
</picture>
</a>
<figcaption>US Navy McDonnell Douglas T-45 Goshawks that followed us in. photo credit <a href="https://creativecommons.org/licenses/by/4.0/">cc-by 4.0</a> by <a href="https://www.franziskuskiefer.de/">Franziskus Kiefer</a></figcaption>
</figure>
<p>The Goshawks parked a few dozen meters away from us; we had a friendly wave to the pilots as they secured the aircraft and headed in from the light rain.</p>
<p>The weather got worse right on time, but at that point we had a rental car and were heading downtown to meet my family that lived there in Corpus Christi for lunch at a cafe in the marina.</p>
<figure>
<a href="/content/2020/11/yallhands-corpus-lexington-museum.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-corpus-lexington-museum.1024.webp" />
<img src="/content/2020/11/yallhands-corpus-lexington-museum.1024.png" alt="The USS Lexington Museum in Corpus Christi provided a thematic entertainment on the rainy afternoon" />
</picture>
</a>
<figcaption>The USS Lexington Museum in Corpus Christi provided a thematic entertainment on the rainy afternoon</figcaption>
</figure>
<p>The day was foggy and wet, and the cold front ensured it stayed that way the rest of the day. We spent the afternoon at the USS Lexington Museum, in keeping with the aviation theme of the trip. That aircraft carrier was cold, wet, and rainy, and it was a great afternoon to be happy of being on the ground rather than up in the soup.</p>
<figure>
<a href="/content/2020/11/yallhands-corpus-bar.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-corpus-bar.1024.webp" />
<img src="/content/2020/11/yallhands-corpus-bar.1024.png" alt="Can't visit Texas without hitting a proper dive bar" />
</picture>
</a>
<figcaption>Can't visit Texas without hitting a proper dive bar. photo credit <a href="https://creativecommons.org/licenses/by/4.0/">cc-by 4.0</a> by <a href="https://www.franziskuskiefer.de/">Franziskus Kiefer</a></figcaption>
</figure>
<p>We went to a real Texas dive bar to observe the local customs. I stuck to soda, and after a good Tex-Mex dinner, poured through the weather forecasts for the following day.</p>
<p>Flying from Phoenix to Austin nonstop is not hard in a Skylane, but going the opposite way generally runs into a headwind, lengthening the trip and requiring a fuel stop. Complicating matters was that the weather there in Corpus Christi wasn’t expected to improve until late in the morning, but at that point weather all across Texas was to be in good shape the whole day, other than a chance of some small storms in the mountains near El Paso.</p>
<p>It wasn’t difficult to find an excuse for a delayed start that Sunday morning: we went for brunch so that our Chicken and Waffle connoisseur could experience the Corpus Christi version of the height of American cuisine.</p>
<figure>
<a href="/content/2020/11/yallhands-corpus-chicken-and-waffle.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-corpus-chicken-and-waffle.1024.webp" />
<img src="/content/2020/11/yallhands-corpus-chicken-and-waffle.1024.png" alt="Worthy of any manner of quest" />
</picture>
</a>
<figcaption>Chicken-and-Waffle: Worthy of any manner of quest. photo credit <a href="https://creativecommons.org/licenses/by/4.0/">cc-by 4.0</a> by <a href="https://www.franziskuskiefer.de/">Franziskus Kiefer</a></figcaption>
</figure>
<p>I tried not to agonize over our flight path too much during brunch; when we were done eating, I was going to file our flight plan as whatever looked best then.</p>
<p>Shortly before noon we arrived back at Corpus Christi’s FBO, which was a busy place that morning. A group of naval aviators were waiting for clearance to depart — a little small-talk as I paid our bill revealed they were the pilots I made go-around the day before. No hard feelings, they said: excuses for extra flight time were always useful. They warned me about the quality of the coffee; apparently a running joke with the FBO staff. (It wasn’t that bad.)</p>
<figure>
<a href="/content/2020/11/yallhands-kcrp-goshawks-departing.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-kcrp-goshawks-departing.1024.webp" />
<img src="/content/2020/11/yallhands-kcrp-goshawks-departing.1024.png" alt="The Goshawks departed right before us." />
</picture>
</a>
<figcaption>The Goshawks departed right before us. photo credit <a href="https://creativecommons.org/licenses/by/4.0/">cc-by 4.0</a> by <a href="https://www.franziskuskiefer.de/">Franziskus Kiefer</a></figcaption>
</figure>
<p>It worked out that the squadron of Goshawks were all departing as we finished pre-flighting and buttoning up our Skylane, so we took the opportunity to watch and wave, and they waggled their ailerons back at us.</p>
<figure>
<video controls="" loop="1" poster="/content/2020/11/yallhands-departing-corpus.jpg">
<source type="video/webm; codecs=vp9" src="https://i.have.insufficient.coffee/2020-11-yallhands-departing-corpus.webm" />
<source type="video/m4v" src="https://i.have.insufficient.coffee/2020-11-yallhands-departing-corpus.mp4" />
Your browser does not support HTML5 video. You can download it directly in one of these formats:
<ul>
<li><a href="https://i.have.insufficient.coffee/2020-11-yallhands-departing-corpus.mp4">maximum quality</a></li>
<li><a href="https://i.have.insufficient.coffee/2020-11-yallhands-departing-corpus.webm">webm vp9 256Kbit quality</a></li>
</ul>
</video>
<figcaption>Departing Corpus Christi Airport, TX; video credit <a href="https://creativecommons.org/licenses/by/4.0/">cc-by 4.0</a> by <a href="https://www.franziskuskiefer.de/">Franziskus Kiefer</a></figcaption>
</figure>
<p>The flight plan I filed was from Corpus Christi to Albuquerque, NM, specifically to <a href="https://airnav.com/airport/KAEG">Double Eagle II Airport</a> a short distance west of town. I had already flown to Double Eagle a few times and knew the area. Our scheduled arrival time was reasonably late in the day at this point, so I called ahead to let the FBO there know we were coming and would need fuel and a rental car. It was going to be late enough in the day we could head into the Old Town and see <a href="https://www.visitalbuquerque.org/about-abq/culture-heritage/holiday-traditions/luminarias/">Albuquerque’s holiday luminaria tradition</a>. Also, I just love eating excellent New Mexican food, and planning this way would involve eating in Albuquerque. It’s legit.</p>
<p>We took off out of Corpus Christi into low clouds but generally good weather on an IFR flight plan. As soon as we were handed to Houston Approach, they cleared us direct to Albuquerque. No joke: a 660 nautical mile direct leg. Check it out:</p>
<figure>
<a href="/content/2020/11/yallhands-kcrp-kaeg-map.png">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-kcrp-kaeg-map.1024.webp" />
<img src="/content/2020/11/yallhands-kcrp-kaeg-map.1024.png" alt="Flight path from Corpus Christi to Albuquerque" />
</picture>
</a>
<figcaption>Flight path from Corpus Christi to Albuquerque</figcaption>
</figure>
<p>This was the first flight segment for my two passengers with good conditions, though we were above a cloud layer for the first third of the trip.</p>
<figure>
<a href="/content/2020/11/yallhands-kcrp-kaeg-cloudcover.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-kcrp-kaeg-cloudcover.1024.webp" />
<img src="/content/2020/11/yallhands-kcrp-kaeg-cloudcover.1024.png" alt="The cloudcover persisted until we were over central Texas" />
</picture>
</a>
<figcaption>The cloudcover persisted until we were over central Texas; photo credit <a href="https://creativecommons.org/licenses/by/4.0/">cc-by 4.0</a> by <a href="https://www.franziskuskiefer.de/">Franziskus Kiefer</a></figcaption>
</figure>
<p>The clouds gave way to the oilfields of west Texas which are always a fascinating sight by air.</p>
<figure>
<a href="/content/2020/11/yallhands-kcrp-kaeg-oilfields.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-kcrp-kaeg-oilfields.1024.webp" />
<img src="/content/2020/11/yallhands-kcrp-kaeg-oilfields.1024.png" alt="Hundreds of miles of oilfields" />
</picture>
</a>
<figcaption>Hundreds of miles of oilfields; photo credit <a href="https://creativecommons.org/licenses/by/4.0/">cc-by 4.0</a> by <a href="https://www.franziskuskiefer.de/">Franziskus Kiefer</a></figcaption>
</figure>
<p>As expected, we fought a significant headwind the whole way to Albuquerque, even at 12,000 feet. The storms that I feared near El Paso absolutely materialized, and covered a larger-than-forecast area, but we only diverted a tiny amount over that long flight, and even that out of an abundance of caution.</p>
<figure>
<a href="/content/2020/11/yallhands-kcrp-kaeg-elpaso-weather.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-kcrp-kaeg-elpaso-weather.1024.webp" />
<img src="/content/2020/11/yallhands-kcrp-kaeg-elpaso-weather.jpg" alt="Keeping close track of the weather near El Paso, Texas" />
</picture>
</a>
<figcaption>Keeping close track of the weather near El Paso, Texas; photo credit <a href="https://creativecommons.org/licenses/by/4.0/">cc-by 4.0</a> by <a href="https://www.franziskuskiefer.de/">Franziskus Kiefer</a></figcaption>
</figure>
<p>The headwind made it long, so when we were handed to a very bored controller at Roswell, New Mexico to transit his airspace, we filed a pilot report rather than listen to the silence.</p>
<p>Approaching the Sandia and Manzano mountains that form the eastern border of Albuquerque, we finally encountered real turbulence. My colleagues bore it well, and we made our way through that natural wind funnel into the city’s valley.</p>
<p>We arrived at sundown, and just as always, Double Eagle pulled our rental car right up to the plane for our convenience, and we quickly made our way to the old town.</p>
<figure>
<a href="/content/2020/11/yallhands-alb-pilot.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-alb-pilot.1024.webp" />
<img src="/content/2020/11/yallhands-alb-pilot.1024.png" alt="Old Town Albuquerque is always great to visit" />
</picture>
</a>
<figcaption>Old Town Albuquerque is always great to visit; photo credit <a href="https://creativecommons.org/licenses/by/4.0/">cc-by 4.0</a> by <a href="https://www.franziskuskiefer.de/">Franziskus Kiefer</a></figcaption>
</figure>
<p>I got my New Mexican food, and we all enjoyed the lit up Old Town area. I suppose it’s not a true spectacle like Manhattan or some of the capitals of Europe at Christmas, but there’s Western Charm in abundance.</p>
<p>We had a comfortable night downtown and our plane was again fueled and ready to go the rest of the way home when we got to the airport.</p>
<figure>
<a href="/content/2020/11/yallhands-kaeg-ramp.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-kaeg-ramp.1024.webp" />
<img src="/content/2020/11/yallhands-kaeg-ramp.1024.png" alt="Waiting on the Double Eagle II ramp" />
</picture>
</a>
<figcaption>Waiting on the Double Eagle II ramp; photo credit <a href="https://creativecommons.org/licenses/by/4.0/">cc-by 4.0</a> by <a href="https://www.franziskuskiefer.de/">Franziskus Kiefer</a></figcaption>
</figure>
<p>I again filed IFR from Albuquerque back to Chandler, but there was no need for this segment.</p>
<figure>
<video controls="" loop="1" poster="/content/2020/11/yallhands-departing-albuquerque.jpg">
<source type="video/webm; codecs=vp9" src="https://i.have.insufficient.coffee/2020-11-yallhands-departing-albuquerque.webm" />
<source type="video/m4v" src="https://i.have.insufficient.coffee/2020-11-yallhands-departing-albuquerque.mp4" />
Your browser does not support HTML5 video. You can download it directly in one of these formats:
<ul>
<li><a href="https://i.have.insufficient.coffee/2020-11-yallhands-departing-albuquerque.mp4">maximum quality</a></li>
<li><a href="https://i.have.insufficient.coffee/2020-11-yallhands-departing-albuquerque.webm">webm vp9 256Kbit quality</a></li>
</ul>
</video>
<figcaption>Departing Albuquerque Double Eagle Airport, NM; video credit <a href="https://creativecommons.org/licenses/by/4.0/">cc-by 4.0</a> by <a href="https://www.franziskuskiefer.de/">Franziskus Kiefer</a></figcaption>
</figure>
<p>We flew across the Continental Divide and, for better winds, climbed up to 14,000 feet. I’d used my oxygen tank on several occasions for the trip so far, this gave me an opportunity to run the rest of it down — after all, it costs the same to refill a 50% full tank as an empty one, and better winds are a nice bonus.</p>
<figure>
<a href="/content/2020/11/yallhands-kaeg-kchd-pulseox.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-kaeg-kchd-pulseox.1024.webp" />
<img src="/content/2020/11/yallhands-kaeg-kchd-pulseox.1024.png" alt="Flying high over the Continental Divide to catch the best winds requires carrying bottled oxygen - and making sure it's working." />
</picture>
</a>
<figcaption>Flying high over the Continental Divide to catch the best winds requires carrying bottled oxygen - and making sure it's working; photo credit <a href="https://creativecommons.org/licenses/by/4.0/">cc-by 4.0</a> by <a href="https://www.franziskuskiefer.de/">Franziskus Kiefer</a></figcaption>
</figure>
<p>We actually did fly through a number of small clouds on this leg of the flight, but they were all isolated, seemingly coming out of nowhere to get in our way. Had we been VFR, they would have been easy to dodge. Even IFR, they were small enough to dodge without being off-course, but icing wasn’t a concern at those temperatures and, ya know, clouds! Zoom!</p>
<figure>
<a href="/content/2020/11/yallhands-kaeg-kchd-clouds.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-kaeg-kchd-clouds.1024.webp" />
<img src="/content/2020/11/yallhands-kaeg-kchd-clouds.1024.png" alt="Small clouds dotted the way, and somehow our flight blasted through most of them." />
</picture>
</a>
<figcaption>Small clouds dotted the way, and somehow our flight blasted through most of them. I could have dodged, but ... fluffy!</figcaption>
</figure>
<p>The route from Albuquerque was nice for my colleagues compared to the south route through El Paso since it gave an excellent demonstration that Arizona isn’t all low desert and cacti. We flew through the Show Low / Pinetop area with its dramatic snow-capped terrain and temperate valleys.</p>
<figure>
<a href="/content/2020/11/yallhands-kaeg-kchd-pine-high-country.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-kaeg-kchd-pine-high-country.1024.webp" />
<img src="/content/2020/11/yallhands-kaeg-kchd-pine-high-country.1024.png" alt="Mountain towns dot the high elevations in Arizona, and look nothing like the low desert" />
</picture>
</a>
<figcaption>Mountain towns dot the high elevations in Arizona, and look nothing like the low desert; photo credit <a href="https://creativecommons.org/licenses/by/4.0/">cc-by 4.0</a> by <a href="https://www.franziskuskiefer.de/">Franziskus Kiefer</a></figcaption>
</figure>
<p>When we were handed to Phoenix Approach, they routed us over <a href="https://en.wikipedia.org/wiki/Theodore_Roosevelt_Lake">Roosevelt Lake, its distinctive bridge, and its dramatic dam</a></p>
<figure>
<a href="/content/2020/11/yallhands-kaeg-kchd-roosevelt-lake.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-kaeg-kchd-roosevelt-lake.1024.webp" />
<img src="/content/2020/11/yallhands-kaeg-kchd-roosevelt-lake.1024.png" alt="Roosevelt Lake is part of Phoenix's water reserves, and a fun recreation area" />
</picture>
</a>
<figcaption>Roosevelt Lake is part of Phoenix's water reserves, and a fun recreation area; photo credit <a href="https://creativecommons.org/licenses/by/4.0/">cc-by 4.0</a> by <a href="https://www.franziskuskiefer.de/">Franziskus Kiefer</a></figcaption>
</figure>
<p>Then it was back home, with the standard approach into Chandler from the East: vectored by Phoenix Approach into the way of Gateway Airport (poor Gateway controllers), then told to contact Chandler Tower and cleared to land.</p>
<figure>
<a href="/content/2020/11/yallhands-kaeg-kchd-approach-map.png">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-kaeg-kchd-approach-map.1024.webp" />
<img src="/content/2020/11/yallhands-kaeg-kchd-approach-map.png" alt="Poor Gateway airport gets their airspace disrupted by most IFR arrivals into Chandler" />
</picture>
</a>
<figcaption>Poor Gateway airport gets their airspace disrupted by most IFR arrivals into Chandler</figcaption>
</figure>
<p>The familiar sight of Chandler’s two runways was a good one. I celebrated being back home with my traditional ritual of landing left of the center-line. Oops.</p>
<figure>
<video controls="" loop="1" poster="/content/2020/11/yallhands-landing-chandler.jpg">
<source type="video/webm; codecs=vp9" src="https://i.have.insufficient.coffee/2020-11-yallhands-landing-chandler.webm" />
<source type="video/m4v" src="https://i.have.insufficient.coffee/2020-11-yallhands-landing-chandler.mp4" />
Your browser does not support HTML5 video. You can download it directly in one of these formats:
<ul>
<li><a href="https://i.have.insufficient.coffee/2020-11-yallhands-landing-chandler.mp4">maximum quality</a></li>
<li><a href="https://i.have.insufficient.coffee/2020-11-yallhands-landing-chandler.webm">webm vp9 256Kbit quality</a></li>
</ul>
</video>
<figcaption>Landing Chandler, AZ; video credit <a href="https://creativecommons.org/licenses/by/4.0/">cc-by 4.0</a> by <a href="https://www.franziskuskiefer.de/">Franziskus Kiefer</a></figcaption>
</figure>
<p>It was good for me to be back home; we flew 1121.3 nautical miles (2077 km) since we had left our fellow Mozillians in Austin, totaling 10 hours in the air. Obviously we added two side-trips to up the number from the 6 hours it took to fly outbound, but it is still simply slower to fight the prevailing winds.</p>
<figure>
<a href="/content/2020/11/yallhands-kchd-back-in-hangar.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/11/yallhands-kchd-back-in-hangar.1024.webp" />
<img src="/content/2020/11/yallhands-kchd-back-in-hangar.1024.png" alt="i3 and Skylane traded places" />
</picture>
</a>
<figcaption>i3 and Skylane traded places; photo credit <a href="https://creativecommons.org/licenses/by/4.0/">cc-by 4.0</a> by <a href="https://www.franziskuskiefer.de/">Franziskus Kiefer</a></figcaption>
</figure>
<p>The diversions for weather meant my companions didn’t have all that much time back in the Phoenix area with me before their flights back home, but we did make an opportunity for another American delicacy Chicken-and-Waffle at <em>Lo-Lo’s Chicken and Waffles</em> and yet more <a href="https://en.wikipedia.org/wiki/Oaxacan_cuisine">different Mexican food</a>.</p>
<p>This ended up as a perfect trip for reafirrming my personal limits with instrument flying, and with flying in general, and a good opportunity to exercise sound decisions in flight planning. I’m glad we braved the rain and low clouds to go to Corpus Christi, and I’m similarly glad we had a solid alternate plan. I’m glad we did alter our return trip to overnight in Albuquerque, and not just for the food. Above all, I’m glad we made this happen. It really was a Great American Road Trip, but one that just happened to average 200 km/h, and only stopped for souvenirs once.</p>Due to a variety of factors, Mozilla changed venues for our 2017 winter all-hands meeting to Austin, Texas, and Austin being a city with relatively poor commercial airline connectivity, ticket prices were fairly high even for me to fly there from Phoenix.Berlin By Air2020-10-05T00:00:00-07:002020-10-05T00:00:00-07:00https://insufficient.coffee/2020/10/05/berlin-by-air<p>I have had business dealings requiring me to visit Berlin regularly for the last 5 years, and in 2018 I finally made the trip outside of the city to <a href="https://acukwik.com/Airport-Info/EDAY">Strausberg Airport</a> to rent a Cessna 172SP Skyhawk. <!-- more --></p>
<p>I had heard, while flying with a flight instructor around the <a href="/2020/09/30/returning-home/">complex airspace in my home town in Northwest Florida</a>, that the procedures were a lot like flying VFR in most of Europe. That started me down a line of thinking about my upcoming trips to Berlin.</p>
<figure>
<a href="/content/2020/10/berlin-map.png">
<picture>
<source type="image/webp" srcset="/content/2020/10/berlin-map.1024.webp" />
<img src="/content/2020/10/berlin-map.png" />
</picture>
</a>
</figure>
<p>Flying in Germany was long on my to-do list, I was simply intimidated: What were the rules? How would I find a school and instructor? How would I even get there from central Berlin?</p>
<p>The research for most of the effort wasn’t difficult, though it helps a lot that I can read German at an elementary level. The biggest problem was simply identifying which outlying airfield was most promising, and writing emails until I got a response.</p>
<p>There are, frankly, not a lot of options for flight schools in the vicinity of Berlin. Strausberg Airport had the <a href="https://www.flugplatz-strausberg.de/?path=content/en/flugschulen.php&nv=sub4&lang=en">best-presented “flight schools” page</a>, so it was instantly a favorite. Aerotours at Strausberg also was the first to write back to me — by a full week! So after a couple of tries of explaining what I wanted — just an adventure flight, I don’t have to be Pilot in Command, thanks! — I got on their schedule with my choice of aircraft — a G-1000 equipped Cessna 172SP, exactly the same year and equippage as I had recently flown in Florida.</p>
<figure>
<a href="/content/2020/10/berlin-strausberg-flughafen.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/10/berlin-strausberg-flughafen.1024.webp" />
<img src="/content/2020/10/berlin-strausberg-flughafen.1024.png" alt="The Strausberg airport, northeast of Berlin" />
</picture>
</a>
<figcaption>The Strausberg airport, northeast of Berlin</figcaption>
</figure>
<p>After two weeks of working in Berlin, the scheduled day arrived, bringing beautiful weather. I was lucky enough to find a colleague who was in for an adventure one spring afternoon, and she drove us northeast out of Berlin to the ostrich-bedecked <a href="https://en.wikipedia.org/wiki/Strausberg">Strausberg</a> airport. (Strauß being the German word for ostrich, which is <em>not</em> where the city gained its name, but it’s a homonym and it’s fun)</p>
<figure>
<a href="/content/2020/10/berlin-ostrich-statue.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/10/berlin-ostrich-statue.1024.webp" />
<img src="/content/2020/10/berlin-ostrich-statue.1024.png" alt="Strauß = Ostrich, so the homonym/joke is Ostrich-hill" />
</picture>
</a>
<figcaption>Strauß = Ostrich, so the homonym/joke is Ostrich-hill</figcaption>
</figure>
<p>Strauserg airport is about the size of most of the airports that dot Arizona’s landscape, and laid out very similarly. On the way, my colleague informed me that a lot of the activity at this airport involved sailplanes, including motor-gliders. That was perhaps the most stark difference: There certainly were quite a few such aircraft positioned around the field, an uncommon sight here, as most of our gliders are at specific gliderports.</p>
<figure>
<a href="/content/2020/10/berlin-gliders.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/10/berlin-gliders.1024.webp" />
<img src="/content/2020/10/berlin-gliders.1024.png" alt="Motorgliders; Stemme has a factory on the field here" />
</picture>
</a>
<figcaption>Motorgliders; Stemme has a factory on the field here</figcaption>
</figure>
<p>Checking in was straightforward, probably in no small part because having an American come through for an adventure flight isn’t apparently a common occurrence. My instructor for the flight trained at Embry-Riddle in Florida, and was excited to show me how cool it is to fly in Germany, and how much of Berlin we could see by air.</p>
<p>This flight school was also not a through-the-fence building, so we took the short walk across the airport’s parking and through a pedestrian gate. On the ramp waited our Skyhawk.</p>
<p>I began a normal preflight, and gave the instructor opportunity to make remarks about my technique. Overall he said he rarely flew with people who were current and knowledgeable about the specific type – but it really is just a simpler version of the Skylanes I fly regularly in my club. The aircraft was in exquisite shape, particularly compared to the same-year 172SP I rented in Florida. But one thing was missing - there was nothing in the aircraft I could find to sump the fuel.</p>
<p>We <a href="https://www.aopa.org/training-and-safety/students/presolo/skills/checking-fuel-samples">drain a little fuel from each fuel tank during pre-flight operations</a> to ensure that we’re carrying the right fuel, and that there’s no water mixed into it. Aircraft fuels are lighter than water, so water would gather at the bottom of the tank, and aviation gas is colored light blue to distinguish it from petrol and jet fuel.</p>
<p>But, I find out, I can’t do that. At least not with this flight school — the fuel, being leaded, had to be handled by trained technicians. They already tested the fuel tanks, and I’d be fine.</p>
<p>Well, that was a new experience. One thing I’ve learned as a pilot is how to take ownership of complex risks and mitigate them with knowledge, training, and technique. I’ve always sumped the fuel anytime I didn’t know for sure the status of the fuel tanks. But not this day, this day I had to trust the staff. Okay.</p>
<p>Preflight — and the preflight paperwork — completed, the flight instructor called up a clearance delivery frequency for departure clearance in German. I followed about half of the exchange, and then we were clear to taxi and depart, with a brief exchange with the Strausberg airport radio. I handled the runup to the instructor’s satisfaction, and we went aloft.</p>
<figure>
<a href="/content/2020/10/berlin-strausberg-town.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/10/berlin-strausberg-town.1024.webp" />
<img src="/content/2020/10/berlin-strausberg-town.1024.png" alt="The town of Strausberg" />
</picture>
</a>
<figcaption>The town of Strausberg</figcaption>
</figure>
<p>We flew over the old town of Strausberg and southwest toward Berlin, in touch with Berlin Approach. About this time, the instructor realized he was going to have to translate instructions for me, apologizing: apparently we were now logged as being a German-speaking flight due to the initial call-up. If the initial clearance delivery had been in English, all our controllers would be using English for us. But no worries for me, I let him do the talking and I did the flying.</p>
<p>Communicating with Berlin Approach worked a lot like the <a href="/2020/09/30/returning-home/">Eglin Flight Rules Area</a>, in that it was VFR with constant-contact within the vicinity of the city, but as far as I could tell, we could fly anywhere but over the federal buildings in the middle of the city. So naturally, we first flew over the Mozilla office, crossing the Spree river just west of the office to get a good view.</p>
<figure>
<a href="/content/2020/10/berlin-crossing-spree.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/10/berlin-crossing-spree.1024.webp" />
<img src="/content/2020/10/berlin-crossing-spree.1024.png" alt="Crossing the Spree river in eastern Berlin" />
</picture>
</a>
<figcaption>Crossing the Spree river in eastern Berlin</figcaption>
</figure>
<p>That turn by the office set us up for a fly-by of <a href="https://en.wikipedia.org/wiki/Berlin_Tempelhof_Airport">Templehof field</a>, one of the primary destinations of the Berlin Airlift. I grew up with pilots around Eglin Air Force Base who had flown in the Berlin Airlift, and interviewed some of them while in high school.</p>
<figure>
<a href="/content/2020/10/berlin-templehof.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/10/berlin-templehof.1024.webp" />
<img src="/content/2020/10/berlin-templehof.1024.png" alt="Historic Berlin Templehof field" />
</picture>
</a>
<figcaption>Historic Berlin Templehof field</figcaption>
</figure>
<p>Templehof is especially interesting to see today, as it’s now a large city park – but unlike so many airports converted that way, the grounds were kept essentially unchanged. It still has its two parallel runways, just now with more skateboarders.</p>
<p>Our next destination was <a href="https://en.wikipedia.org/wiki/Berlin_Sch%C3%B6nefeld_Airport">Berlin-Schönefeld</a>, to request a practice instrument “low approach.” Since every flying thing gets billed to you in an itemized fashion in Germany, it’s much cheaper to avoid touching the wheels to the pavement, hence the low approach.</p>
<p>We prepared for the approach, briefing the ILS chart and gathering the Schönefeld weather information, and then my instructor requested the practice approach. There was some back-and-forth, none of which I caught, and he informed me that, sorry, due to the air show that they weren’t open for approaches today.</p>
<p>I made a mental note about that whole air show business and we turned back toward Strausberg.</p>
<figure>
<a href="/content/2020/10/berlin-east-of-erkner.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/10/berlin-east-of-erkner.1024.webp" />
<img src="/content/2020/10/berlin-east-of-erkner.1024.png" />
</picture>
</a>
</figure>
<p>There’s a lot of lakes, small towns, and windmills east of Berlin, so we settled into a scenic cruise back to the field. Chatting with the instructor, I learned some more differences with flying in the United States. One was some clarification that if it isn’t printed in the Pilot’s Operating Handbook for the aircraft, we shouldn’t do it. For example, the 172SP handbook only lists short and soft-field takeoff performance, so I should have selected short-field takeoff flaps, though they were clearly unnecessary. Rules are rules, after all.</p>
<p>My favorite was finding that it’s not permitted to land at an airport unless the airport is attended. (Unless you’re experiencing an emergency, of course). I asked, does that mean all airports are towered? No, apparently. Some airports simply have a radio in the restaurant, and someone will answer when called to confirm the airport is open and observe field conditions. They don’t provide <em>clearances</em>, so it’s still up to the pilot to provide separation, but they do get to confirm that the airport exists in both space and time, and has not fallen into a parallel dimension.</p>
<p>What happens if they don’t answer? Airport is closed, apparently.</p>
<p>This made me think of all the times it’s taken me more than 5 minutes to contact remote western Flight Service stations to read out <a href="https://www.faasafety.gov/spans/noticeView.aspx?nid=7155">pireps</a>. I imagine pilots waiting on the radio operator’s coffee pot to finish brewing is likely the same in either country.</p>
<p>We talked about my intentions to get my commercial certificate in the future, and we used that as an excuse to practice a <a href="https://www.boldmethod.com/learn-to-fly/maneuvers/why-every-pilot-should-practice-power-off-180-landing-technique/">couple of power-off 180 precision landings there</a>; at that point I hadn’t flown that commercial maneuver before, so his was my first instruction on it. My landings were fine, if imprecise – but then again, a Cessna 172 doesn’t drop like a rock like a Cessna 182 with its prop full forward. I am happy to say I’ve much improved my power-off 180 technique in the years since.</p>
<figure>
<a href="/content/2020/10/berlin-strausberg-ramp.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/10/berlin-strausberg-ramp.1024.webp" />
<img src="/content/2020/10/berlin-strausberg-ramp.1024.png" alt="The ramp at Strausberg" />
</picture>
</a>
<figcaption>The ramp at Strausberg</figcaption>
</figure>
<p>Shutting down, the aircraft was almost immediately taken over by staff members who wheeled it away for careful refueling and cleanup for its next flight. Such service!</p>
<figure>
<a href="/content/2020/10/berlin-pilots.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/10/berlin-pilots.1024.webp" />
<img src="/content/2020/10/berlin-pilots.1024.png" alt="My colleague and myself after the flight" />
</picture>
</a>
<figcaption>My colleague and myself after the flight</figcaption>
</figure>
<p>In the time since this flight, my colleague finished her single-engine private pilot’s certificate there in Germany.</p>
<figure>
<a href="/content/2020/10/berlin-strauss-jc.jpg">
<picture>
<source type="image/webp" srcset="/content/2020/10/berlin-strauss-jc.1024.webp" />
<img src="/content/2020/10/berlin-strauss-jc.1024.png" alt="Author posing with the airport's ostrich" />
</picture>
</a>
<figcaption>Author posing with the airport's ostrich</figcaption>
</figure>
<p>I got a fantastic story to tell, and another opportunity to learn more about both <a href="/tag/aviation">flying</a> and about Germany.</p>
<p>When it was all over, I had to remind the flight school I needed to pay them. It wasn’t cheap, but it sure was valuable. Also, I found out that the Berlin Air Show was the next weekend, which I absolutely attended: That’s another story.</p>I have had business dealings requiring me to visit Berlin regularly for the last 5 years, and in 2018 I finally made the trip outside of the city to Strausberg Airport to rent a Cessna 172SP Skyhawk.