» ironclad's history

29 December A.D. 2014 @ 7:32 PM

Like a lot of Common Lisp packages that I've written, Ironclad was inspired by somebody else. The following is my recollection of the history of Ironclad.

In one of the perennial debates on comp.lang.lisp about whether Common Lisp was a decent large for writing high-performance code, Pierre R. Mai posted an implementation of MD5 written for CMUCL that was at least competitive with implementations written in C. (This was over a decade ago and MD5 was still on the cusp of being secure.) Hash algorithms were one place where people thought Common Lisp couldn't compete with C—the requirements for efficient, unboxed arithmetic were thought to be beyond the reach of Common Lisp implementations.

Given the close history of SBCL and CMUCL, Pierre's implementation was quickly made available as an addon in SBCL's contrib/ directory. Intrepid SBCL hackers Christophe Rhodes and Alexey Dejneka also figured out how to compile portable Common Lisp implementations of modulo-2^32 arithmetic into efficient machine code. Obviously, the portable implementation wasn't going to be equally efficient on all implementations, but it was a good starting point.

Being in college, and needing something to avoid doing work for my classes, I started writing an implementation of SHA-1 that could be added as a contrib to SBCL. It was rejected, for reasons that I can't recall, but seemed reasonable at the time. However, it also set me to thinking: it would be silly to have separate MD5, SHA-1, SHA-2, SHA-512, etc. packages; it would be difficult to swap out implementations if you needed to, and each package was likely to have slightly different naming conventions, calling conventions, etc. etc. What you really wanted was a common interface for all of them.

And thus was the first release of Ironclad conceived.

I will not detail here the exact path by which bits were added to Ironclad. Hash algorithms came first, and then the big names in encryption algorithms; the AES competition was taking place around this time, so I added a few of the promising candidates from that competition. Basic, functional, insecure versions of RSA were added. There wasn't any grand plan to what algorithms were chosen: anytime I felt that my studies were too tedious is when something tended to be added to Ironclad.

Various refactorings took place along the way. The encryption algorithms and encryption modes had been implemented with macros masquerading as C++ templates and as such, took quite a long time to compile when changes were made. Changing them to better utilize generic function dispatch improved compilation and load time while maintaining performance. I distinctly remember getting frustrated when several bugs were reported in hash algorithms and having to change several different copies of code multiple times. I had cut-and-pasted code because I wasn't convinced that proper refactorings could provide the same performance, but the maintenance burden convinced me to do some benchmarking, and it turned out I had been wrong.

The best refactoring took place when I was writing non-Ironclad code and realized that I would really like to read and write integers of different endiannesses. Ironclad had this capability, of course, but it seemed silly to pull all of Ironclad in for this one bit of functionality. Thus was nibbles split out as a separate library, and slowly it gained its own set of improvements (inline assembly for SBCL, for instance), which in turn improved Ironclad as well.

I work significantly less on Ironclad than I used to. I still try to answer questions and investigate bug reports filed on Github, and an occasional week of nights spent hacking will produce something useful. But personal factors (e.g. no longer being in college) have lessened my motivation to write Ironclad particularly and Common Lisp generally.

There have also been numerous changes in the cryptographic landscape over Ironclad's lifetime. Increasing emphasis has been placed on not merely having secure algorithms, but implementing them in a secure way. Ironclad blatantly ignores much of the received wisdom about implementing algorithms securely, and a thorough audit would turn up many rookie mistakes. (See, for instance, the list of best practices at cryptocoding.net.) Not to mention that some of the techniques for implementing algorithms securely would be well-nigh impossible to implement in portable Common Lisp (bit-masking of raw pointer values comes to mind). I don't have any good solutions to this problem in general; I suppose you could write appropriately unportable code for those implementations that expose the necessary bits, and provide wrappers to known-good crypto implementations in C. But half the fun of Ironclad was doing all of this in Common Lisp, rather than writing FFI code.

The public-key algorithm implementations are particularly bad; improving those would be a lot of work. All the cryptographic tricks alluded to above are likely required, and then there's also things like writing your own bignum library, since implementations's bignums were likely not written with cryptographic applications in mind. I simply do not have the inclination to track down all the relevant standards (for padding messages and the like), academic papers (for telling you how not to do things), and reading other crypto implementations (for showing you how things might be done properly, if you can puzzle out the why from the lack of comments) to do the requisite work.

However, it is clear that Ironclad has been useful despite its many flaws in this area; the patches and the well-wishes I have received over the years are abundant evidence of this. If people wanted to improve Ironclad, the following suggestions may prove useful:

» good politics

23 April A.D. 2014 @ 1:17 PM

But [John McCone] was absolutely right about [South Vietnamese president Ngo Dinh Diem]'s overthrow. The most liberal members of Kennedy's team were the most single-minded advocates of the coup. They considered Diem a retrograde figure who was blocking the emergence of modern politics in South Vietnam and making it harder for the United States to prevail there. Even though the coup ended in bloodshed and murder, John Kenneth Galbraith (who had wanted to get rid of Diem for years) wrote Averell Harriman to praise the affair as “another great feather in your cap.”

Thirty years later it was liberals who pressed Bill Clinton hardest to stop the genocidal Balkan wars of the 1990s. With the cold War over, the “beast” of right-wing anti-Communism had largely fallen silent as a factor in foreign policy debate. The most vocal supports of what came to be called “humanitarian intervention” were instead human rights advocacy groups, international relief organizations, even the media. Madeleine Albright, ambassador to the U.N. at the time, spoke for them when she challenged Colin Powell, then the chairman of the Join chiefs of Staff, “What's the point,” Albright asked Powell across the table at the White House, “of having this superb military you're always talking about if we can't use it?”

Liberals, in short, have not always been on the defensive when it come to the use of American power. Nor have conservatives always treated foreign policy downsizing as weak and unpatriotic. Playing the public's interest in “peace” has been a prominent part of every single Republican administration's political strategy since the 1970s. Richard Nixon led the way in such maneuvering. His opening to China, he exulted, would be “good to hit the Democrats with at primary time.” Running for reelection two decades later, George H.W. Bush seemed almost embarrassed by his foreign policy accomplishments. The congressional republicans who opposed Barack Obama over Syria in September 2013 were not the first to think doing less might be good politics.

Maximalist: America in the World from Truman to Obama by Stephen Sestanovich

» around the sun

9 May A.D. 2013 @ 7:23 PM

My surprise reached a climax, however, when I found incidentally that he was ignorant of the Copernican Theory and of the composition of the Solar System. That any civilized human being in this nineteenth century should not be aware that the earth travelled round the sun appeared to me to be such an extraordinary fact that I could hardly realize it.

”You appear to be astonished,” he said, smiling at my expression of surprise. “Now that I do know it I shall do my best to forget it.”

“To forget it!”

”You see,” he explained, “I consider that a man's brain originally is like a little empty attic, and you have to stock it with such furniture as you choose. A fool takes in all the lumber of every sort that he comes across, so that the knowledge which might be useful to him gets crowded out, or at best is jumbled up with a lot of other things, so that he has a difficulty in laying his hands upon it. Now the skilful workman is very careful indeed as to what he takes into his brain-attic. He will have nothing but the tools which may help him in doing his work, but of these he has a large assortment, and all in the most perfect order. It is a mistake to think that that little room has elastic walls and can distend to any extent. Depend upon it there comes a time when for every addition of knowledge you forget something that you knew before. It is of the highest importance, therefore, not to have useless facts elbowing out the useful ones.”

”But the Solar System!” I protested.

”What the deuce is it to me?” he interrupted impatiently: “you say that we go round the sun. If we went round the moon it would not make a pennyworth of difference to me or to my work.”

—from A Study in Scarlet by Sir Arthur Conan Doyle

I had read a fair number of Sherlock Holmes stories before seeing the recent reboots in the movies and on the BBC. And after going back and reading the stories, both for the second time and first time, I am continually impressed with how many small details they have worked in and how faithful they have been to the original stories.

» contemporary church life

7 May A.D. 2013 @ 7:02 PM

It is one of the remarkable features of contemporary church life that so many are attempting to heal the church by tinkering with its structures, its services, its public face. This is clear evidence that modernity has successfully palmed off one of its great deceits on us, convincing us that God himself is secondary to organization and image, that the church's health lies in its flow charts, its convenience, and its offerings rather than in its inner life, its spiritual authenticity, the toughness of its moral intentions, its understanding of what it means to have God's Word in this world. Those who do not see this are out of touch with the deep realities of life, mistaking changes on the surface for changes in the deep waters that flow beneath. An inspired group of marketers might find a way of reviving a flagging business by modifying its image and offerings, but the matters of the heart, the matters of God, are not susceptible to such cosmetic alteration. The world's business and God's business are two different things.

—from God in the Wasteland: The Reality of Truth in a World of Fading Dreams by David F. Wells

» industrial waste

28 March A.D. 2013 @ 8:51 PM

Like many Americans, I had come to consider the hundreds of millions of tons of municipal solid waste produced annually as an indicator the “the throwaway society.” Then, ten years into my study of solid waste, I stumbled on a waste statistic quietly put out by the EPA in a document called Guide for Industrial Waste Management (U.S. EPA 1999). This technical manual, meant to provide tips to factory managers for handling waste at their plants, noted, without further comment, that manufacturing industries were generating some 7.6 billions tons a year of solid waste. Some digging on my part uncovered an older, unpublished report that was the source of this estimate as well as two follow-up government documents that cited other industrial, mining, extractive, and agricultural operations as bringing the total industrial waste tonnage generated in the United States up to around 12 billion tons (U.S. EPA 1987, 1988; OTA 1992). These amounts were an order of magnitude greater than the tonnage of municipal solid waste that every book, volunteer effort, government program, or household conversation about trash and its problems seemed to focus on. Yet very little had been published about this far larger quantity.

—from Recycling Reconsidered: The Present Failure and Future Promise of Environmental Action in the United States by Samantha McBride

» religious ritual

24 March A.D. 2013 @ 1:52 PM

Religious ritual, which seems so idiotic to the secular mind, has the same feel as Carl's nth chance to go straight or Phil's nth repetition of Groundhog Day. The words and motions of the mass give the faithful repeated chances to get it right. At the nth repetition of “this is my blood, the cup of salvation” you for the first time grasp, really grasp, the meaning of redemption through Christ's sacrifice. Well...part of it at any rate.

—from The Bourgeois Virtues by Deirdre N. McCloskey

» an obviously grotesque child

9 March A.D. 2013 @ 8:43 PM

A child like Mary Ann, [Flannery O'Connor] observed, is obviously grotesque, and in the modern world such a child is thought to “discredit the goodness of God.” How can a good God allow such a child to die? the Ivan Karamazovs of the world ask. How, moreover, can a good God allow such a child to be born? The modern unbeliever prides himself on his realism, his willingnness to recognize suffering and to ponder the problem of evil directly. But in O'Connor's estimation such an outlook is not realistic; is it naive, sentimental, and even dangerous. It is the believer, not the unbeliever, who is the realist. In a child like Mary Ann, the believer sees the likeness of every human person—deformed, limited, imperfect. In human deformity the believer sees “the raw material of good.” In human suffering the believer sees the grounds of our ommon humanity, recognizing that it is through suffering, above all, that human beings are stirred to the love of one another, and to the love of God, who showed his love for humanity through his willingness to suffer as one of us.

—from The Life You Save May Be Your Own by Paul Elie

» idolizing the body

17 February A.D. 2013 @ 9:03 PM

...contemporary medical Gnosticism seemingly idolizes the body, but primarily as an expression of the mind's (or the will') quest for perfection or permanence. The body is altered almost at whim, reinforcing its role as the malleable—and someday, perhaps, fully replaceable—envelope for something far more real and pure. Some reshape the body to fit a desired image, while others eeks endless fixes to keep thmselves alive. Even the reistance among some Christians to withdraw futile mechanical support from a dying relative can be a form of Gnosticism, valuing the ability to control and manipulate the body over the mysterious gift of an embodied life—a gift that was never actually ours to keep.

—from Reclaiming the Body by Joel Shuman and Brian Volck

» new software releases

14 January A.D. 2013 @ 10:55 PM

Updated versions of ironclad (0.32.1), nibbles (0.11) and chipz (0.8) have been released. The usual crop of bugfixes (Gray streams in ironclad and chipz, among others) and new features (float accessors in nibbles) are present, as well as compatibility with ASDF 2.27.

» helping one another

5 January A.D. 2013 @ 3:41 PM

Consequently, as a professorial leader who is interested in enhancing the performance of my educational organization (which includes me) by reducing or eliminating the anaclitic depression blues, I write the students in my introductory class a letter. I have found that I have to present my thoughts in writing so that students can assimilate the content at their convenience; they simply can't seem to understand what I say if I profess my point of view orally.

My letter reads as follows: “You may take examinations alone, with another person, or with as many other people as you like. 'Other people' includes classmates, parents, children, spouses, students from other classes, professors or 'hired guns.' I go absolutely blind with rage if I catch anyone cheating. I define cheating as the failure to assist others on the exams if they request it” (Harvey, 1997a)

How do you think our dean reacted when one of my outraged (and terrified) colleagues, apparently in an effort to avoid suffering from the anaclitic depression blues, showed him the letter? For starters, he invited me to his office for “a little discussion.”...

He burst forth in a voice powerful enough to dislodge the green eyeshades from the furrowed brows of my beloved colleagues ensconced in the deep recesses of the accounting department, “Professor Harvey, are you aware of the absolute chaos that would be generated at the George Washington University if everyone began to help one another?”

Are you aware of the absolute chaos that would be generated at The George Washington University if everyone began to help one another?

What an extraordinarily relevant question for someone in a leadership role to ask—not only about The George Washington University but also about any other organization. To his everlasting credit, though, the dean immediately followed up his pithy query with another that was equally, if not more, poignant in nature.

“Professor Harvey, did I just say what I think I said?” he asked.

“I'm pretty sure you did,” I replied.

—from How Come Every Time I Get Stabbed in the Back, My Fingerprints Are on the Knife? by Jerry B. Harvey

» commentary on job

20 December A.D. 2012 @ 10:33 PM

What was it that Job “saw” when God spoke to him out of the whirlwind that he had previously only “heard by hearing of the ear,” so that he despised himself and repented in dust and ashes (Job 42:5-6)? Job has persistently held God to account in his protests over against his “comforters,” who tried to exonerate God by their “theodicies.” Job's friends thought his speech laying the responsibility on God was outrageous and blasphemous, but Job insisted on crying out against God since God is, according to “the hearing of the ear” (perhaps we might the say “The Doctrine of God!”), the one who is supposed to be in charge. Now God, in declaring his awesome and universal majesty out of the whirlwind, actually approves what Job had said over against all the explanation of the “theologians.” So God declares (42:7-9) that Job had spoken the truth, terrifying as it was and is. Job now sees that in the voice of his suffering he had unwittingly spoken the truth, and he is terrified by it: “I have uttered what I did not understand, things too wonderful for me which I did not know” (42:3). Job sees that through suffering the truth had literally been wrung out of him. He sees where previously he had heard and complained. He thus “despises himself and repents in dust and ashes.”

—from On Being A Theologian of the Cross by Gerhard O. Forde

» schwarzenegger's california

19 December A.D. 2012 @ 11:10 AM

[Schwarzenegger's] view of his seven years trying to run the state of California, like the views of his closest associates, can be summarized as follows. He came to power accidentally, but not without ideas about what he wanted to do. At his core he thought government had become more problem than solution: an institution run less for the benefit of the people than for the benefit of politicians and other public employees. He behaved pretty much as Americans seem to imagine the ideal politician should behave: he made bold decisions without looking at polls; he didn't sell favors; he treated his opponents fairly; he was quick to acknowledge his mistakes and learn from them, and so on. He was the rare elected official who believed, with some reason, that he had nothing to lose, and behaved accordingly. When presented with the chance to pursue an agenda that violated his own narrow political self-interest for the sake of the public interest, he tended to leap at it. “There were a lot of times when we said, 'You just can't do that,'” says his former chief of staff, Susan Kennedy, a lifelong Democrat, whose hiring was one of those things a Republican governor was not supposed to do. “He was always like, 'I don't care.' Ninety percent of the time it was a good thing.”

Two years into his tenure, in mid-2005, he'd tried everything he could think of to persuade individual California state legislators to vote against the short-term desires of their constituents for the greater long-term good of all. “To me there were shocking moments,” he says. Having sped past a DO NOT ENTER sign, we are now flying through intersections without pausing. I can't help but notice that, if we weren't breaking the law by going the wrong way down a one-way street, we be breaking the law by running stop signs. “When you want to do pension reform for the prison guards,” he says, “and all of a sudden the Republicans are all lined up against you. It was really incredible and it happened over and over: people would say to me, 'Yes, this is the best idea! I would love to vote for it! But if I vote for it some interest group is going to be angry with me, so I won't do it.' I couldn't believe people could actually say that. You have soldiers dying in Iraq and Afghanistan, and they didn't want to risk their political lives by doing the right thing.”

—from Boomerang by Michael Lewis

» new archive release

16 December A.D. 2012 @ 2:56 PM

I've released archive 0.9; it can be found in the usual place. Notable in this release is better handling of directories, both in extracting them and forming archives with them.

» matching sexps in emacs lisp

11 December A.D. 2012 @ 8:01 AM

Occasionally, you come across a piece of code that looks quite complicated. And then you realize that the code is really overcomplicating the situation, not that the task is inherently difficult. Today's specimen:

(defun icalendar--convert-block-to-ical (nonmarker entry-main)
  "Convert block diary entry to iCalendar format.
NONMARKER is a regular expression matching the start of non-marking
entries.  ENTRY-MAIN is the first line of the diary entry."
  (if (string-match (concat nonmarker
                            "%%(diary-block \([^ /]+[ /]+[^ /]+[ /]+[^ ]+\)"
                            " +\([^ /]+[ /]+[^ /]+[ /]+[^ ]+\))\s-*"
                            "\s-*\(.*?\) ?$")

All of the line noise after diary-block is manually matching subexpressions of a sexp with regular expressions. Words fail me.

» exact float conversion

28 November A.D. 2012 @ 10:11 PM

Somebody at work asked how to determine whether an integer (assuming to be 32-bit) was exactly convertable into a (IEEE single) float. There's the obvious:

exactly_convertable_p (int32_t x)
  float f = x;
  int y = f;
  return x == y;

but it's more fun to reason things out from first principles. The __builtin_ctz call below is GCC-specific, but it should be straightforward to write your own ctz function:

exactly_convertable_p (int32_t x)
  /* The easy cases: an exponent of 0 and mantissa of the integer.  */
  if ((-1 << 24) <= x && x <= (1 << 24))
    return true;

  /* Count trailing zeros and see if we can use a non-zero exponent.  */
  int first_low_bit = __builtin_ctz(x);
  if (first_low_bit == 0) {
    return false;

  int32_t shifted = x >> first_low_bit;
  return (-1 << 24) <= shifted && shifted <= (1 << 24);

It should be fairly obvious how to extend this to 64-bit integers and IEEE double floats. Bonus points for doing a templated solution that works for given integer and float types.