Every few years, international political tensions disrupt the global supply of rare earths, and every time this results in a scramble to find more ore or to build refineries faster.¹ China controls around 70% of rare-earth processing, but even if you solved that bottleneck tomorrow, you would still have a problem: rare earths are not interchangeable commodities.

Neodymium, for example, is central in permanent magnets, which drive wind turbines and electric motors. The problem is that this element cannot be swapped with a different element, because that would force you to redesign the corresponding alloy. A different alloy needs different processing. Different processing produces a different structure, different magnetic properties, and different performance in the final device. One substitution, and the whole application doesn’t work anymore. Why?

The cascade from composition through processing through structure to performance is not specific to magnets. All materials work this way. We tend to think of a material as a property on a datasheet, but it is really a web of cause-and-effect relationships that goes from atoms to applications. There is a framework that maps these relationships as a system: process-structure-properties-performance, or PSPP. The rest of this post is about what this framework looks like and where it comes from.

# What is a material?

Before exploring how systems thinking enters materials science, I think it helps to be precise about what a material is. However, when I turned to standard textbooks looking for a definition, I couldn’t find one.^{2 3} They classify materials, describe their properties, discuss their history, but never say what the word actually means.

So I searched further for a working definition of “material”, and to my surprise, I found that defining the term is not at all straightforward.^{4 5 6} As I understand the problem, three factors make this harder than I initially assumed:

1. The word “material” is treated as a primitive — a word everyone understands well enough that it doesn’t need a formal definition.
2. The field is interdisciplinary, and no single discipline has felt the need to own the definition.
3. There is no clear boundary between a “chemical” and a “material.” The distinction depends on length scale, but where do you draw the line? Is a single molecule of polyethylene a material? What about a monolayer or a colloidal suspension?⁷

Eventually I found two definitions that I could work with. In Materials Chemistry, Fahlman defines a material as:⁸

Nature Materials gives a similar definition, but with a broader scope:⁹

These definitions, together with the broader discussion cited earlier, point to three features that seem to be common to all materials:

1. They are condensed-state substances.
2. Their properties emerge from the interactions, assembly, and defect structure of their constituent subunits.
3. They are created or designed for a specific purpose.

In other words, a material is not raw matter sitting in the ground: it is matter that someone has processed with an end in mind. The definition itself encodes a relationship between processing, structure, and purpose, and this entanglement is not just a matter of semantics, because the concept cannot be reduced to a single property or a single scale. If the definition is this entangled, perhaps the right answer is not a sharper definition but a better framework to think about materials.

# Three questions and a chain

The study of materials is often divided into three disciplines: materials science, materials engineering, and materials design. However, this division is cleaner on paper than in practice. A more useful way to think about them is as three questions, each building on the last.

The first question is why. Why does this material behave the way it does? This is materials science, which investigates the relationships between a material’s structure and its properties. A materials scientist synthesizes a new material and asks what gives it its characteristics.

The second question is how. How do we make a material behave the way we want? This is materials engineering: designing or manipulating the structure to produce a predetermined set of properties. Where the scientist asks why something works, the engineer asks how to produce it and make it work differently.

The third question is which. Which material and which process satisfy a specific need? The answer takes different forms depending on the starting point:

• Materials discovery screens all possibilities.
• Materials design explores a specific region of the design space, often adding new materials or properties to it.
• Materials selection picks the best candidate from a space that is already known.

All three sit close to the application and depend on the answers to the other two questions.

A material’s properties depend on the spatial organization of its constituents at every hierarchical level, from atomic arrangement up to macroscopic form. But structure is not static. At any moment, a material carries structural features that have not yet reached thermodynamic equilibrium. These features reflect not only what it is made of, but how it was made and what it has endured in service. Processing stays with the material long after it is produced.

This causal chain — processing determines structure, structure determines properties, properties determine performance — is the central paradigm of materials science and engineering, and it is known as process-structure-properties-performance (PSPP).^{10 11} The diagram below shows how it works:

Materials science typically works the chain from left to right: you start with a process and ask what properties come out. This is the forward problem. You pick apart individual links and examine them in isolation, like how does this heat treatment change the grain size, and how does grain size affect hardness? Each link gets its own experiments, its own models.

Materials design goes the other way. You start with what you need the material to do and work backward: what properties would deliver that performance, what structures would produce those properties, and what processing would create those structures? This is the inverse problem, and it only becomes tractable if you can hold the full chain in view at once.

# Materials as systems

So far, PSPP looks like a causal chain where one thing leads to the next. But that picture is too simple. PSPP treats a material as a system, and that word carries specific meaning.

A system is an arrangement of elements, interconnections, and a function that work together to fulfill a purpose.¹² The elements can be tangible (atoms, grains, phases) or abstract (a model, a design constraint). The interconnections define how elements influence each other. The function is what the system is designed (or has evolved) to do.

A system’s behavior over time is tracked through stocks and flows. A stock is a measurable quantity that accumulates. Flows are the rates at which a stock changes: inflows add to it, outflows subtract. When inflows and outflows balance, the system reaches dynamic equilibrium. In a material, structure is the central stock. Every processing step — every heat treatment, every deformation pass, every hour in service — is a flow that deposits or removes structural features. The microstructure you observe at any moment is the cumulative record of everything the material has experienced.

Consistent behavior patterns in a system arise from feedback loops: circular relationships where the system’s output influences its own behavior. Materials are full of them. Grain growth during annealing slows as grains coarsen, because larger grains have less boundary energy driving further growth. Crack propagation accelerates as a crack lengthens, because the stress concentration at the tip increases with crack size. These loops (some stabilizing, some reinforcing) are what give materials their characteristic responses to time and environment.

What makes a system more than the sum of its parts is emergence: the interactions between components produce behaviors that no single component exhibits alone. A pile of iron atoms is not steel. The arrangement matters, and the properties that result from that arrangement cannot be deduced by studying iron and carbon in isolation.

A material is a hierarchically organized system composed of subsystems, each operating at a different length scale. At the atomic scale: bonding and crystal structure. At the microstructural scale: grain size, phase distribution, and defect populations. At the macroscopic scale: geometry, surface condition, and manufacturing history. Each level has its own characteristic structures and its own models, but none is self-contained, and changes at one scale propagate up and down the hierarchy.

# From Smith to systems engineering

The idea that materials behave as systems did not arrive all at once. In his cornerstone article on materials design, Olson traces the origins of this idea to the metallurgist and historian of science Cyril Stanley Smith, who was among the first to describe materials as hierarchical systems.¹³ Clarence Zener extended Smith’s picture by adding the path dimension: processing history. In any real material, some structural features have not had time to equilibrate. The system is never at rest, and it carries the memory of how it was made.

Morris Cohen then introduced the principle of reciprocity between structure and properties: properties can be rationalized in terms of structure, but structure can equally be rationalized in terms of the properties it produces. If the structure–property link runs both ways, then tools built to explain can also be turned around to design.

What was still missing was a way to integrate these insights into something operational. That came from systems engineering. The first step is analysis to identify the problem and decompose it into component subsystems and their interactions. This step is followed by synthesis: develop models for each subsystem, integrate them to generate candidate solutions, build prototypes, and test. Feedback from testing drives iterative redesign until the objectives are met. Applied to materials, each scale becomes a subsystem with its own models, its own experiments, and its own interface to the scales above and below.

# Reading the chain backward

For most of its history, materials development advanced through testing compositions, adjusting processes, and accumulating know-how. Serendipity played a significant role in this process, and still does.^{14 15} If we wanted a better magnet, we made magnets and tested them until something worked.

But if the links between processing, structure, properties, and performance are causal in both directions, then the chain can be read backward. Start with a need. Translate it into performance requirements, then into target properties, then into the structures that would produce them, and finally into the processing routes that would create those structures.

This changes what the rare-earth problem actually is. Without PSPP, a supply disruption is a procurement problem: find more ore, build more refineries, secure the chain. With PSPP, it is a design problem: identify which scales are coupled to the element you are losing, which feedback loops constrain substitution, and which processing routes are available for candidate replacements. Those are different questions, and they lead to different responses and different timescales.

The practical tools for carrying out this inversion — computational modeling, materials informatics, integrated computational materials engineering — are subjects for future posts. What matters here is the shift in how we ask the question.

Design of experiments (DOE) provides many tools for studying physical systems. All share the goal of efficiently identifying how specific variables (factors) control the properties (responses) of a system. Factorial designs and fractional factorial designs are two well-known approaches, but they assume that factors can be varied independently. In many chemical systems, this assumption fails.

Mixtures are one such case. The response depends on the relative proportions of the ingredients, not on their absolute quantities. This constraint applies to pharmaceutical formulations, glass development, polymer blends, food products — any system where the components' proportions must sum to 1. Mixture designs are built specifically for these systems.

# A simplex example

The following example, adapted from John Cornell’s “A Primer on Experiments with Mixtures,” shows what a simple mixture design and analysis looks like. Suppose you produce a polymer-based yarn used for draperies and want to improve its toughness.

Your current formulation is a blend of polyethylene (PE) and polypropylene (PP). A colleague suggests that substituting polypropylene with polystyrene (PS) might help. You want to know if and how this substitution affects the yarn’s mechanical properties, and you want an efficient way to find out.

# Planning and designing the experiments

First, you need to choose a property that you can measure — your response variable $y$. You decide to take a closer look at the elongation of fibers produced with different polymer blends, which measures how much you can lengthen your fibers before they break.¹

Next, you need to plan your experiments (in DOE terminology: you need to plan your experimental runs). How are you going to vary the proportions of PE, PP, and PS? You need to ensure that you cover the experimental region while minimizing the number of experiments that you’ll run.

Mixture designs are built for situations where the components must sum to a constant — typically 100% or 1. The experimental space is represented using simplex-based geometries (triangular for 3 components, tetrahedral for 4, etc.) and each experimental run is optimally placed inside the simplex, ensuring that the compositional space is properly explored.

The simplest mixture design is the Simplex Lattice Design (SLD). In this type of design, each factor (in this case, PE, PP, and PS) can take $m+1$ values ($0, 1/m, 2/m, \dots, m/m$) where $m$ represents the number of levels. A level is the specific value that a factor takes in a given run. The value of $m$ is also important for later modeling, because it defines the order of the polynomial that can be fitted to the data: $m=1$ means you’ll only be able to fit first order polynomials, $m=2$ second order, $m=3$ third order, and so on.

You decide to keep it simple and explore only pure or binary blend compositions, meaning that each factor is going to take two levels ($m=2$). This setup includes three points for each component: 1 for a single-component, 0.5 for the binary blend, and 0 for blends that do not include that component. This allows you to fit a second-order polynomial and see how pairs of components interact.

Calculating the proportions and plotting them, the design looks like this:

# Analyzing the results

You prepare the 6 polymer blends, spin them into yarns, and measure their elongation. Using shorthand notation — $x_{1}$ for polyethylene, $x_{2}$ for polystyrene, $x_{3}$ for polypropylene — and averaging three replicates per run, the results are:

At this point, you’ll want to fit this data into a model that can be used to extrapolate the behavior of the system. There is one problem, though. Consider a standard polynomial with an intercept $\beta_{0}$:

Where $\beta_{1}$, $\beta_{2}$, and $\beta_{3}$ are the regression coefficients of each component. In mixture space, when all components approach zero we have $x_{1} \rightarrow 0$, $x_{2} \rightarrow 0$, $x_{3} \rightarrow 0$. However, this violates the mixture constraint, since the sum of all components $\sum_{i} x_{i}$ must equal 1. There is no point in mixture space where all $x_{i} = 0$, so the intercept $\beta_{0}$ has no physical meaning.²

The solution is to use Scheffé polynomials, special polynomial forms that respect the mixture constraint. For a ternary mixture system, the second-order Scheffé polynomial is:

Here, $\beta_{1}$, $\beta_{2}$, and $\beta_{3}$ represent the expected response when component 1, 2, and 3 respectively comprise 100% of the mixture, whereas the terms $\beta_{12}$, $\beta_{13}$ and $\beta_{23}$ represent binary interactions. These latter terms have physical meaning and represent synergistic or antagonistic effects between components.

The fitted model becomes:

With six experiments, you now have a model that describes the elongation for any binary blend of the three polymers. Plotting the model over the simplex shows the full picture:

This type of visualization is called a response surface. It shows how a response (the elongation) changes across the entire compositional space. Each point within the triangle represents a unique three-component composition, while the distances from the three sides correspond to the proportions of each component. The contour lines show predicted elongation values, revealing trends that would not be obvious from the raw data alone.

The results show that your colleague’s intuition was partially right. The PE-PS combination has a strong synergistic effect: the binary blend achieves 15.3 elongation, 45% higher than the simple average of the pure components (10.55). But in absolute terms, the PE-PP blend still wins. The highest elongation values (16-17 range) occur along the PE-PP edge, with the maximum near the 50:50 PE:PP blend (16.9 from design point 6). PS-PP combinations show antagonistic behavior, particularly in the lower-right region of the triangle.

The model provides clear guidance for optimization. If maximum elongation is the primary goal, aim for compositions between 40-60% PE and 40-60% PP with minimal PS content. If other considerations require PS, keep it below 20% while maintaining high PE content to take advantage of the PE-PS synergy. The relatively flat response surface around the PE-PP optimum also suggests these formulations will be robust to minor mixing variations during production.

# Where to learn more

This example covered the simplest case. Several important topics were left out:

• Modeling ternary interactions.
• Designing experiments with more than three mixture components.
• Restricting the experimental region by adding further constraints.
• Coupling mixture designs with process factors.

The following resources cover these and more.

# Books

• “Experiments with Mixtures” by John Cornell (10.1002/9781118204221 ): The most thorough reference on mixture designs available. Published in 2002 and showing its age in some aspects, but still unmatched in depth. Assumes strong statistical foundations; some chapters are heavily mathematical.
• “A Primer on Experiments with Mixtures” by John Cornell (10.1002/9781118204221 ): Cornell’s shorter, more accessible version of the above. Covers much of the same content in a more direct way.
• “Formulation Simplified” by Mark Anderson, Patrick Whitcomb, and Martin Bezener (10.4324/9781315165578 ): A hands-on introduction aimed at practitioners who want to get started without heavy mathematical detail. Published 2018.
• “Response Surfaces, Mixtures, and Ridge Analyses” by George Box and Norman Draper (10.1002/0470072768 ): Covers advanced response surface modeling topics. Not limited to mixture designs, but useful for complex systems.
• “Design and Analysis of Experiments” by Douglas Montgomery (Link to publisher website ) and “Design and Analysis of Experiments” by Angela Dean, Daniel Voss, and Danel Draguljić (10.1007/978-3-319-52250-0 ): Both these books provide broad and thorough introductions to DOE fundamentals, from planning to designing and modeling. Among the various approaches, they also include short chapters on the basics of mixture experiments.

# Software

There are two main options for running mixture analyses, each with different tradeoffs between ease-of-use and cost:

1. A proprietary DOE program with a graphical interface.
2. R or Python with dedicated libraries.

# Commercial solutions

Several GUI-based DOE platforms support mixture designs:

• Minitab ,
• Design-Expert
• SPSS
• JMP
• SAS
• NCSS

These are GUI-based statistical platforms with built-in DOE functionality. Each has a learning curve, but they are generally straightforward to use once set up.

I avoid this kind of software for two reasons:

1. They’re expensive. For example, at the time of writing a single one-year license subscription for Design-Expert costs 1140 USD, while a single one-year license subscription for Minitab is priced at 1851 USD.
2. They provide black-box solutions. Being closed-source means you can’t inspect what a certain function in the program actually does. This might not be a concern for you now, but it might be in the future when you need to understand exactly how a result was obtained.

# Free coding alternatives

Everything these programs do can be replicated in code, with greater flexibility and full reproducibility. For mixture designs, the two main options are R (R Project ) and Python (Python.org ).

R was built for statistics; Python is a general-purpose language that has become a standard for data analysis. The tradeoff is straightforward: full control over the analysis, at the cost of learning to program. Both provide dedicated libraries that cover most of what commercial software offers.

• R libraries: R has a mature DOE ecosystem . For mixture work specifically, the mixexp package (documentation , JSS article ) handles design generation, analysis, and basic response surface plots. All plots and analyses in this post were done in R using mixexp and the Ternary package (documentation ). The R script used for this post is available.
• Python libraries: Python’s DOE ecosystem is less developed than R’s for mixture designs. The available options:
  • DoEgen : The most complete Python DOE library, actively developed.
  • dexpy : Based on Design-Expert, developed by Stat-Ease. Supports SLD, simplex-centroid designs, and D-optimal designs. Not updated since 2018.
  • ExperimentsPyDesign : Covers various experimental designs, though mixture support is limited to simple SLD.

The polymer yarn example used six runs to map an entire ternary compositional space. Traditional one-factor-at-a-time (OFAT) approaches would need far more experiments and still miss the interaction effects that drove the key findings.

CREST (Conformer-Rotamer Ensemble Sampling Tool) is a program for exploring molecular conformations. Developed by the Grimme group at the University of Bonn, it automates the search for low-energy conformers and rotamers, which is particularly useful for flexible molecules that can adopt many configurations.^{1 2}

The official CREST documentation provides clear installation instructions, but following them on a Mac with an M-series chip leads to problems. By default, cmake links against Apple’s Accelerate framework. On Arm-based Macs (M1, M2, etc.), this either fails or produces binaries that segfault at runtime. The solution is to compile against openblas and lapack instead of Accelerate.

# Building against openblas and lapack

Let’s start by getting the source code. Download the latest release of CREST following the instructions on the official documentation . Be sure to clone the repository and update the necessary submodules. This can be done by running git submodule update --init after git clone, as explained here (credit to Jingdan Chen for pointing this out).

Now we need to get the right libraries: cmake, gcc, openblas and lapack. Doing this with Homebrew is straightforward:

brew install gcc cmake openblas lapack

Next, we need to tell cmake where to find these libraries. This is where the official docs don’t work when applied on a Mac. CREST’s documentation suggests doing this with:

export FC=gfortran CC=gcc

On a Mac with an Arm chip, this resolves to macOS’s default compilers, which will link against Accelerate. Point to the Homebrew-installed compilers instead:

export FC=gfortran-14 CC=gcc-14

Now tell cmake where to find the lapack and openblas libraries you installed with Homebrew. Without this, cmake falls back to the macOS system libraries. Look for the path where Homebrew installed them, and run:

cmake -DCMAKE_PREFIX_PATH="/opt/homebrew/opt/lapack;/opt/homebrew/opt/openblas" -B _build

With everything configured correctly, you can finish the compilation:

make -C _build

This builds CREST in the _build directory, linked against openblas and lapack.

# Troubleshooting

If the build fails, check these first:

• Verify which version of gcc you have installed. Typing gfortran-14 when you have version 13 will produce confusing errors. Run brew info gcc to confirm.
• Confirm the paths to lapack and openblas match your Homebrew configuration. They may differ depending on how Homebrew is set up on your system.
• If you see errors about missing dependencies, make sure all submodules are initialized (git submodule update --init).

The entire problem comes down to which math libraries cmake links against. The error messages do not make this obvious, and I spent a couple of hours on it before finding the fix. Hopefully this saves you the same trouble.

The family consists of scandium, yttrium, and the 15 lanthanides, which sit between barium and hafnium on the periodic table. Identifying the remaining 16 would take more than a century after Gadolin’s work.^{2 3}

At the time, chemists were identifying new elements at a rapid pace. Rare earths were the most difficult to characterize. Extracted from minerals like yttria and ceria, they did not fit any of the patterns chemists had developed for other elements.

# Neither rare nor earths

The name “rare earths” reflects the assumptions of the chemists who first encountered these substances, not what we now know about them. When chemists first isolated them in the late 1700s, they obtained them as oxides, which were then called “earths” (from the French terre and German Erde).⁴ The minerals containing them, yttria and ceria, were known only from a single location, the village of Ytterby in Sweden, and their apparent scarcity there gave the family the other half of its name.^{5 6}

Rare earths are not actually rare. They are more abundant in Earth’s crust than mercury or cadmium. The difficulty is that they occur in trace concentrations across many minerals rather than in concentrated deposits, like a thousand euros in cents scattered across a beach instead of a single ten-euro bill in your pocket.

The real difficulty was not scarcity but chemical similarity. The 15 lanthanides have nearly identical electron configurations, which makes separating them like sorting a mix of dark blue and navy blue marbles by color. Specialized techniques are needed to tell them apart.

This chemical similarity also explains why yttria and ceria are not unique sources. The same elements were later found in monazite, a mineral that would become the main industrial source for rare earths. What seemed at first to be a Swedish curiosity is a family of elements distributed throughout Earth’s crust.

# First discoveries

Chemistry advanced rapidly in the late 18th and early 19th centuries. Boyle, Lavoisier, and Dalton established that the elements are the simplest substances, those that cannot be broken down further by chemical means. By the mid-19th century, nearly 70 of the 90 naturally occurring elements had been identified.⁴

The rare earths were the exception to this rapid progress. The two minerals containing them, yttria and ceria, had been identified by 1787, but the elements within them could not be reliably separated for more than 160 years, and the last of them, promethium, was not isolated until 1947. Even Dmitri Mendeleev and Julius Lothar Meyer, who built the first periodic tables in the 1860s, could not fit them into their classification. The puzzle was only resolved by the development of atomic theory in the 20th century, which showed why the lanthanides in particular have nearly identical chemical properties.

# Yttria

In 1787, Carl Axel Arrhenius, a Swedish army officer and amateur geologist, collected an unusual black mineral near the village of Ytterby and called it simply “black stone.”⁷ Swedish chemist Bengt Reinhold Geijer published the first report on it in 1788, but the decisive analysis came in 1794, when Finnish chemist Johan Gadolin found that nearly 38% of the stone consisted of an unknown “earth” (now known to be an oxide), along with iron and silicate.

In 1795, Swedish chemist Anders Ekeberg confirmed Gadolin’s findings and named the oxide “yttria” after Ytterby. Like most chemists at the time, Ekeberg treated the new “earth” as an element in its own right, a misconception that persisted until 1808, when Humphry Davy’s electrolysis experiments showed that yttria was a compound. The metallic element within it would later be isolated and named yttrium.

The black stone continued to yield results. In 1802, Ekeberg identified yttria in another mineral, yttrotantalite, and from it isolated tantalum. German chemist Martin Heinrich Klaproth and French chemist Louis Nicolas Vauquelin independently confirmed Gadolin’s analysis. Klaproth then named the original mineral “gadolinite” in his honor.

# Ceria

A second Swedish mineral, cerite, had been suspected of containing an unknown “earth” since the mid-1700s, but confirming it took decades. In 1803, two independent teams reported the result almost simultaneously. The Swedish chemists Jöns Jacob Berzelius and Wilhelm Hisinger were one, and the German chemist Martin Heinrich Klaproth, who had earlier confirmed Gadolin’s work on yttria, was the other. Both teams published their findings in the Neues Allgemeines Journal der Chemie in the same year.

A naming dispute followed. Klaproth proposed “ochroite earth,” while Berzelius and Hisinger proposed “ceria,” after the dwarf planet Ceres, which had been discovered two years earlier. Ceria won, largely due to Hisinger’s standing as an industrialist and patron of science, though most of the analytical work had likely been done by the young Berzelius.

The two teams interpreted the result differently. Klaproth treated the new “earth” as an element in its own right, the same misconception that had attached to yttria. Berzelius correctly identified it as the oxide of a new element, an interpretation consistent with the work that would later make him one of the founders of modern chemistry: precise atomic mass calculations, the formalization of atomic theory, and the modern system of chemical symbols. Klaproth, already known for the discovery of uranium and zirconium, continued his work on strontium, titanium, and tellurium.

The yttria and ceria discoveries set a pattern that would repeat throughout the 19th century: the rare earths rarely came alone, and characterizing them required collaboration across national boundaries.

# Mosander’s investigation

It took two decades for chemists to realize that the original “earths” were not pure elements but metal oxides. Ceria became cerium oxide, yttria became yttrium oxide. The reclassification was straightforward, but the trouble began with the atomic masses.

Measurements of the atomic masses of these supposedly pure elements varied between samples, which had no good explanation if the elements really were pure. The most plausible interpretation was that they were mixtures.

Carl Gustaf Mosander, assistant to Jöns Jacob Berzelius, took up the problem in the 1820s and worked on it for the next two decades. He began with cerium, subjecting it to a long series of reactions. In one experiment he heated cerium oxide with sodium and chlorine to obtain metallic cerium, and he noticed that cerium compounds varied in color and density depending on their mineral source. He concluded that cerium oxide must contain at least one additional element. His colleague Friedrich Wöhler urged him to publish, but Mosander refused to commit to a result until he was certain of it, a caution that frustrated his colleagues but was essential to the discoveries that followed.

# Two earths become six

In 1839, after years of work, Mosander confirmed that cerium oxide was not a single substance. He isolated a new element from it and named it lanthanum, from the Greek for “to escape notice.”⁸ Two years later, in 1841, he identified a third element in cerium oxide and named it didymium, from the Greek for “twin,” because its chemistry resembled that of lanthanum. Wöhler and Berzelius found the name unusual, but the result stood. Didymium would itself turn out to be a mixture in the 1880s, when it was resolved into praseodymium and neodymium.

Mosander’s analytical methods were as important as the elements he isolated. He developed techniques for repeated crystallization and gravimetric analysis that required extreme care, and in 1843 Berzelius documented them in detail, showing how Mosander had separated cerium, lanthanum, and didymium from what had appeared to be a single substance.

Mosander then applied the same methods to yttrium oxide, working from gadolinite. In 1843 he isolated two more elements, erbium and terbium, with yttrium as the remaining fraction. He used oxalate precipitation to separate them, exploiting differences in reactivity and in the color and crystal form of the precipitates. What had begun as two “earths,” cerium and yttrium, was now six elements, and the rare earths were not what chemists had first taken them to be.

# Spectral analysis

Mosander’s separation methods were slow. A single result could require months or years of repeated crystallizations and precise measurements, and as organic chemistry advanced through the middle of the 19th century, work on the rare earths slowed.

In 1859, Robert Bunsen and Gustav Kirchhoff introduced spectral analysis. Heated elements emit light at characteristic wavelengths, and the resulting pattern of spectral lines is unique to each element. Identification could now be made directly from the spectrum, without the long sequence of separation steps that Mosander had relied on.

The technique arrived just as Dmitri Mendeleev and Julius Lothar Meyer were independently developing the periodic table in the late 1860s. Their table organized the elements by atomic weight and chemical properties, exposing relationships between them and even predicting the existence of elements that had not yet been found.

The two tools were complementary. The periodic table identified gaps where new elements should exist, and spectral analysis allowed chemists to look for them directly. The slow crystallization work of Mosander’s generation gave way to a faster mode of discovery.

# A new generation

Spectral analysis and the periodic table brought a new generation of chemists to the rare earths. Swiss chemist Jean Charles Galissard de Marignac built on Mosander’s methods to recalculate the atomic weights of cerium, lanthanum, and didymium with much greater precision. His measurements arrived shortly after the 1860 Karlsruhe Congress, where the need for standardized atomic weights had been the central question, and they helped European chemists converge on consistent values.

Another Swiss chemist, Marc Delafontaine, worked on Mosander’s discoveries, in particular terbium. Several chemists had worked on isolating it, which makes the discovery hard to attribute to a single individual, but Delafontaine’s systematic studies clarified its properties. He then turned to didymium, the “twin” element Mosander had identified, and noticed subtle differences in its spectral lines that suggested it was not a single element.

The French chemist Paul Émile Lecoq de Boisbaudran later confirmed this. Working with samarskite, a mineral rich in rare earths, he used spectral analysis to show that didymium was a mixture. It would eventually be resolved into praseodymium and neodymium. The pattern set by Mosander’s lanthanum and his original didymium repeated itself: an element that had been treated as single turned out to be several.

# The final discoveries

In 1885, Austrian chemist Carl Auer von Welsbach succeeded where others had failed and separated didymium into two elements. One gave green oxides, which he named praseodidymium (“green twin”), and the other gave pink oxides, which he named neodidymium (“new twin”). They are now known as praseodymium and neodymium.

Von Welsbach pursued the practical applications as well as the chemistry. His incandescent gas mantle, which used rare-earth salts, mainly thorium and cerium, to produce bright light from a gas flame, was the basis of the lighting company Osram and made him commercially successful. It is one of the earliest cases of rare-earth research moving directly into industrial use.

In 1896, Eugène-Anatole Demarçay identified europium in samarium samples and named it after Europe. At the end of the 19th century, von Welsbach and the French chemist Georges Urbain independently showed that ytterbium contained two elements. The names they proposed (Urbain’s neoytterbium and lutetium, von Welsbach’s aldebaranium and cassiopeium) gave rise to a dispute that lasted for decades. In Germany, “cassiopeium” remained in use in place of lutetium until the Second World War.

Element 61 was the last gap. It was filled in 1947, when scientists at the Manhattan Project isolated traces of it from uranium fission products and named it promethium, after the Titan who brought fire to humanity. Promethium is highly radioactive and occurs in nature only in trace amounts.

With promethium, the family was complete. What had begun in the 1780s with two unidentified “earths” had become a chemical family of 17 elements. The history of their discovery is less a story of breakthroughs than of patient, generational work: a small number of chemists, working in close communication across national borders, slowly resolved a chemistry that had no good analytical handle until the 20th century gave them one.

Casadevall and Fang made the case for it in a 2015 editorial in Infection and Immunity.¹ Their argument was that historical awareness shapes better research: without it, researchers repeat old mistakes and overlook old discoveries. What does the history of a field give a working researcher that the current technical literature does not?

# Ideas do not survive on technical merit alone

The history of science is full of cases where the best available evidence lost to social forces: institutional authority, political protection, or inertia. Thomas Kuhn built his account of scientific progress around this observation in The Structure of Scientific Revolutions.²

Ignaz Semmelweis is the standard example. In the 1840s he found that poor hygiene among doctors was causing deadly infections in maternity wards, and that a simple handwashing protocol cut mortality. The medical establishment rejected the findings and destroyed his career, not because the data was wrong but because the conclusion threatened the authority of senior physicians.

Galileo’s case is usually told as a clash between science and religion, but politics mattered just as much. His telescope observations and his defense of Copernican cosmology became publishable in part because the Venetian Republic gave him political cover. Without that cover, the work might not have spread far beyond his own notebooks.³

Stephen Toulmin called a related pattern the “Alexandrian Trap”: researchers become so invested in the formal structure of their arguments that they lose contact with what the arguments are about. The same thing has happened many times in the history of science, and a researcher who has seen it before is more likely to recognize it now.

The common thread is that scientific ideas do not survive on technical merit alone. They have to pass through the institutions, careers, and allegiances of the people who evaluate them, and those filters are still in place.

# How scientists are trained to ignore history

Most scientists are trained to see the history of their own field as irrelevant to their work. Science is taught as a sequence of results, and history gets reduced to a handful of anecdotes: Newton’s apple, Fleming’s moldy petri dish, Kekulé’s dream of the benzene ring. The context that produced those results, and the failures that shaped the methods used to produce them, falls out of the syllabus.

The contrast with law is instructive. A law student spends years reading old cases because legal practice is explicitly shaped by precedent, and an argument cannot be made without knowing what has been argued before. A scientist is usually trained the opposite way, told to evaluate current evidence on its own terms and to treat history as background color.

The cost is that the same lessons keep being relearned. A researcher who stumbles into the trap Semmelweis fell into will not recognize the pattern unless someone has taught it. The same holds for the political suppression of correct work, the premature abandonment of productive lines, and the slow acceptance of findings that should have been obvious at the time. None of this shows up in the current literature. It is in the historical record.

# The case against looking back

The counterargument is that science thrives by ignoring its history. Unlike law, where precedent is binding, science progresses by overturning earlier views, and being too attached to the history of a field encourages deference to ideas that should be discarded. A physicist who has internalized all of classical mechanics may have a harder time taking a radical departure from it seriously.

The view is consistent with how experimental science claims to work. A hypothesis is tested against evidence and either survives or does not, and the historical circumstances under which it was proposed do not change the outcome of the test. By that standard, history is at best irrelevant to the experiment and at worst a source of bias that gives too much weight to what was once believed.

This forward-only view has two problems. A hypothesis never gets evaluated in a vacuum: the judgment of which results are worth publishing, which data is clean enough to trust, and which directions are worth pursuing is not mechanical. Those judgments are made by researchers whose intuitions were built from experience, and experience is a kind of private history. The second problem is that the failure modes of past research are not random. They repeat, and a researcher who has studied them can see them coming.

# What history gives a researcher

Casadevall and Fang’s point is that science does not happen in a vacuum. Every result is produced inside a social and institutional context that shapes what questions were asked, what methods were used, and what findings got through peer review. A researcher who knows how that context has shaped past work is better equipped to recognize how it is shaping current work.

The practical effect is on judgment. A researcher familiar with how institutional pressures shaped Galileo’s and Semmelweis’s work is more likely to notice the same forces in a modern grant committee. A researcher who has seen productive research programs abandoned prematurely is less quick to give up on their own. A researcher who has studied how past discoveries were actually made is less likely to believe they came from isolated genius, which makes collaboration easier and setbacks less discouraging.⁴

Historical study also corrects a bias in how credit is distributed. Contributions from women and from scientists outside the dominant European and American institutions become visible only when someone goes back and reads the original literature. The same pattern holds for ethics: the rules that get written into codes of conduct were usually learned after something went wrong, and the case files for those events are in the historical record, not in the current methods section.

Looking back and looking forward are not in competition. The history of a field is the longest-running dataset about how research actually gets done in it, and ignoring that dataset means ignoring the one closest to hand.

# Crystalline and amorphous solids

Solids fall into two structural categories: crystalline and amorphous. In crystalline solids, such as diamond or metallic copper, atoms occupy a highly ordered, repeating lattice. In amorphous solids, including glass, the atoms are disordered, resembling a liquid frozen in place rather than a regular lattice.^{2 3}

The periodic arrangement gives crystalline solids two characteristic properties. They have a sharp melting point: at the right temperature, the entire lattice breaks down at once. They are also anisotropic, meaning their mechanical properties depend on the direction of an applied force, because some directions in the lattice are stronger than others.

Glass differs in both respects. Because there is no lattice to break, it does not have a sharp melting point and softens gradually as temperature rises, transitioning from rigid to fluid over a range. The disordered atomic arrangement has no preferred direction, so glass is isotropic: its mechanical properties are the same along every axis.

The contrast is sharpest in silica ($\mathrm{SiO}_2$). In its crystalline form, silica is quartz, with a sharp melting point and anisotropic mechanical properties. In its amorphous form, the same chemical composition produces a glass with no melting point and isotropic properties.

Glass is not limited to silica. Many other substances, including metals and polymers, can form amorphous structures when cooled rapidly enough to suppress crystallization. What they all share is the absence of long-range order in the atomic arrangement.

# Metastability

Glass is metastable: it sits in a local energy minimum that is not the global minimum, but the energy barrier separating the two is too large to cross under ordinary conditions. For almost any glass-forming substance, the global minimum is a crystalline arrangement.^{4 5}

Crystallization requires atoms to settle into the periodic positions of a lattice, and that takes time. When a melt is cooled fast enough, there is not enough time for them to do so, and the resulting solid carries the disordered arrangement it had in the liquid state. The faster the cooling, the more frozen-in disorder.

Compared to a crystal of the same composition, a glass has a larger molar volume (its atoms are not packed as tightly), higher entropy (the disordered arrangement has more accessible microstates), and higher enthalpy and Gibbs free energy. All of these reflect the same fact: glass is in a higher-energy state, but the activation energy required to reach the lower-energy crystalline state is too large to cross at ordinary temperatures.

# The glass transition

The transformation from a glass-forming liquid into a glass is called the glass transition, and the process of carrying it out (cooling a liquid fast enough to bypass crystallization) is called vitrification. Unlike the melting transition of a crystal, which happens at a single sharp temperature, the glass transition is gradual: as a liquid is cooled, its viscosity rises, and over some temperature range it becomes so viscous that, on the timescales of observation, it stops flowing. The temperature at which this happens is called the glass transition temperature, $T_g$.³

$T_g$ is not a thermodynamic constant of the material but a quantity that depends on the cooling rate. Faster cooling traps the atoms at a higher temperature, before they have time to relax into lower-energy arrangements, and produces a higher $T_g$. Slower cooling gives them more time to relax and produces a lower $T_g$. The same substance can have different glass transition temperatures depending on its thermal history.

This time-dependence is the central distinction between glass and crystalline solids. Crystals melt at a fixed temperature ($T_m$), and the transition is sharp. Glasses do not melt at all in the strict sense. They pass continuously between rigid solid and viscous liquid, and the location of the transition depends on time as much as on temperature.

# Supercooled liquids

A liquid cooled below its freezing point without crystallizing is a supercooled liquid. As it cools further, its atomic structure continues to rearrange through a process called structural relaxation, which slows as the temperature drops. The characteristic timescale for this rearrangement is the relaxation time, $\tau_s$, which increases sharply as the liquid approaches the glass transition.

When the cooling rate exceeds the structural relaxation rate, the atomic arrangement is frozen in: continued cooling produces no further structural change, and the supercooled liquid becomes a glass. The temperature at which this happens is called the fictive temperature, and for many materials it coincides with $T_g$.

# Conclusion

Glass is structurally disordered, thermodynamically out of equilibrium, and stable on human timescales. These features, together with the glass transition that connects the liquid and solid states, are common to all glasses, whether oxide, metallic, or polymeric. What varies between systems is the chemistry that sets the relaxation timescales and the value of $T_g$.

# Bush and the OSRD

Vannevar Bush ran the Office of Scientific Research and Development (OSRD) during World War II. The OSRD coordinated the American wartime scientific effort between government laboratories, universities, and private industry, operating at a scale and under time pressure that had no peacetime precedent. Pieces of the Action is Bush’s memoir of that work, written as a management manual rather than as a history. It is opinionated and concerned mainly with the practical question of how to run technical people at scale.

# Why R&D is hard to manage

R&D is hard to manage because the goal is not fixed at the start.¹ A conventional project can be broken into steps and scheduled. A research project usually cannot. New results reshape the direction, unexpected failures close off promising routes, and the endpoint is often different from the one originally intended.

That same uncertainty is what makes R&D valuable. The technologies that turn out to matter come from work that could not have been planned in advance, which is why R&D is worth doing. The management problem is how to allow enough slack for the unexpected to happen without losing all direction.

The scale of modern R&D investment is easy to illustrate. Amazon, Alphabet, Meta, Apple, and Microsoft spent over $200 billion on research and development in 2022 alone. But spending is not the bottleneck. What separates a productive research program from an unproductive one is whether the organization has a working method for deciding which problems to pursue and which to drop.

That method lives in the judgment of experienced researchers and managers rather than in any formal process, which makes it hard to build and harder to transfer between organizations. Companies good at R&D train their people to recognize problems worth working on, to approach them methodically, and to iterate toward solutions. Companies bad at R&D treat research as a budget line and assume that outputs will follow.

# Context and case studies

R&D management is also heavily context-dependent. The traditional project-management constraints of time, cost, and scope (the “iron triangle”) apply to any research project but are not sufficient.² A project that succeeds on all three can still fail on factors the iron triangle does not measure: team culture, the politics of the stakeholders, the development methodology, and how the work transitions from a laboratory prototype into production.³

What works for one project often does not work for another, because the context differs in ways that cannot be captured by a generic methodology. A manager who wants to apply the lessons of a past success has to identify which parts of the original success were context-dependent and which were not. That separation is itself the skill being applied.

Fields that deal with context-dependent problems, such as law, medicine, and business management, have developed the case study as a way of carrying forward lessons that do not generalize cleanly into rules. The method originated in legal education under Christopher Columbus Langdell.^{4 5} A case study preserves the context that a general principle would strip out, and lets the reader decide which aspects of the situation are load-bearing and which are incidental.

Pieces of the Action is, in this sense, a case study of the largest coordinated R&D effort in modern history. Bush is not trying to prove a general theory of how research should be managed. He is reporting what he did, why he did it, and what he thought worked. The reader who wants general principles has to extract them, and that is how the book is meant to be read.

# Read it

The wartime R&D effort produced radar, the proximity fuse, guided missiles, jet engines, and the scaled production of penicillin. Bush ran the American side of that effort as head of the OSRD. He was not the inventor of any of these technologies, but he was the administrator who decided what to fund, who should work on what, how the work should be divided between academic and industrial laboratories, and when to push a project from research into production.

The book covers both sides of that job. Bush was an engineer himself, which gives the technical discussion credibility, but the technical material is not what makes the book unusual. The unusual part is the managerial material: specific, opinionated advice on how to run committees, how to work around the politics of wartime bureaucracy, how to allocate research freedom without losing focus, and how to remove people whose arrogance or rigidity slows everyone else down. An entire chapter is dedicated to that last problem, and Bush calls the people in question tyros.

Read it. The technical details are dated, but the management problems are not, and few people have Bush’s combination of first-hand experience with large-scale coordinated research and willingness to be specific about what actually worked.

ORCA is a quantum chemistry program for molecular electronic structure calculations, developed and maintained by Frank Neese’s research group at the Max-Planck-Institut für Kohlenforschung in Mülheim an der Ruhr, Germany. It supports density functional theory (DFT), coupled-cluster theory, semi-empirical methods, and more.

Version 6.0.0, released in August 2024, is a major redesign (see the foreword to its release ). This guide covers installing it on macOS with Apple Silicon (M1, M2, etc.), including Open MPI setup for parallel calculations.

# Installation

# Getting the files

To get started, download the ORCA package from the official ORCA download page . Note that you’ll need a registered account to access the download—registration is free if you don’t have an account yet.

For macOS on Arm64, the download page lists two options (both containing pre-compiled binaries linked to Open MPI 4.1.1):

1. ORCA 6.0.0, MacOS X, Arm64, Installer Version: Extracts, installs, and adds ORCA to your PATH automatically.
2. ORCA 6.0.0, MacOS X, Arm64, .tar.bz2 Archive: Manual extraction and PATH setup, but more control over the installation location.

This guide uses the Installer Version, which downloads as orca_6_0_0_macosx_arm64_openmpi411.run.

# Setting file permissions

Before running the installer, make it executable:

cd ~/Downloads
chmod +x orca_6_0_0_macosx_arm64_openmpi411.run

# Running the installer

Run the installer from the same directory:

$ ./orca_6_0_0_macosx_arm64_openmpi411.run

Creating directory orca_6_0_0
Verifying archive integrity...  100%   MD5 checksums are OK. All good.
Uncompressing ORCA 6.0.0 Installer ...  100%

By default, this operation installs the ORCA binaries in the directory /Users/<username>/Library/Orca/, where <username> is your system’s user.

# Setting the PATH

The installer should automatically add ORCA to your system’s PATH. Verify by running:

$ orca

This program requires the name of a parameterfile as argument
For example ORCA TEST.INP

If you see this output, ORCA is installed. You can also confirm by checking your .zshrc (macOS ships with zsh by default) for a line like:

# ORCA 6.0.0 section
export PATH=/Users/<username>/Library/orca_6_0_0:$PATH

If this line is missing, add it manually, pointing to the correct installation directory. Save the file and restart your terminal.

# Parallelization with Open MPI

To run calculations in parallel, ORCA needs Open MPI.

Download Open MPI 4.1.6 from the official Open MPI page . The file is openmpi-4.1.6.tar.gz.

Before compiling, install the Fortran compiler via Homebrew :

brew install gcc

Unpack and prepare Open MPI for compilation. This example installs it in a dedicated folder in the home directory:

# Create a directory for Open MPI
$ mkdir $HOME/openmpi
$ cd $HOME/openmpi

# Move the downloaded file and extract it
$ cp ~/Downloads/openmpi-4.1.6.tar.gz .
$ tar -xzvf openmpi-4.1.6.tar.gz
$ cd openmpi-4.1.6

Compile Open MPI with Fortran support and the flags needed for ORCA:

./configure --prefix=$HOME/openmpi --without-verbs --enable-mpi-fortran --disable-builtin-atomics
make all
make install

./configure --prefix=$HOME/openmpi --without-verbs --enable-mpi-fortran --disable-builtin-atomicsC LDFLAGS="-ld_classic"

Compilation takes time. To speed it up, use multiple cores with the -jN flag:

make -j4 all
make -j4 install

After compilation, create a symbolic link to make Open MPI accessible system-wide:

sudo ln -s $HOME/openmpi/lib/libmpi.40.dylib /usr/local/lib/libmpi.40.dylib

Update the PATH and LD_LIBRARY_PATH in your .zshrc (or .bash_profile if using bash):

export PATH=$HOME/openmpi/bin:$PATH
export LD_LIBRARY_PATH=$HOME/openmpi/lib:$LD_LIBRARY_PATH

Run source ~/.zshrc (or restart your terminal), then verify:

$ mpirun --version

mpirun (Open MPI) 4.1.6

Report bugs to http://www.open-mpi.org/community/help/

If you see this output, Open MPI is ready.

# Testing

Verify the setup by running a simple calculation. Create a folder for the test files:

# Navigate to the desktop
$ cd ~/Desktop

# Create a new folder for the test
$ mkdir orca_test
$ cd orca_test

# Create an input file for the calculation
$ touch water.inp

Add a basic Hartree-Fock (HF) calculation on a water molecule using the Def2-SVP basis set (from the official ORCA tutorial ):

!HF DEF2-SVP

* xyz 0 1
O   0.0000   0.0000   0.0626
H  -0.7920   0.0000  -0.4973
H   0.7920   0.0000  -0.4973
*

Save the file and run ORCA in serial mode:

$ orca water.inp

                                 *****************
                                 * O   R   C   A *
                                 *****************

--- truncated output ---

Timings for individual modules:

Sum of individual times          ...        0.245 sec (=   0.004 min)
Startup calculation              ...        0.081 sec (=   0.001 min)  33.0 %
SCF iterations                   ...        0.114 sec (=   0.002 min)  46.5 %
Property calculations            ...        0.050 sec (=   0.001 min)  20.6 %
                             ****ORCA TERMINATED NORMALLY****
TOTAL RUN TIME: 0 days 0 hours 0 minutes 0 seconds 302 msec

If the output ends with ORCA TERMINATED NORMALLY, the serial setup works. Next, test parallel execution by adding a %PAL block that specifies the number of processes. Update water.inp:

!HF DEF2-SVP

%PAL NPROCS 4 END

* xyz 0 1
O   0.0000   0.0000   0.0626
H  -0.7920   0.0000  -0.4973
H   0.7920   0.0000  -0.4973
*

For parallel runs, use the full path to the ORCA executable (replace <username> with your macOS username):

$ /Users/<username>/Library/orca_6_0_0/orca water.inp

           ************************************************************
           *        Program running with 4 parallel MPI-processes     *
           *              working on a common directory               *
           ************************************************************

--- truncated output ---

Timings for individual modules:

Sum of individual times          ...       17.403 sec (=   0.290 min)
Startup calculation              ...       16.408 sec (=   0.273 min)  94.3 %
SCF iterations                   ...        0.721 sec (=   0.012 min)   4.1 %
Property calculations            ...        0.275 sec (=   0.005 min)   1.6 %
                             ****ORCA TERMINATED NORMALLY****
TOTAL RUN TIME: 0 days 0 hours 0 minutes 17 seconds 708 msec

If the output confirms 4 parallel MPI processes and terminates normally, both ORCA and Open MPI are working.

Answering that question requires crossing from physics into perception. Color is a sensation produced by the brain’s processing of signals from the retina, not a property of light itself. The same spectral power distribution can look different under different illuminants; different distributions can look identical. This is metamerism: two lights with different spectral compositions produce the same cone responses and appear identical to a normal observer under standard conditions.^{1 2}

Psychophysics quantifies this relationship between physical stimuli and perceived sensations. Guy Brindley formalized the distinction in 1970 by categorizing perceptual observations into two types:

1. Class A observations: Two physically different stimuli are perceived as identical — the observer cannot distinguish them.
2. Class B observations: All other cases where stimuli are distinguishable.

Color matching is the canonical Class A case. It is also the experimental foundation of the CIE colorimetry system.

# The CIE colorimetry system

The CIE colorimetry system, developed by the Commission Internationale d’Éclairage, translates spectral power distributions into standardized color coordinates using color matching experiments.^{3 4}

In these experiments, observers view a split field: one half shows a test color, the other a mixture of three primary lights. The observer adjusts the primaries until both halves match. Repeating this across the visible spectrum and averaging over many observers produces a statistical model of human color vision.

The CIE 1931 model uses additive color mixing based on Color Matching Functions (CMFs), denoted $\bar{x}$, $\bar{y}$, and $\bar{z}$. CMFs are not the spectral sensitivities of the cone cells directly, but linear transformations of them, derived from standardized color matching experiments involving foveal vision, specific field sizes, dark surroundings, and averaged observations from multiple individuals.

By convolution of the sample spectrum $M(\lambda)$ with the CMFs, we calculate tristimulus values $X$, $Y$, and $Z$. These values represent the amounts of the three primary colors (red, green, and blue) required to match the given color.

The tristimulus values define a point in a three-dimensional color space. For visualization, this space is reduced to two dimensions using the $x$ and $y$ chromaticity coordinates:

The $x$ and $y$ coordinates specify a chromaticity (hue and saturation) independently of luminance. This is what makes the diagram useful for comparing colors across different brightness levels.

# Python code

# Dependencies

The heavy lifting is done by colour-science , a Python library that implements most major colorimetric systems, color space conversions, and color difference metrics. The remaining dependencies are standard: numpy, pandas, and matplotlib. Install them with:

1pip install numpy pandas matplotlib colour-science

Then import them:

1import colour as cl
2import matplotlib.pyplot as plt
3import matplotlib.ticker as tck
4import numpy as np
5import pandas as pd
6
7# Disable some annoying warnings from colour library
8cl.utilities.filter_warnings(colour_usage_warnings=True)

# Plot settings

The following settings match the blog’s plot style. If you are following along in a Jupyter notebook, you can skip this block. I use seaborn for its default presets (context, ticks, colorblind palette) and the golden ratio from scipy to set the figure aspect ratio.

 9import seaborn as sns
10from scipy.constants import golden_ratio
11
12# Set seaborn defaults
13sns.set_context("notebook")
14sns.set_style("ticks")
15sns.set_palette("colorblind", color_codes=True)
16
17# Use white color for elements that are typically black
18plt.style.use("dark_background")
19
20# Remove background from figures and axes
21plt.rcParams["figure.facecolor"] = "none"
22plt.rcParams["axes.facecolor"] = "none"
23
24# Default figure size to use throughout
25figure_size = (7, 7 / golden_ratio)

# Plotting the CIE (2°) color space

The colour-science library provides a ready-made function for plotting the CIE 1931 chromaticity diagram. The diagram shows all chromaticities visible to a 2° standard observer; the spectral locus traces the monochromatic wavelengths, and any real color falls inside it. We will plot our chlorophyll colors on this diagram later.

26# Instantiate figure and axes
27fig, ax = plt.subplots(1, 1, figsize=(7, 7))
28
29# Plot CIE color space for a 2°
30cl.plotting.plot_chromaticity_diagram_CIE1931(
31    cmfs="CIE 1931 2 Degree Standard Observer",
32    axes=ax,
33    show=False,
34    title=None,
35    spectral_locus_colours="white",
36)
37
38# Axes labels
39ax.set_xlabel("x (2°)")
40ax.set_ylabel("y (2°)")
41
42# Axes limits
43ax.set_xlim(-0.1, 0.85)
44ax.set_ylim(-0.1, 0.95)
45
46# Ticks separation
47ax.xaxis.set_major_locator(tck.MultipleLocator(0.1))
48ax.xaxis.set_minor_locator(tck.MultipleLocator(0.01))
49ax.yaxis.set_major_locator(tck.MultipleLocator(0.1))
50ax.yaxis.set_minor_locator(tck.MultipleLocator(0.01))
51
52# Grid settings
53ax.grid(which="major", axis="both", linestyle="--", color="gray", alpha=0.8)
54
55# Padding adjustment
56plt.tight_layout()
57
58plt.show()

# Importing and scaling data

The data are pre-recorded UV-Vis absorption spectra of Chlorophyll A and Chlorophyll B in 70% and 90% acetone solutions, taken from Chazaux et al.⁵ You can download the .csv file as chlorophyll_uv_vis.csv .

59column_names = ["lambda", "chl_a_70", "chl_a_90", "chl_b_70", "chl_b_90"]
60measured_samples = pd.read_csv(
61    "chlorophyll_uv_vis.csv", names=column_names, header=0, index_col="lambda"
62)
63measured_samples

1001 rows × 4 columns

Each column records absorbance ($A$) as a function of wavelength ($\lambda$) for one chlorophyll–solvent combination:

60fig, ax = plt.subplots(1, 1, figsize=figure_size)
61
62# Define the labels for the plot's legend
63chl_labels = [
64    "Chlorophyll A (70 % Acetone)",
65    "Chlorophyll A (90 % Acetone)",
66    "Chlorophyll B (70 % Acetone)",
67    "Chlorophyll B (90 % Acetone)",
68]
69
70# Iterate over dataframe and plot each spectrum
71for col, sample_label in zip(measured_samples.columns, chl_labels):
72    ax.plot(measured_samples.index, measured_samples[col], label=sample_label)
73
74# Ticks separation
75ax.xaxis.set_major_locator(tck.MultipleLocator(50))
76ax.xaxis.set_minor_locator(tck.MultipleLocator(10))
77
78# Axes labels
79ax.set_xlabel("Wavelength [nm]")
80ax.set_ylabel("Absorbance [a.u.]")
81
82# Display legend
83ax.legend()
84
85plt.show()

Chlorophyll A absorbs primarily in the blue (~430 nm) and red (~670 nm) regions. Chlorophyll B shows a similar pattern with peaks near 460 nm and 650 nm, its blue peak shifted slightly toward the green. The acetone concentration affects peak intensities but not spectral shape.

Because the four spectra have different absolute absorbance values, we need to normalize them before comparing colors. Normalization discards quantitative information (concentrations), but since our goal is qualitative (what color does each spectrum produce?), this is acceptable.

We use MinMax scaling to map each spectrum to the 0–1 range while preserving its shape:⁶

A simple custom function avoids pulling in scikit-learn for a one-line operation:

86def normalize(x: pd.Series | np.ndarray) -> pd.Series | np.ndarray:
87    """MinMax scaling from 0 to 1
88
89    Args:
90        x (pd.Series | np.ndarray): series or array to normalize
91
92    Returns:
93        pd.Series | np.ndarray: series or array of normalized values
94    """
95    x_scaled = (x - x.min()) / (x.max() - x.min())
96    return x_scaled

We apply it to the full DataFrame at once, since pandas vectorizes the operation across all columns:

97abs_norm = normalize(measured_samples)
98abs_norm

1001 rows × 4 columns

A quick plot confirms the spectra now share the same 0–1 scale while retaining their characteristic shapes:

 99fig, ax = plt.subplots(1, 1, figsize=figure_size)
100
101for col, sample_label in zip(abs_norm.columns, chl_labels):
102    ax.plot(abs_norm.index, normalize(abs_norm[col]), label=f"{sample_label} norm")
103
104# Ticks separation
105ax.xaxis.set_major_locator(tck.MultipleLocator(50))
106ax.xaxis.set_minor_locator(tck.MultipleLocator(10))
107ax.yaxis.set_major_locator(tck.MultipleLocator(0.1))
108ax.yaxis.set_minor_locator(tck.MultipleLocator(0.025))
109
110# Axes labels
111ax.set_xlabel("Wavelength [nm]")
112ax.set_ylabel("Absorbance [a.u.]")
113
114# Display legend
115ax.legend()
116
117plt.show()

# Converting absorbance to transmittance

The spectra above record absorbed light. The color we perceive depends on the light that passes through the sample: the transmittance. The conversion from absorbance ($A$) to percent transmittance ($\%T$) follows the Beer-Lambert law:

Note that because we are using normalized absorbance values (0–1) rather than actual absorbance, the resulting transmittance values do not represent true physical transmittance. They preserve the spectral shape, which is sufficient for a qualitative color comparison, but should not be interpreted quantitatively.

In code:

120def abs_to_trans(A: pd.Series | np.ndarray) -> pd.Series | np.ndarray:
121    """Convert absorbance to transmittance
122
123    Args:
124        A (pd.Series | np.ndarray): series or array of absorbance values
125
126    Returns:
127        pd.Series | np.ndarray: series or array of transmittance values
128    """
129    T = 10 ** (2 - A)
130    return T

Running it on the normalized absorbance values:

131transm_norm = abs_to_trans(abs_norm)
132transm_norm

1001 rows × 4 columns

The transmittance spectra, plotted below, should mirror the absorbance spectra inverted:

133fig, ax = plt.subplots(1, 1, figsize=figure_size)
134
135for col, sample_label in zip(transm_norm.columns, chl_labels):
136    ax.plot(transm_norm.index, transm_norm[col], label=f"{sample_label}")
137
138# Ticks separation
139ax.xaxis.set_major_locator(tck.MultipleLocator(50))
140ax.xaxis.set_minor_locator(tck.MultipleLocator(10))
141ax.yaxis.set_major_locator(tck.MultipleLocator(10))
142ax.yaxis.set_minor_locator(tck.MultipleLocator(5))
143
144# Axes labels
145ax.set_xlabel("Wavelength [nm]")
146ax.set_ylabel("Transmittance [%]")
147
148# Display legend
149ax.legend()
150
151plt.show()

The inverse relationship between absorbance and transmittance is visible: peaks in absorbance correspond to troughs in transmittance. A side-by-side comparison:

152fig, ax = plt.subplots(2, 1, sharex=True, figsize=figure_size)
153
154# Iterate over absorbance dataframe
155for col, sample_label in zip(abs_norm.columns, chl_labels):
156    ax[0].plot(abs_norm.index, normalize(abs_norm[col]), label=f"{sample_label}")
157
158# Iterate over transmittance dataframe
159for col, sample_label in zip(transm_norm.columns, chl_labels):
160    ax[1].plot(transm_norm.index, transm_norm[col], label=f"{sample_label}")
161
162# Axes labels
163ax[0].set_ylabel("Absorbance [a.u.]")
164
165ax[1].set_xlabel("Wavelength [nm]")
166ax[1].set_ylabel("Transmittance [%]")
167
168# Ticks separation
169for axis in ax:
170    axis.xaxis.set_major_locator(tck.MultipleLocator(50))
171    axis.xaxis.set_minor_locator(tck.MultipleLocator(10))
172    # Display legend
173    axis.legend()
174
175ax[0].yaxis.set_major_locator(tck.MultipleLocator(0.1))
176ax[0].yaxis.set_minor_locator(tck.MultipleLocator(0.025))
177ax[1].yaxis.set_major_locator(tck.MultipleLocator(10))
178ax[1].yaxis.set_minor_locator(tck.MultipleLocator(5))
179
180plt.show()

# Calculating the CIE colors

The pipeline for each spectrum:

1. Build a SpectralDistribution and interpolate to 1 nm intervals (CIE specification).
2. Compute $XYZ$ tristimulus values with sd_to_XYZ, using the CIE 1931 2° observer CMFs and the D65 daylight illuminant.
3. Convert $XYZ$ to $xy$ chromaticity coordinates.

The results for absorbance-based and transmittance-based colors are stored in separate lists, then merged into a single DataFrame.

181# Define color matching functions
182cmfs = cl.MSDS_CMFS["cie_2_1931"]
183
184# Define illuminant
185illuminant = cl.SDS_ILLUMINANTS["D65"]
186
187chl_abs_clr = []
188
189# Iterate over each normalized absorbance spectrum
190for col in abs_norm.columns:
191    # Initialize spectral distribution
192    sd = cl.SpectralDistribution(data=abs_norm[col])
193
194    # Interpolate sd to conform to the CIE specifications
195    sd = sd.interpolate(cl.SpectralShape(350, 750, 1))
196
197    # Calculate CIE XYZ coordinates from spectral distribution
198    cie_XYZ = cl.sd_to_XYZ(sd, cmfs, illuminant)
199
200    # Convert to CIE xy coordinates
201    cie_xy = cl.XYZ_to_xy(cie_XYZ)
202
203    # Append the results to the list of sample colors
204    chl_abs_clr.append(
205        {"sample": col, "x_A": np.round(cie_xy[0], 4), "y_A": np.round(cie_xy[1], 4)}
206    )
207
208chl_transm_clr = []
209
210# Iterate over each normalized transmittance spectrum
211for col in transm_norm.columns:
212    # Initialize spectral distribution
213    sd = cl.SpectralDistribution(data=transm_norm[col])
214    sd = sd.interpolate(cl.SpectralShape(350, 750, 1))
215
216    # Calculate CIE XYZ coordinates from spectral distribution
217    cie_XYZ = cl.sd_to_XYZ(sd, cmfs, illuminant)
218
219    # Convert to CIE xy coordinates
220    cie_xy = cl.XYZ_to_xy(cie_XYZ)
221
222    # Append the results to the list of sample colors
223    chl_transm_clr.append(
224        {"sample": col, "x_T": np.round(cie_xy[0], 4), "y_T": np.round(cie_xy[1], 4)}
225    )
226
227# Convert dictionaries to dataframes and join them together
228colors = pd.merge(pd.DataFrame(chl_transm_clr), pd.DataFrame(chl_abs_clr))
229colors

# Visualizing colors on the CIE color space

Plotting the absorbed colors on the CIE 1931 chromaticity diagram:

230# Instantiate figure and axes
231fig, ax = plt.subplots(1, 1, figsize=figure_size)
232
233# Plot CIE color space for a 2°
234cl.plotting.plot_chromaticity_diagram_CIE1931(
235    cmfs="CIE 1931 2 Degree Standard Observer",
236    axes=ax,
237    show=False,
238    title=None,
239    spectral_locus_colours="white",
240)
241
242color_list = ["r", "g", "b", "m"]
243
244for i, c in enumerate(color_list):
245    ax.scatter(
246        colors["x_A"][i],
247        colors["y_A"][i],
248        label=chl_labels[i],
249        color=c,
250        edgecolors="k",
251        alpha=0.8,
252    )
253
254# Axes labels
255ax.set_xlabel("x (2°)")
256ax.set_ylabel("y (2°)")
257
258# Axes limits
259ax.set_xlim(0.18, 0.32)
260ax.set_ylim(0.06, 0.16)
261
262# Ticks separation
263ax.xaxis.set_major_locator(tck.MultipleLocator(0.01))
264ax.xaxis.set_minor_locator(tck.MultipleLocator(0.005))
265ax.yaxis.set_major_locator(tck.MultipleLocator(0.01))
266ax.yaxis.set_minor_locator(tck.MultipleLocator(0.005))
267
268# Grid settings
269ax.grid(which="major", axis="both", linestyle="--", color="gray", alpha=0.8)
270
271# Display legend
272ax.legend()
273
274# Adjust plot padding
275plt.tight_layout()
276
277plt.show()

The same plot for the transmitted colors:

278# Instantiate figure and axes
279fig, ax = plt.subplots(1, 1, figsize=figure_size)
280
281# Plot CIE color space for a 2°
282cl.plotting.plot_chromaticity_diagram_CIE1931(
283    cmfs="CIE 1931 2 Degree Standard Observer",
284    axes=ax,
285    show=False,
286    title=None,
287    spectral_locus_colours="white",
288)
289
290color_list = ["r", "g", "b", "m"]
291
292for i, c in enumerate(color_list):
293    ax.scatter(
294        colors["x_T"][i],
295        colors["y_T"][i],
296        label=chl_labels[i],
297        color=c,
298        edgecolors="k",
299        alpha=0.8,
300    )
301
302# Axes labels
303ax.set_xlabel("x (2°)")
304ax.set_ylabel("y (2°)")
305
306# Axes limits
307ax.set_xlim(0.25, 0.4)
308ax.set_ylim(0.35, 0.45)
309
310# Ticks separation
311ax.xaxis.set_major_locator(tck.MultipleLocator(0.01))
312ax.xaxis.set_minor_locator(tck.MultipleLocator(0.005))
313ax.yaxis.set_major_locator(tck.MultipleLocator(0.01))
314ax.yaxis.set_minor_locator(tck.MultipleLocator(0.005))
315
316# Grid settings
317ax.grid(which="major", axis="both", linestyle="--", color="gray", alpha=0.8)
318
319# Display legend
320ax.legend(labelcolor="k", edgecolor="k")
321
322# Adjust plot padding
323plt.tight_layout()
324
325plt.show()

The transmitted-color coordinates confirm what the spectra suggest. Both chlorophylls absorb in the blue and red regions and transmit primarily in the green, but the shift in Chlorophyll B’s blue absorption peak toward longer wavelengths pushes its transmitted color toward yellow-green. Chlorophyll A, with its blue peak at shorter wavelengths, sits closer to a neutral green.

# Where to go from here

The same pipeline applies to any molecule with a UV-Vis spectrum: dyes, pigments, optical filter glasses, fluorescent proteins. Swapping in the CIE 1976 $L^{*}a^{*}b^{*}$ color space (via colour-science’s XYZ_to_Lab function) would give perceptually uniform coordinates, making it possible to compute meaningful color differences ($\Delta E$) between samples. For quantitative work, the normalization step should be revisited — using actual absorbance values with a defined path length and concentration preserves the physical relationship between Beer-Lambert transmittance and perceived color.