An Unexpected Partnership and Bitmap Printing Has Resulted in a Solution for Pediatric Epilepsy Surgery

Though movies and TV shows love to show the trope of the lonely genius, working alone to save the world, in reality collaboration is a necessity in order to realize the most incredible projects. And what better way to find these collaborations than through conferences? One notable example is actually shown by two of the keynote speakers at AMUG 2023, Nicholas Jacobson, a professor at the University of Colorado’s Anschutz Medical Campus, and Rob Ducey, a technical supervisor at animation specialist Laika Studios. The two met by chance at AMUG 2019, bonding over a little-known 3D modeling and printing process called bitmap printing. Now, they are working together on a project to use the technology to create innovative designs for pediatric epilepsy, cardiology and cleft palates. We sat down with the two to learn more.

3DN: Could you please introduce yourselves?

Nick: Hi, I’m Nick Jacobson and I’ve been involved in 3D printing since 2004. My background is in architecture, and I first worked on 3D printing in architecture school using binder jetting with refined corn starch on a Z-corp printer. We made landscape and architectural models and this was just at the dawn of the computational design boom in architecture. Students we just starting to learn how to code and program, so we became very interested in translating these ideas through digital fabrication to create unusual forms.

This love of form generation followed me throughout my architectural career, where I worked for some great architects such as Renzo Piano, Zaha Hadid, and Harry Teague. After some years practicing architecture, I was accepted into the Harvard Graduate School of Design and the Harvard Business School. Here, I was part of an early research program looking into Voxel Printing on what was then Connex and now Stratasys. We developed software to model and 3D printing using Voxel technology. Initially the work focused on structural engineering and design.

Currently, I work at the University of Colorado on the Anschutz Medical Campus and have a lab focused computation and 3D printing for medical applications. The software we developed at Harvard has proven to be valuable for the medical world, simply by modifying the input data. It may seem like a stretch but it’s very similar to working with structural analysis data. Currently, we are focused in 3 areas: Surgical guidance and education models, Prosthetics, and Implants… all revolving around soft tissue applications.

Rob Ducey (left) and Nick Jacobson (right)

Rob: Hello, I’m Rob Ducey and I’m currently the technical supervisor for what we call rapid prototype at LAIKA. In a nutshell, our group handles modeling, facial animation, and fabrication for the complex puppets that are the stars of our films. The moniker “rapid prototype” is a legacy of the innovations we pioneered on our first film Coraline back in 2009 using 3D printing for facial animation. This work received a sci-tech Oscar in 2016. I became involved with 3D printing on that project back in 2006 when Brian Mclean and Martin Munier came to the studio with the bold plan to use Objet polyjet printers to generate the thousands of faces required to create the facial performances of the characters in that film. I was tasked with building a pipeline to model, animate, and ultimately print these faces. Ever since that initial production, we have continued to innovate our processes to take advantage of and now push the limits of multi-material 3D printing. A humble fine art major, I now get to tinker with some of the most cutting-edge technology in computer aided fabrication.

3DN: You are from two very different disciplines, how did you meet?

Nick: We met at AMUG many years back when LAIKA was giving the keynote. Their presentation was amazing and highlighted how they used bitmap printing technology to create much of their work. It was cathartic to hear others using the same esoteric technology in a huge way. I was in awe of them and didn’t have the guts to chat with them eighth away. However, later on Brian and Rob Ducey came to my talk on voxel modeling and bitmap printing for surgery. Afterwards, they were the first two to come up and talk to me and we bonded over our shared admiration for bitmap printing and what is possible. Then we connected virtually and discussed our individual approaches to the technology. It was immediately obvious that we had simple solutions to some of the things the other were wanting to attempt. For example, the special effects world had long had solutions to working with thin hair like structures, which made working with tractography a breeze, something we had been wrestling with for a while. In our first meetings, and still today, we rattle off tons of ideas that the special effects world has been doing for years that allow for advancements in medical applications, things that no one had through of before because there is little to no overlap between the professions. All it took was for a simple connection to be made which could allow for these conversations.

Rob: Here at the studio, we are always on the hunt for new ideas and technical developments we can employ to expand our capabilities and redefine what is possible in stop motion animation no matter where they come from. In our use of multi-material printing, we discovered a small community of like-minded tinkerers who had access to features still considered “in development”. After some persistence and pestering, we were able to have conversations and visits with university researchers at institutions such as MIT and Virginia Tech who were curious about what a bunch of artists and filmmakers in a warehouse in the rain-soaked northwest were doing with these machines.  These conversations turned us on to AMUG, which we first attended in 2015 and then again in 2019 to give a keynote. When attending AMUG, we are always laser focused on finding others we can learn from or simply share our excitement for multi-material 3D printing. I saw the description of Nick’s presentation and made sure we were there. We approached Nick excited to find someone in what was then a very small club of “voxel printers”. Partly it was to show off what we had learned, but also to see what secrets he had uncovered. It was like finding someone in a lost tribe.

Laika has used 3D printing for a number of years for its films (photo credits: Laika)

3DN: Could you tell us more about your project? Additionally, could you explain bitmap printing?

Nick: Our projects have focused in 3 areas: 1. Advancing bitmap printing techniques and workflows 2. Multimodal data fusion for medical applications focused on soft tissue 3. Parametric design for prosthetics.

Bitmap printing is the ability to print directly from a stack of bitmap images. It’s very much like how an inkjet printer works, except in this case the pages are layers. It’s very simple and leverages how a polyjet printer natively works, we are are essentially bypassing the internal slicer of the printer and sending our own slices. These slices control the location of every single droplet. However, this means that we need to define each droplet and that’s harder. Most, if not all, modeling software used for 3D printing utilizes a surface mesh-based modeling paradigm which represents objects by their boundaries, ignoring internal volumetric data. In many applications, such as medical and structural, volumetric information is valuable to be able to capture. So, in order to bitmap print, we use a different paradigm for modeling called ‘voxel modeling’ which models volumetric data. The result allows us to directly translate image-based data, such as DICOM data, into a 3D printed object with little to no loss of data. The other major benefit of voxel modeling and bitmap printing is the ability to fabricate a gradient of material by controlling droplet mixing ratios through dithering.

1. Currently, the major problem with bitmap printing is that in order to do it, you have to essentially create the software to create the files yourself. There are no off the self-workflows which allow someone one to simply create a bitmap print. So, we have been working to combine our different workflows and create new workflows which help us to better understand and tune our ability to create bitmap prints. There are many factors which need to be taken into consideration and which still need to be worked out. Material mixing of mechanical and color properties are still hard to predict and we often have unexpected results. This is because of the sheer number of calculations a computer needs to do to digitally preview and predict how a print will turn out. Our major focus is geared toward quantifying color based mixing ratios with translucency and qualifying mechanical properties as they vary across a gradient.
2. Multimodal data fusion is an inherently simple part of voxel-based modeling. Voxel modeling utilizes compositing instead of booleaning, which makes the combining of numerous complex data sources simple and computationally inexpensive. Furthermore, voxel modeling allows for the ability to volumetrically change color without modifying the form. These factors allow for us to combine many data sources, both bitmap and vector based, in one model such as MRI, fMRI, PET, MEG, Tractography, and implanted devices. Our work has focused on developing the workflows which allow us to control each data source individually while combing all the sources into a single holistic model.
3. Parametric design is a very common modeling technique in both architecture and is the ‘bread and butter’ of stop motion animation. We have been working to create ‘rigged’ modeling software which allows for the facile manipulation of complex forms over time. Our focus with this work has been geared toward designing patient specific prosthetics which need to be modified over time. For this work we have combined medical data with standard stop motion animation software to essentially create a stop motion series of models for human prosthetics.

The project could help treat pediatric epilepsy (photo credits: Nicholas Jacobson and Robert Ducey)

Rob: After our initial meeting it became clear that there was a lot to learn from each other about how to design volumetrically for printing. We recognized some inherent qualities of the medical imaging process that leant itself to volumetric data. At the studio we had been developing ways of taking volumetric data from a visual effects workflow and converting that to multi-material printing. The key to that process was “bitmap printing” or, as it has now come to be known as “voxel printing”. Up until this point we had only concerned ourselves with the exterior appearance of a printed object and the outside layers that affected this. Our partnership with Fraunhofer and their product Cuttlefish and was critical to mastering that. This gave us a familiarity and access to printing from slices, which is essentially what bitmap printing or voxel printing is. We engineered a process to author slices from our volumetric data that we could print. Nick’s work was especially interesting in the various modalities of medical imaging he was working with in DICOM format. Those were directly analogous to slices and were fascinating to see in printed form. We knew there could be some obvious ways to integrate that data into a VFX pipeline and generate some cool looking prints. From there we began discussing ways to feed that back into the areas of research Nick and the team at the University of Colorado were exploring. This led to other discussions of leveraging techniques and methods well explored for animation and visual effects that we might apply to ongoing medical research. A unique crossover for LAIKA is the fact that we require similar precision, scales, and levels of detail in the parts we engineer and generate that you might expect in medical applications. Additionally, we bring some insights about how to make something visually compelling.

3DN: What applications do you envision for this?

Nick: We envision many applications for this and have only just scratched the surface! For now, we are focused on two areas: Pediatric Epilepsy Surgery and Cleft Palate. Pediatric Epilepsy Surgery is perhaps the most powerful application of bitmap printing for presurgical planning and intraoperative guidance. This is because epilepsy is mostly physiologically based and in order to plan for treatment surgeons review many 2D based volumetric data sources, individually and sequentially, to form a mental understanding of the patient. We are focusing on combining the numerous, and growing, available datasets into one holistic 3D model to be used for planning and guidance. We plan to start a prospective trial before the end of the year to evaluate the effectiveness of having such a model for treatment planning and post operative outcomes.

Our work with cleft lip and palate is focused on a presurgical technique called Naso Alveolar Molding. This is a common treatment for infants born with cleft lip and palate that starts almost immediately after birth. The treatment involves placing an acrylic device onto the palate which directs growth with the goal of closing the cleft. The result eliminates a surgery and makes all subsequent surgeries more successful. However, adjustments need to be made weekly, requiring families to visit the clinic weekly for 3-9 months. This is hard, even if you live close. However, there are few providers and access to this treatment is prohibitive to the vast majority of patients. So, our work is focused on providing a 3D printed solution which would allow for patient to receive a set of models in the mail, eliminating the need for weekly office visits. This would allow for anyone with the ability to take a dental impression the ability to receive this treatment.

Rob: Nick and his team are driving the research about how to apply what we’re learning in the field of medicine. For the studio, this has helped clarify both how we can technically design using voxels and about how biology designs form and function volumetrically from the perspective of medical experts. Applying those concepts of biology to design is something additive processes are uniquely poised to exploit.

The process could help with Naso Alveolar Molding for cleft palate patients, a condition wherein holes are present at birth in the palate (soft and hard) or in the lip (photo credits: Sunrise Orthodontics)

3DN: Any last words for our readers?

Both: If you would like to learn more about the project or about us in general, be sure to attend AMUG 2023. We will be giving the keynote speech, “Collaborations Between an Animator, an Architect, and a Surgeon: The Keys to Impactful Innovation in Medicine” on Tuesday, March 21st. If you haven’t already, you can register for the conference HERE.

What do you think the work that Nick and Rob are doing? Do you think 3D printing conferences play a key role in encouraging collaboration in the industry? Let us know in a comment below or on our LinkedIn, Facebook, and Twitter pages! Don’t forget to sign up for our free weekly Newsletter here, the latest 3D printing news straight to your inbox! You can also find all our videos on our YouTube channel.

*Cover Photo Credits: Nicholas Jacobson and Robert Ducey