Robotic hand and human hand reaching toward each other – AI in statistical programming

AI in statistical programming: better work or fewer jobs?

AI can do some routine work in statistical programming, but will it be used to eliminate jobs in the field? At the recent PHUSE Single Day Event in Basel, some speakers saw room for more creative work, while audience members posed questions about jobs. Which approach wins will depend on what employers do with the time that AI tools save.

Will AI replace statistical programmers in pharma?

AI in statistical programming can speed up routine coding and drafting, creating room for more demanding scientific work. Turning that possibility into better jobs depends on what employers do next. They can use the extra capacity to strengthen research, or look for ways to produce the same outputs with fewer people.

That choice gave urgency to discussions at the PHUSE Single Day Event in Basel on 17 September 2026, which entimo attended. Held at the Novartis Campus, the event explored Reimagining the Clinical Data Scientist Role: Future-Proofing Skills and Partnerships. Practical demonstrations sat alongside questions about professional skills, accountability and the organisation of work.[1]

Pharma speakers were broadly optimistic. They described AI taking on repetitive activities while statisticians and programmers spent more time on problems requiring experience, creativity and judgment. Some audience questions were harder to settle. Where would the jobs be as more work became automated? And if a person still had to check and approve the result, how much time had really been saved?

The answers are connected. Before companies can decide what to do with greater productivity, they need to understand what AI actually delivers—and what people must still contribute.

The symposium exceeded every expectation, both in content and in the quality of professional engagement.

How is AI changing statistical programming?

AI can provide a faster starting point for work that still needs professional judgment. It can help draft code and analysis documents, giving specialists something to examine and improve. Whether that saves time overall depends on the quality of the draft and the effort required to bring it to completion.

There is evidence of the opportunity in software development. A study of the effects of AI on skilled work published in Management Science in February 2026 combined three randomised workplace experiments involving 4,867 developers. It estimated a 26% increase in completed tasks among those using an AI coding assistant. Results varied across the experiments. The study measured general software development, so its findings cannot establish equivalent gains in regulated pharmaceutical work or show that fewer employees are needed. [2]

Clinical research offers a more targeted example. Pfizer researchers evaluated an AI system across 71 statistical analysis plans. Their 2026 Clinical Trials paper reported average first-draft generation times of 1.0–3.4 minutes, compared with a conventional first-drafting benchmark of one to two days. These were initial drafts. The study did not establish that reviewed and approved plans could be completed in minutes. [3]

For a statistician, the attraction is easy to see. Time spent assembling an initial plan could be used to challenge an assumption or discuss an uncertain endpoint. But that opportunity begins only when the draft is useful enough to work with.

This is where the Basel discussion about human approval becomes central to the productivity question.

Person playing chess against a robotic arm – human review of AI-generated code

Why does AI-generated code still need human review?

Human review is needed because AI can produce an output that looks convincing while missing something important to the study. That does not make assistance pointless. Its value depends on the total effort needed to reach a dependable result, including the time spent checking, correcting and validating it.

Johnson & Johnson’s Guillaume Abgrall, Principal II Statistical Programming Lead, and Tadeusz Lewandowski, Head of Data Engineering, demonstrated this approach in Basel. Their R-code generation workflow used reporting specifications and Analysis Data Model (ADaM) metadata. Human checkpoints covered specification interpretation, variable analysis, implementation planning and code review. Their presentation stated the principle plainly:

The human is the pilot. Keep it that way.

Guillaume Abgrall and Tadeusz Lewandowski, Johnson & Johnson

A separate example showed why those checks matter. Using simulated data, Novartis’s Tiina Kirsilä demonstrated an application that worked and passed its tests. Its laboratory charts nevertheless used inconsistent vertical scales across patients. Recognising the risk of a misleading display required knowledge that the automated checks had missed. [5]

A good draft can remove hours of routine production. A convincing mistake can consume those hours again. Someone must understand what an analysis needs to show and whether the output represents it properly.

The regulatory direction reinforces that responsibility. In January 2026, the US Food and Drug Administration (FDA) and European Medicines Agency (EMA) published joint principles for good AI practice in drug development. These call for assessment proportionate to risk, traceable information and evaluation of the system as a whole, including interactions between people and AI. Faster generation therefore has to be considered alongside the evidence needed to trust and use the result. [6]

The case for human expertise is strong. It still leaves the audience’s employment question unanswered.

What does AI mean for statistical programming jobs?

AI does not dictate a single employment outcome. A pharmaceutical company could use the same team to run more studies or explore more analytical questions. It could also retain experienced reviewers while reducing recruitment. When the technology creates additional capacity, management decisions help determine who benefits from it.

Evidence from the wider labor market explains why opportunities for people entering the profession deserve attention. A Stanford working paper revised in August 2026 examined US employment among workers aged 22–25 in occupations exposed to AI. Employment stood 19% below the level it would have reached had it followed the trend among less-exposed peers. The difference arose mainly through weaker hiring. The researchers found no evidence of widespread, economy-wide displacement and cautioned that the pattern did not establish that AI caused it. [7]

A different setting produced a different picture. A Danish working paper dated March 2026 used records through the end of 2024. It found no differential employment or hiring changes at workplaces encouraging chatbot adoption, including among younger workers. Work was being reorganized without corresponding reductions in employment. These studies cover different places and periods, and neither measures pharmaceutical statistical programming specifically. [8]

For pharmaceutical employers, that uncertainty makes their own intentions more important to explain. Employees should be able to understand whether greater capacity will support more research, change the composition of teams or reduce opportunities to enter the profession.

A company considering fewer junior appointments also needs to ask where its future reviewers will come from. If AI produces the first implementation, a new programmer still needs opportunities to understand and question it. They need to learn why a seemingly minor decision can change an analysis.

The hopeful future described in Basel depends on preserving that expertise and giving it more room to contribute. Supporting it also places a responsibility on the companies building the tools.

How is entimo approaching AI in regulated work?

entimo’s approach is to build AI around the people responsible for regulated work. Automation should make routine tasks easier and leave more room for analysis and creative problem-solving, while keeping human judgment and approval meaningful. That is the balance the company is pursuing as it carefully introduces AI into its entimICE FastTrack platform.

The aim is a workflow in which users can understand what AI contributed, how the result was checked and who approved it. Human review, validation and traceability give substance to that responsibility. They help teams assess quality as production becomes faster. [6]

This reflects the emphasis on human expertise heard at PHUSE Basel. Faster delivery should strengthen what people can achieve while preserving their responsibility for the work. Employee development therefore belongs in the implementation plan from the start.

These principles are built into entimICE FastTrack, our platform for traceable, audit-ready clinical data analysis, and are part of everything entimo offers.

What skills will statistical programmers need?

Statistical programmers need strong coding, statistical understanding and the ability to assess AI-generated work. Clinical knowledge, validation and communication are equally important. For entimo, the direction suggested by PHUSE Basel is a broader role: helping define the work, establishing confidence in the result and explaining its significance to colleagues. [1]

That starts with understanding code well enough to read, test and debug it, including code generated by AI. Depending on the working environment, programmers will need to maintain or deepen their expertise in SAS, R or Python. Clinical data standards are part of the same foundation. A programmer must understand how a dataset was derived, why a population was selected and whether the output answers the intended question.

AI literacy builds on that knowledge. It involves giving precise instructions, recognising when an answer rests on an unsupported assumption and designing a check that could expose the problem. The Johnson & Johnson workflow made these responsibilities visible at several points in the task, rather than leaving them all to a final reviewer. [4]

Technical judgment also has to be communicated. A programmer who identifies an ambiguous specification needs to explain why it matters and work with statisticians or clinical colleagues to resolve it. Cytel’s Hannes Engberg Räder made a related point for trial statisticians. His examples covered AI support with protocols, code and inconsistencies between study documents; scientific judgment and leadership of discussions remained with the statistician. [9]

That is where the promise of greater creativity becomes practical. It might mean finding a clearer way to present a result, improving an analysis or fixing a recurring problem in how data move through a study. Kirsilä’s presentation connected the satisfaction of programming with problem-solving, ownership and building something worthwhile. Those contributions can remain valuable as the mechanics of writing code change. [5]

For entimo, this is why investment in AI should go hand in hand with investment in employees. Experienced programmers need time to learn new tools and broaden their responsibilities. Junior colleagues need supervised work that teaches them why an analysis is trustworthy.

The Johnson & Johnson presentation offered one practical example: reviewing generated R code had become an unexpected learning opportunity for programmers unfamiliar with the tools. [4]

A promise of greater creativity will feel thin if every hour saved is immediately filled with another production target. Employees should be able to see what they will have the opportunity to learn and which responsibilities they can grow into.

When AI saves a programmer a day, someone decides what fills it. Giving that person the opportunity to investigate a difficult question, improve an analysis or develop a better way of working would make the promise of greater creativity tangible.

That is the promise pharma should be asked to keep.

References

All links in the reference list open in a new tab.

  1. PHUSE. Basel, Switzerland — Single Day Event: programme and presentation archive. Event held 17 September 2026. Accessed 8 October 2026.
  2. Cui KZ, Demirer M, Jaffe S, et al. The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers. Management Science. Published online 27 February 2026. doi:10.1287/mnsc.2025.00535.
  3. Landman R, Emir B, Zhang R, et al. Leveraging generative AI to transform statistical analysis plan authoring in clinical trials. Clinical Trials. 2026;23(4):383–392. doi:10.1177/17407745261422365.
  4. Abgrall G, Lewandowski T. Bringing Mock Shells to Life (PDF). Presentation at the PHUSE Single Day Event, Basel, Switzerland. 17 September 2026.
  5. Kirsilä T. The Best Part of Programming Was Never Typing Code (PDF). Presentation at the PHUSE Single Day Event, Basel, Switzerland. 17 September 2026.
  6. US Food and Drug Administration; European Medicines Agency. Guiding Principles of Good AI Practice in Drug Development (PDF). January 2026.
  7. Brynjolfsson E, Chandar B, Chen R. Canaries in the Coal Mine? Six Facts about the Recent Employment Effects of Artificial Intelligence. Stanford Digital Economy Lab. Working paper. Revised 12 August 2026.
  8. Humlum A, Vestergaard E. Still Waters, Rapid Currents: Early Labor Market Transformation under Generative AI (PDF). Working paper. 13 March 2026.
  9. Engberg Räder H. Working Smarter with AI: The Evolving Role of the Trial Statistician (PDF). Presentation at the PHUSE Single Day Event, Basel, Switzerland. 17 September 2026.

In this article

  • Will AI replace statistical programmers in pharma?
  • How is AI changing statistical programming?
  • Why does AI-generated code still need human review?
  • What does AI mean for statistical programming jobs?
  • How is entimo approaching AI in regulated work?
  • What skills will statistical programmers need?

Planning AI in your statistical programming team?

Discuss with us how to implement AI in a way that your statisticians, programmers, and auditors can rely on.

Contact us

More blogs

All articles

Details
Published: October 15, 2026
FDA "Advancing Real-Time Clinical Trials" graphic with the US Food and Drug Administration seal – real-time clinical trials and SCEs

Real-time clinical trials: what they mean for SCEs

The FDA announced plans for speeding up the clinical trial process by enabling real-time transmission of data.

What is the FDA real-time clinical trials pilot?

In late April 2026 the FDA unveiled a pilot for real-time clinical trials (RTCTs). Two early studies – AstraZeneca’s TRAVERSE (Phase 2 lymphoma) and Amgen’s STREAM-SCLC (Phase 1b lung cancer) – are already streaming pre-specified efficacy and safety endpoint data directly to FDA reviewers “in real time” as data accrue. In these proof-of-concept trials, the FDA has established cloud-based data feeds via Paradigm Health and has “received and validated” ongoing signals from the sites. The goal is to collapse the long lag time between phases and move toward continuous trials. As former FDA Commissioner Marty Makary noted, roughly 45% of the time between Phase I and submission is currently “dead time” waiting on analysis and paperwork. Real-time trials aim to cut that dramatically by letting FDA scientists see, e.g., tumor responses or fevers as they happen. “When a patient develops a fever, or a tumor shrinks, FDA regulators can see in the cloud, in real time, exactly what is happening,” said Makary. An FDA Request for Information (RFI) has just opened to design a broader AI-enabled pilot this summer; comments are due by May 29.

  • Goals: Eliminate the hiatus between phases by sharing endpoints continuously. Shorten review timelines (“shave years off approval,” according to one FDA official) without compromising safety.
  • How: Use AI and cloud tools to stream key data. The FDA’s Chief AI Officer Jeremy Walsh emphasizes partnership: “real-time trials are not just technically possible; they are transformative for the entire clinical ecosystem”.
  • Status: Two current cancer trials to show feasibility. The FDA plans to invite more sponsors into a pilot by late summer.

What statistical and governance questions do real-time trials raise?

Observers caution that this shift raises new statistical and governance questions. Rolling data feeds mean multiple looks at outcomes. In particular, continuous AI models can “drift” an endpoint definition mid-trial, undermining the statistical plan. Every AI update needs formal versioning: log each model change, document its effect on the primary endpoint, and adjust the analysis plan accordingly. Otherwise the data FDA sees today may be based on a different algorithm than tomorrow’s. Legally, the FDA assumes a quasi-DSMB role here. If the FDA catches an early safety signal it could intervene, so the roles of sponsor, CRO, FDA and (if present) DSMB must be clearly defined. In short, novel SOPs will be needed for data sharing, and SCEs must be able to audit every streamed update against the trial’s master protocol.

What changes with the UK clinical trial reforms?

Meanwhile, in the UK new CT laws (April 2026) enact the most sweeping change in more than 20 years. Highlights: introducing “notifiable trials” for lower-risk studies. Such trials get automatic MHRA approval as long as ethics clearance is in place, greatly speeding startup. Safety reporting rules were relaxed: sponsors are no longer required to report SUSARs directly to sites and IRBs, and urgent safety measures can be notified within seven days (vs. the old three-day deadline). MHRA has issued draft GCP guidance in parallel. Altogether the UK changes aim to cut red tape and boost research competitiveness. (MHRA grew staff by 20% and cleared its Brexit backlog in preparation.) These reforms, like the FDA’s, push toward a more adaptive, risk-based framework – for example, by allowing earlier, streamlined trial starts for simpler studies.

What is entimo's experience with faster regulatory data exchange?

Entimo AG has provided GxP-compliant Clinical Data Repositories (CDR) and Statistical Computing Environments (SCE) to the pharmaceutical industry for more than two decades, enabling sponsors to excel in submissions to regulatory authorities towards timely market approvals. During the COVID pandemic many vaccine trials where processed with entimICE solutions, exercising a much more frequent data exchange with the FDA than ever before. From this experience, the authorities and industry derived what can be done to fast-track submissions to bring medicines to patients faster, and entimo has a good understanding of what is required for software vendors to support this acceleration.

How should SCEs prepare for real-time clinical trials?

These shifts will have direct impact on statistical computing environments (SCEs) in industry and CROs. To handle real-time/continuous data and more adaptive designs, modern SCEs like entimICE FastTrack should be prepared as follows:

  • Cloud-based, automated pipelines: Ensure the SCE can ingest streaming data feeds. Modern SCE platforms already allow scheduling of analyses to run automatically when new data arrive. Partner with IT/EDC teams to connect site databases (or a hub) into the SCE data repository in near-real-time.
  • Robust version control and audit trails: Use strict versioning for all analysis code, datasets, and AI models. Every change, whether a new data lock or an AI model update, must be logged with an identifier and timestamp. A compliant SCE already offers unified version tracking of programs and data; extend this to cover real-time data events (e.g., log which data snapshot corresponds to each analysis run).
  • Dynamic analysis planning: Be ready to pre-specify adaptive rules. For example, if conducting interim looks on a live endpoint, pre-define how often analyses occur. If using AI-derived endpoints, plan how to validate or “freeze” those models during the trial. The SCE should facilitate quick sensitivity analyses: e.g., re-run results under different definitions or excluding recent updates, to check robustness.
  • Enhanced quality controls: Real-time monitoring requires continuous QC. Build automated checks for data quality and consistency (missing data rates, outliers) as data stream in. For example, modern SCE tools can attach checkpoints to alert statisticians if an unusual data shift occurs. Also ensure all data transformations are reproducible: metadata and derivations should accompany raw values.
  • Regulatory compliance and security: Confirm that the SCE meets 21 CFR 11 and other GxP requirements. Real-time feeds mean the SCE will partially parallel FDA’s oversight. Ensure roles/permissions are clear (which user can see what data and actions). Encrypt data in motion to the FDA and maintain logs of what was transmitted. Update standard operating procedures for audits. Inspectors will likely want to see how real-time data were handled.
  • Team training and collaboration: Prepare biostatisticians, data scientists, and programmers for this model. They may need new skills in cloud analytics, streaming data handling, and adaptive methods. Encourage close collaboration among clinical data managers, biostats, and IT. Undertake dry runs or mock real-time scenarios (e.g., feed “live” data into a test SCE) to uncover gaps.
  • Leverage modern SCE features: Consider next-generation SCE solutions built for continuous workflows. For example, some SCEs integrate an analysis workflow that re-executes programs on any data update. Others allow role-based dashboards showing key endpoints evolving over time. Tools that provide real-time visualization of accrued data or interim results can help both statisticians and clinicians stay aligned.

By selecting an SCE ready for these challenges, sponsors and CROs can turn these regulatory changes into an advantage. The underlying message from FDA leaders is clear: clinical development should be “unbroken” and data-driven. SCEs that enable live data sharing, continuous QA, and flexible analysis will be well positioned to meet the new paradigm. In summary, treat the FDA’s pilot and UK reforms as a call to modernize your data infrastructure and processes. Embrace cloud platforms, reinforce versioning/audit mechanisms, and revise SOPs for streaming endpoints. In doing so, your SCE will be ready to deliver timely insights in this emerging, adaptive trial era.

Do you want to find out more?

Reach out to the entimo team to learn how entimICE FastTrack can enable your organization towards real-time data exchange with the authorities.

Contact us

References

All links in the reference list open in a new tab.

  1. US Food and Drug Administration. FDA Announces Major Steps to Implement Real-Time Clinical Trials. Press announcement.
  2. Government Executive. FDA pilot of real-time clinical drug trials using cloud and AI. April 2026.
  3. Regulatory Affairs Professionals Society (RAPS). Euro Roundup: UK clinical trial reforms take effect in biggest regulatory shift in 20 years.
  4. Clinical Trial Vanguard. FDA real-time clinical trials: why AI systems that update themselves are breaking clinical trial design. Opinion.
  5. US Food and Drug Administration. FDA Direct podcast: The power of real-time clinical trials.
  6. US Department of Health and Human Services. FDA announces major steps to implement real-time clinical trials. Press room.

In this article

  • What is the FDA real-time clinical trials pilot?
  • What statistical and governance questions do real-time trials raise?
  • What changes with the UK clinical trial reforms?
  • What is entimo's experience with faster regulatory data exchange?
  • How should SCEs prepare for real-time clinical trials?
  • References

Technology for people

Discover how we build our work around people behind clinical data. 

Contact us

More blogs

All articles

Details
Published: April 30, 2026
Illustration of a sports car with no roads – the clinical research industry buying sports cars and ignoring the roads

The industry is buying sports cars and ignoring the roads

The industry is buying sports cars and ignoring the roads

Agentic AI, data standards, and the infrastructure challenge that no one wants to address.

By Shivoy Pandita

Within the domain of clinical trials, every technological revolution appears to correspond to a pattern. Whenever a new and cutting-edge technology is introduced to the market, vendors produce demos and present at numerous conferences. The industry has generally reacted with excitement, but roughly six months to 1.5 years after release, companies find that their teams have the same discouraging realisation that the technology is only successful in controlled settings and proves problematic in their actual environments. This cycle has already occurred with RPA and first-generation NLP implementations and will continue with agentic AI unless the clinical industry adopts a new strategy. 

More than 40 sessions at the PHUSE US Connect 2026 conference involved educational discussions related to AI, LLMs or agentic workflow. Therefore, it is evident that there is a great deal of focus and attention on transformative technologies. Examples include multi-agent pipelines that produce ADaM datasets, generating SAPs for protocols using LLMs (this is no longer theoretical) and discover outliers in safety data before human intervention.

The sessions on data standards had titles such as "SDTMIG v4.0 — Are You Ready For It?", "Reimagining SDTM Mapping with Agentic AI and Automation" and "Transforming Legacy Data into CDISC Standards" A separate agenda included hands-on workshops detailing how to successfully implement and utilize CDISC 360i metadata-driven automation as well as an extensive CDISC 360i topic over the three-day conference. While these sessions may not have been the most exciting, they sent a clear message: a large portion of the industry continues to struggle with major challenges in terms of finding accurate and structured source data, obtaining consistent, accurate and easily understood metadata, obtaining SDTM mapping that may have been created by someone who was available and for which no documentation exists, and understanding the interpretation of ADaM Datasets that are technically valid, but require special expertise to interpret. Agentic AI requires clean, uniform and machine-readable structured data to provide accurate results. Without this possibility, agentic AI may produce wrong results, while demonstrating great confidence in the responses provided. From a regulatory submission standpoint, this creates an unacceptable level of risk.

 

The hidden cost of legacy clinical data infrastructure

At this year's PHUSE US Connect 2026, a recurring theme across sessions was the challenge of clinical data infrastructure that hasn't kept pace with modern demands. Presentations highlighted how evolving submission requirements, including CDISC standards for metadata and SDTM mapping are placing increasing pressure on existing data workflows. When AI-driven solutions enter these environments, their effectiveness depends entirely on the quality and consistency of the inputs they receive. A well-designed AI pipeline cannot compensate for missing context, ambiguous mappings, or undocumented legacy decisions. The result is that organizations investing in AI tooling often spend significant time on data remediation before realizing the expected efficiency gains, making infrastructure modernization not just a technical priority, but a strategic one.

When clinical data management processes were simple (i.e., you receive programming specs, program code, check code output, submit), investigators could compensate for structural inconsistencies (or "know where the skeletons were buried") by asking people about it. AI cannot call up someone who knows why there is non-standard definition for DOSE based on how it was created historically, nor can AI assume that Tab #2 ("SDTM mapping") of an Excel file ("FINAL_v3_USE THIS ONE") will contain the mapping document for the data from this study. AI can only use what is given to it, and do so quickly (because AI systems are optimized for rapid execution on well-defined inputs, not for reconstructing missing context). Companies spend months building an AI TFL generation pipeline and then a few more manually "cleaning" the input required to run the AI. The efficiency of AI quickly diminishes. Executive teams often ask why their investment has not yielded a decent return. Vendors receive all of the blame for the problems despite they were actually just being given broken inputs to work with in the first place. The problems are historically associated with the integrity of the data used to create the submissions for each trial.

Standards as infrastructure, not compliance

To the industry, achieving CDISC compliance is seen as a regulatory requirement that must be accomplished before submission, must be audited, and must be visible on your QC list. Standards provide the foundational infrastructure upon which an artificial intelligence tools will operate effectively. Even with the highest level of sophistication in building the most intelligent autonomous vehicle, if the roads are not paved, clearly delineated or are set up differently from one city to another, the autonomous vehicle will not operate properly. It will operate with a high level of efficiency when you are demonstrating how it works, but will only operate with a driver present in production mode. CDISC 360i represents the most ambitious effort to create a standard of infrastructure around the linkage of concepts, data collection, analysis and submission in a functional, machine-able chain between the design phase of a study and the final regulatory submission. The session at PHUSE this week on 360i include "Agentic AI-Powered SDTM Automation: A CDISC 360i Metadata-Driven Approach," and "CDISC 360i: From Vision to Implementation — Lessons Learned." These sessions signify progress in the area of CDISC compliance.

However, while 360i addresses how to achieve CDISC compliance going forward, it does not address the problem of remediating over 10 years of disparate data that exists in archives, the data from active studies that were created using older conventions and the operational procedures that companies continue to follow and operate under as if their standards are delivered once the study is completed.

What bridges now and next

At entimo we approach this challenge with a different perspective. The question is not simply about how the data will be organized, it's about ensuring that the correct action takes place at the proper time, consistently, with no human memory required for activation.

Clinical data pipelines are filled with implicit trigger points. When a database lock occurs, it can only be resolved once someone notices the error, notifies another person, who then starts extraction, who then sends the file to programming, etc. If a protocol amendment is approved, there's a need to update metadata, review mappings and flag downstream specifications before being able to execute the correct workflow step. If there’s a Safety Signal in interim data, who will be the one to identify it? The nature of these hand-offs is such that that's where errors accumulate, and delays are compounded. If the trigger is a human memory or an email thread, the AI system has a high likelihood of failing to perform safely. Business Event triggers perform the same function as the implicit hand-offs; replacing them with explicit, defined, and system-wide events. If something meaningfully occurs within the data landscape, a signal is emitted, the next step of the workflow is executed, with full traceability, without any dependency on a person being physically present/available and attentive, at exactly the right moment. This concept isn't new within the software architecture world; however, it is an extremely new concept within clinical data operations and the timing of this shift is purposeful.

AI requires more than clean data to be effective.

What the PHUSE US Connect 2026 agenda is actually telling us

The PHUSE US Connect 2026 agenda represents the pharmaceutical industry at a pivotal moment in terms of advances in AI technology. As AI sessions demonstrated what is achievable, the standards sessions revealed persistent issues with the foundation of the industry. Organizations need to prioritize solutions to these foundational issues if they want to achieve reliable and trustworthy results for both patients and regulators. The teams who will derive the greatest benefit from agentic AI in the next three years will not be those who moved quickly to adopt agentic AI. They will be the teams that take advantage of this time to address foundational issues, consider standards as part of their infrastructure; design processes around events, rather than manual hand-offs, ask themselves “What's a reliable AI operating model?”, prior to asking themselves “How do I find the best AI vendor?”.

This type of internal dialogue is much more difficult than evaluating a vendor's demo. It involves assuming responsibility for many years of accumulated issues and funding what will probably not appear as innovative as it will a building block of successful innovation.

Unless there is an effective road to support a sports car, it will not get you anywhere.


Entimo sponsored the Poster Session at PHUSE US Connect 2026 in Austin last week. We presented on AI-supported validation of open-source software and the next generation of automation through business event triggers. Reach out to us to learn more.

 

Image above courtesy of Gemini.

In this article

  • The hidden cost of legacy clinical data infrastructure
  • Standards as infrastructure, not compliance
  • What bridges now and next
  • What the PHUSE US Connect 2026 agenda is actually telling us

Technology for people

Discover how we build our work around people behind clinical data. 

Contact us

More blogs

All articles

Details
Published: April 09, 2026
Happy couple on a sofa among moving boxes – symbol of entimICE FastTrack and Posit as the ideal couple for R in clinical research

entimICE FastTrack and Posit: the ideal couple for clinical research

The R programming language has become a cornerstone in clinical research due to its extensive range of statistical tools and graphical capabilities.

Why R package management matters in clinical research

The R programming language has become a cornerstone in clinical research due to its extensive range of statistical tools and graphical capabilities. Central to its flexibility and power is the comprehensive ecosystem of packages available through CRAN (Comprehensive R Archive Network), Tidyverse, Pharmaverse, Bioconductor and GitHub. However, managing these packages presents both significant challenges and notable benefits for researchers.

Posit (formerly known as RStudio) is a comprehensive integrated development environment (IDE) for R, offering a range of tools and features that significantly enhance R package management. Its benefits in the context of clinical research are particularly pronounced due to the complex and sensitive nature of the data and analyses involved.

entimICE FastTrack is an integrated Clinical Data Repository (CDR) and Statistical Computing Environment (SCE) platform, designed and built with decades of experience from both entimo and its industry customers. It provides effortless regulatory compliance, while enabling researchers to manage tasks and monitor progress towards successful submissions.

Bringing entimICE FastTrack and Posit together pairs two great systems, and helps researchers make the most of R’s powerful capabilities.

Challenges of using R

Because it is such a powerful tool, R can pose challenges for programmers and scientists who want to get the most out of it. Below are five common ones – read further to find out how Posit and entimICE FastTrack work together to help overcome them.

Version Control and Reproducibility

One of the most pressing challenges in clinical research is ensuring reproducibility. Because R packages are frequently updated, changes in functions or dependencies can lead to discrepancies in research results over time. Maintaining consistent versions of packages across different stages of research and among different collaborators is crucial, but can be difficult without rigorous version control practices.

Dependency Management

Many R packages rely on other packages, creating complex dependency chains. A single update in one package can potentially disrupt the functionality of others. This interdependency requires careful management to avoid conflicts and ensure that all necessary packages work harmoniously together.

Installation and Compatibility Issues

Installing and configuring R packages can be problematic, particularly when dealing with packages that require compilation or are dependent on system-specific libraries. These issues are amplified in clinical research environments where stringent IT policies and varying operating systems are common.

Documentation and Learning Curve

While the breadth of available packages is a strength, it also presents a learning curve. Not all packages are equally well-documented, and deciphering how to use a package correctly and effectively can be time-consuming. This is particularly important in clinical research, where the accuracy and reliability of analyses are paramount.

Regulatory Compliance

Clinical research is heavily regulated, requiring strict adherence to guidelines for data handling, analysis, and reporting. Ensuring that all R packages used are compliant with these regulations and that their usage is documented and validated is a significant challenge.

Benefits of using R with Posit

Adding the strengths of R to the extended possibilities offered by Posit offers numerous benefits to users in the pharma and research sector. Here are seven concrete advantages – and then keep reading to learn more about how these tools mesh with entimICE FastTrack.

Streamlined Package Installation and Updates - Posit Package Manager (PPM)

PPM provides one or more centralized repositories for R packages, which simplifies the process of finding and installing the correct versions of packages. It offers curated and vetted packages, ensuring that the packages are reliable and reducing the risk of installing malicious or buggy software. PPM supports versioned snapshots, allowing users to install packages as they existed on a specific date, which is crucial for reproducibility. Packages can be maintained within the repository, removing dependencies on external third parties (such as CRAN or GitHub).

Enhanced Reproducibility - Project Environments

Posit allows you to create project-specific libraries where each project can have its own set of packages and versions, avoiding conflicts between projects. Packrat and renv are integrated into Posit, facilitating the creation of isolated, reproducible environments. This ensures that analyses can be reliably reproduced by other researchers or at a later date.

Dependency Management and Tracking

Posit can automatically detect and install package dependencies, ensuring that all required packages and their dependencies are available and compatible. It provides tools to visualize and manage package dependencies, helping to identify and resolve conflicts easily.

User-Friendly and Intuitive Interface

Posit’s user-friendly interface includes features such as point-and-click package installation, updates, and management, making it accessible even for researchers who are not programming experts. Access to package documentation and vignettes directly within the Internal Development Environment helps researchers understand and use packages more effectively.

Enhanced Productivity Features

Posit provides advanced script management features, including syntax highlighting, code completion, and debugging tools, which enhance coding efficiency and reduce errors. For researchers developing their own packages, Posit offers robust development tools, including testing frameworks and build management, both of which streamline the development process.

Compliance and Security

PPM can restrict access to certain packages and versions, ensuring that only approved and compliant packages are used in clinical research settings. Posit provides audit trails for package installations and updates, which are essential for regulatory compliance in clinical research.

Community and Support Resources

Posit has a large and active user community that runs forums and discussion groups where researchers can seek help and share knowledge. Posit provides professional support services, which can be invaluable in troubleshooting complex package management issues and ensuring smooth operations in a clinical research environment.

Posit and entimICE FastTrack

Combining Posit and entimICE FastTrack brings together the strengths of both environments, with significant benefits for clinical data analysis and reporting.

Effortless Compliance

entimICE FastTrack ensures effortless compliance and end-to-end traceability for all artifacts, including data, programs, images, and large files such as biomarkers. Integrated version management tracks changes, while pre-configured workflows streamline validation and lifecycle management. Seamless integration with Posit keeps all data and program code changes updated in the repository.

Two transluscent chain links set against a black background

Personalized Computing Environment

entimICE FastTrack's intuitive UI enables users to design custom dashboards and context-specific collections of objects of interest. Interactive visual reports track dependencies from raw data to submission-ready deliverables and monitor study progress. Users can complete all R- and Python-based programming tasks using Posit, which can provide a dedicated repository/execution environment per study to ensure compliance.

Shared or Individual Work Areas

entimICE FastTrack supports collaborative work on clinical data without the need for exporting snapshots or cumbersome check-out/check-in procedures. The data repository provides a single source of truth for all data processing steps. Additionally, users can establish individual work areas for exploratory work.

Ready for Exploratory and Submission Workflows

The system supports both exploratory data analysis using a variety of statistical tools, including Posit, and generating submission-ready deliverables through tightly integrated R functionalities. entimICE FastTrack offers clinical data scientists the freedom to explore where appropriate, alongside strict guidance and conformance where necessary.

Language Agnostic

entimICE FastTrack is truly language-agnostic. While fully supporting R and Python through Posit integration, the system also accommodates other toolsets, including industry-standard SAS, its programming language along with tools like Enterprise Guide or SAS Studio. Integration with SAS Viya is currently in development.

Pairing Up

R package management is a double-edged sword in clinical research. The challenges including version control, dependency management, installation issues, documentation, and regulatory compliance make meticulous planning and management necessary. However, the benefits such as enhanced analytical capabilities, reproducibility and efficiency make R an invaluable asset in clinical research. Effective package management practices are essential to leveraging R's full potential while mitigating the inherent challenges, ultimately contributing to the advancement of clinical research.

Using Posit for R package management in clinical research offers numerous benefits. These include streamlined package installation and updates, enhanced reproducibility, effective dependency management, and a user-friendly interface. Additionally, Posit fosters collaboration, enhances productivity, and provides extensive community and professional support. These features make Posit an invaluable tool for managing R packages, ultimately contributing to more reliable and efficient clinical research practices.

The integration of Posit with entimICE FastTrack allows data scientists and biostatisticians in highly regulated environments to enjoy the best of both worlds: They gain the freedom to use a large variety of R packages for exploratory analysis, while ensuring full compliance during submission runs using an exclusive validated R environment. This powerful combination of two tools significantly enhances clinical data science, leading to faster and more successful submissions.

More information

Explore entimICE FastTrack

In this article

  • Challenges of using R
  • Benefits of using R with Posit
  • Posit and entimICE FastTrack
  • Pairing Up

Technology for people

Discover how we build our work around people behind clinical data. 

Contact us

More blogs

All articles

Details
Published: April 16, 2025
Colorful patch of wildflowers symbolizing heterogeneous data – separating data storage and computing in clinical research

Data storage evolves: separating storage and computing

Increasing volumes and changing types of data require new approaches.

By entimo Staff

Separating storage from computation

Considerations of cost and ownership are encouraging companies in pharmaceutical research to separate their data storage from their computation needs. The widespread perception that storage is inexpensive and easily extensible compared to computing power is not necessarily correct. The real deal is that organizations should scale storage and computing separately, as they do not always rise in tandem, and thus the separation makes sense to better provide for savings. Looking at a situation more closely reveals that ownership of data used in analysis and computational processes may be more complex. A computational environment might not be considered the owner of the data it uses, and so that data is stored elsewhere. Similarly, data silos within organizations may be set up such that data cannot be used for downstream processes, so that exports or copies have to be created to perform the computations.

A field of wheat in the sunshine symbolizes homogeneity.

Clinical data used to be more homogeneous.

Clinical data is changing

Another development affecting how organizations evaluate storage and computation is the changing nature of clinical data that are gathered in the course of trials. In previous years, clinical data was primarily stored as SAS datasets natively in file systems, along with additional items derived from or working with those datasets, items such as documents, programs, output, and logs. As data volumes have grown—thanks to more studies, bigger studies, more types of data used in studies, as well as keeping data for secondary use in the active systems—the industry standard has shifted away from clinical/tabular data and toward heterogeneous collections such as x-omics data, biomarker, and healthcare data from wearables. The need for version control and audit trails imposes new and different storage requirements on these kinds of collections, in contrast to SAS datasets in a file system, where one of the main challenges was rapid availability stemming from the need for rapid input-output when working with SAS.

Ownership of data in the repository

The desire for end-to-end traceability along with regulatory requirements have led entimo, in both its products entimICE DARE and entimICE FastTrack, to follow the principle of ownership of the data in their repository. This applies to all metadata, and also to data, which is mostly held in a central file system. Following an evolutionary path, entimICE FastTrack is designed to offer more flexibility and connectivity for external storage. Within entimICE DARE, there is still an option to store clinical data in a database, with entimICE retaining ownership.

Projecting external data into the repository

As the size and nature of data gathered during clinical trials expand and change, the tools needed to analyze and understand that data will are also evolving. Consider the example of a trial sponsor that decides to build clinical data storage for all data that comes in from EDC systems and other sources. While the storage is organized in a fashion specific to the company, it uses approaches that are well understood. Further, the sponsor needs all other tools to interact with the data in this cluster.

Engineers are working to ensure that entimICE FastTrack can adapt to this new environment by projecting data from the storage into its own repository, following a proxy concept and other comparable methods. The software only projects relevant data, and is able to track changes on the projected data as well as mirroring this information in its own traceability records. This capability is important for keeping information needed for compliance all within entimICE FastTrack.

Using a clinical repository approach, entimICE FastTrack can have full ownership and control on dedicated production life cycle areas in the external data. This guarantees the control for production runs within entimICE, where greater oversight of specific workflows is necessary to guarantee regulatory compliance. The file repository as well as the data projection remain transparent for end users. For example, in a study’s development lifecycle area, programs from the file system and datasets from the projection can appear side by side, and can both be accessed and used.

A colorful patch of wildflowers in bloom symbolizes heterogeneity.

Contemporary data is more heterogeneous.

Evolving with heterogeneous data

The combination of these approaches enables entimICE Fast Track to evolve with the changes in how organizations throughout the drug development process are addressing increasing amounts of data not only in increasingly heterogeneous formats but also data sources outside of entimICE.

In this article

  • Separating storage from computation
  • Clinical data is changing
  • Ownership of data in the repository
  • Projecting external data into the repository
  • Evolving with heterogeneous data

Technology for people

Discover how we build our work around people behind clinical data.

Contact us

More blogs

All articles

Details
Published: December 12, 2021
Close-up of a single pencil tip – illustrating how concurrency shapes a statistical computing environment (SCE)

Inevitable choices: how concurrency shapes an SCE

The way software addresses concurrency fundamentally shapes its architecture.

By Douglas Merrill

Concurrency is everywhere

“Concurrency is everywhere in modern programming, whether we like it or not” reads an MIT introduction to the subject. In a typical computing environment, machines and software will be contending with concurrency on numerous levels: from multiple processors on the same chip through multiple pieces of software running at a single workstation up to multiple—sometimes multitudes—computers running on one network.

Developers use techniques such as separate processes, threads, time-slicing, shared memory, message-passing and more to avoid problems such as race conditions, where the outcome of a process is not always the same, depending on uncertain order in which computations are executed.

Why concurrency matters in a regulated SCE

For a Statistical Computing Environment in a carefully regulated sector like pharma development, keeping track of concurrent operations is crucial. Regulatory submissions, and ultimately the safety of new medications, will depend on the integrity of the data being submitted. That, in turn, depends on how well the software can handle demands from many different users.

“Concurrency is all over the place, no matter how you look at it,” says Alexander Lüders, a senior software engineer at entimo. “You can work with a single shared repository, or you can check out sections of data into a sandbox, but the issue remains because you eventually have to merge the checkout back to the repository.“

Why sandboxes and check-outs fall short

As a result, entimo has chosen to work directly with the questions of concurrency. “A sandbox still has the problem of how to resolve conflicts,” adds Marc Jantke, one of entimo’s board members. “You can have situations where more than one user wants to check something back into the master branch, and what do you do then? You still have to work through that question.”

A group of about 20 pencils, each of a different color, form a circle with their tips and the bodies of the pencils running outward to the edges of the photo.

Jantke points to other challenges for systems that involve check-outs and redundant copies. First, making and keeping all of them consumes resources and time, especially in environments where validation may require retaining an entire audit trail of data versions. Further, the data volumes for clinical studies may be large enough that a check-out approach in the computing environment could slow down performance. If an organization had infinite resources, that would not be an issue, but every company faces some kinds of limits, and a sandbox/check-out architecture may reach them more quickly than an approach that works with concurrency directly. Second, the computing required to accommodate check-outs can become a performance question, particularly given the size and relationships in data volumes in modern pharmaceutical studies. Again, this is less of a theoretical problem and more of a practical one, given that even the largest development programs have to work within budgets. “Those are non-trivial challenges,” says Jantke.

“A check-out approach faces the problem of making a consistent copy. That can lead to a lot of blockages for other users or would force us to have everything under version control,” adds Lüders. On a large study or in an analysis involving many users, that could slow such a system significantly.

Dealing with concurrency directly

In short, you can’t get around concurrency, so you might as well deal with it directly. That’s the business case entimo has made, and it has proven to work as our customers are able to collaborate efficiently even with hundreds of users working concurrently on a shared repository.

In this article

  • Concurrency is everywhere
  • Why concurrency matters in a regulated SCE
  • Why sandboxes and check-outs fall short
  • Dealing with concurrency directly

Technology for people

Discover how we build our work around people behind clinical data.

Contact us

More blogs

All articles

Details
Published: June 10, 2021
Small wooden footbridge over a mountain stream – symbol of limited capacity before a 27-fold efficiency gain in clinical data migration

A 27-Fold efficiency gain in data migration

Product development team members at entimo combined process optimizations to generate a 27-fold increase in efficiency. Find out how.

By Alexander Lueders, Douglas Merrill, Vegard Stenstad and Henrik Steudel

A key feature in entimICE is traceability which, among other things, keeps track of which files have been created or updated by a program run. (A program in this context is an end-user business program such as SAS/R, or an entimICE script. The program may have many different versions, with each version reflecting a change made to its source or its parameters.) When a user runs a program, entimICE creates a program execution for the current program version. This execution contains information about which objects have been read or written or deleted. Usually the objects that the program uses are files, but entimICE also keeps track of macro calls and database view usages.

The challenge

One of the entimICE software development teams at entimo had a task to create a tool that would migrate data from an old relational data model to a new document-based data model for a large (1 TB or more) volume of data. Further, the tool needed to be able to validate that all of the data had been transferred from the old system to the new one.

“A program execution trace database is very complex,” says Vegard Stenstad, who led the effort. “There are nested executions, as well as used objects and conflicting executions.” Further, the tool had to be able to stop and resume during the migration. It needed to work as quickly as possible.

Understanding the tool requires looking at the use case and the existing database structure. When an entimICE user runs a program in an area where tracing is enabled, the application records everything that the program does. That includes recording information about the program itself, any files used for input or output, log files, database connections, and more. This data is then stored in a relational database (Oracle).

In a production environment, this relational database can obviously grow very large over time. Further, modern environments produce data in numerous different structures. The program execution trace under consideration here distributes data across multiple independent data stores and works with varied data structures. In this case, the team chose MongoDB because its document-based model is a better fit for execution trace records than a relational database. The migration achieves two important goals; it reduces volume and load on the central classic Oracle database, and at the same time it benefits from MongoDB’s built-in clustering capabilities.  

A simplified schematic view of a relational database.

Fig. 1 A very simplified view of the relational database

Building the tool

One of the target use cases for the tool was a set of roughly five million program executions. Those executions involved an average of 43 objects, so that led to as many as 550 million lookups in the complete task. These significant numbers meant that the team not only had to find a way to do the migration, the team members had to find a way to get the task to run within an amount of time that met business and operational requirements.

The tool accomplishes its goal in five steps:

  • Reading the data from the old database
  • Cleaning the data
  • Transforming the data
  • Persisting the data
  • Validating the migrated data

The last step, validating the data, is crucial because it ensures that the integrity of the data has been kept. Because of the importance of validating data and data integrity, those were the team’s initial priorities and performance took a back seat. As a result, the first implementation was slow: it took up to 14 days to manage a complete migration of around 1 TB of data.

Each step toward the goal had a corresponding part in the tool. First, a pre-processor initialized the conditions that the tool required. The reader and chunk iterator dealt with organizing and reading chunks of data from Oracle, as well as stopping and starting. The transformer did the actual work of transforming the data. The persister was initially not separate, but was split out during the process of optimization. It stored the data to the new database. The validator ensured that all data was transferred correctly.

A table showing the improvements gained from each type of optimization.

Fig 2. Improvements gained by optimization

How does the tool work?

To migrate the data, the team initially built a multi-threaded tool using an iterative chunking approach to the data, and in which each thread is independent, handling its own reading, processing, and writing. There was a synchronized singleton that coordinated among the threads, and distributed the chunks of data to the process.

The team divided the tool into five parts: a pre-processor, a combined reader and “chunk iterator,” a transformer, a persister, and a validator. The pre-processor initialized the conditions that the tool required. The reader and chunk iterator dealt with organizing and reading chunks of data from Oracle, as well as stopping and starting. The transformer did the actual work of transforming the data. The persister was initially not separate, but was split out during the process of optimization. It stored the data to the new database. The validator ensured that all data was transferred correctly.

Why was it important to have a pre-processor? Within entimICE, nested executions could lead to a situation in which some executions could be deleted, so developers made a design choice to clone executions. This approach led to duplications, which should be ignored in a migration. The pre-processor was built to identify all of the clones, leading to a list of top-level executions for the migration, which thus picks up all of the nested executions as well. The pre-processor creates an “ignore list” of the clones, which do not require migration.

The “chunk iterator” was a synchronized singleton that provides each thread with a list of executions to process in chunks of 1000 executions. The chunks were stored in MongoDB, and any errors were noted. When the tool was started, it recorded and stored a max execution ID, so that it would not process any data that was added after the start of the tool. The reader was an extremely simple part of the tool, just requesting chunks from the chunk iterator and calling the transformer to process the chunk.

The transformer translated old entities to new entities; that is, Oracle execution traces were transformed into MongoDB data execution traces. Along the way, it improved the handling of program execution data by removing irrelevant objects, correcting program versions, and so forth. Upon completion of the translation, it added the processed chunk of 1000 executions to the persist thread queue.

The persister was originally part of the transformer, but during optimization the team moved it to its own thread. It persisted data to the MongoDB and marked chunks as completed. It was a very simple and reusable tool, one that used a blocking queue to receive from the processing threads. As long as there was data in the queue and the readers were going, it would keep looping and whenever it found something it would write that to MongoDB.

The final part of the tool was the validator, which was run after the migration was completed. It built a list of program executions in Oracle that should have been transferred to MongoDB and compared that list against what was actually written to MongoDB.

Improving performance

The team found ways to enhance the tool, and wound up improving its performance massively. They employed five major techniques to optimize:

  • Increasing JDBC driver fetch size
  • Bulk writing of data to the new document database
  • Pre-fetching data & using entity graphs when possible to avoid lazy loading
  • Pre-loading commonly used lookup tables
  • Caching of commonly used data

The first optimization concerned the ignore list that the pre-processor was building. That list initially took almost four hours to generate. Delving into the parameters of the processes involved, the team found that the default query fetch size for Oracle was 10, so the pre-processor was going back and forth an immense number of times to fetch small queries. They re-set the size to 100,000. After that change, the ignore list took less than five minutes to generate. That near fifty-fold improvement in the pre-processing had a medium-sized effect on the overall task. The team gained some efficiencies in the validator because it also queried Oracle directly, so setting the fetch size higher improved performance in validation as well.

In a daytime mountainous landscape a robust railway bridge crosses a steep gorge. Trains that cross on it can carry many more people than the wooden bridge depicted above.

After the fetch size was increased, writing to MongoDB became the bottleneck for generating the ignore list. The tool was sending what to write as a task list, but the actual writing was still happening one by one. Changing the method from “saveAll” to “insert” forced bulk writing, which removed writing speed as a constraint. As part of this second optimization, the team also moved persisting to its own thread. When combined with bulk writing, time required for persisting took dropped from 90-120 seconds down to 2-6, depending on the chunk size. (The times discussed in this article are all related to the test hardware and could be changed by using production hardware. The conclusions for the configuration and software usage remain valid, and the points about optimization can be generalized beyond any particular configuration.) The team decided this was sufficiently efficient. These optimizations together also had a medium-sized effect on overall performance.

The third optimization turned out to be a game-changer for the team. In the original design, the tool looped over program executions and loaded them one by one, also loading any nested executions. The team changed to bulk loading parent executions along with the first level of nested executions. “Fetching them for up to 1000 executions at the same time, which is a maximum set by a limit within Oracle, we also introduced EntityGraph, which tells part of the Oracle software that it should pre-fetch child entities and grandchild entities when it is loading a program execution,” explains Alexander Lueders, who was part of the team that developed the tool. Memory management became a problem with the pre-fetching, so the team reduced the size of each chunk. Those changes in design reduced migration time from 14 days to four days. The team had moved the tool much closer to the performance needed for practical use. “That was a huge impact,” adds Alexander.

The fourth optimization involved pre-loading data. The team introduced pre-loading because when the tool fetched parent and child executions, it wound up getting clones. The solution was to find the original and pre-load that execution. Taking this approach also reduced the query to only the data they needed and not the entire entity, so the tool could afford to store it in memory. The data was pre-loaded during initialization and a map was built in memory. Building the map took about 30 second, and saved millions of queries. This was cleaner and more efficient, but with a minor effect on overall performance.

The fifth optimization brought another big jump in performance. “We had two lookups – before and after the execution – for the version of for every object that was used in an execution,” explains Henrik Steudel, another team member. “With an average of 43 objects per execution that added up to a total of 550 million lookups for our database of 5 million executions, and because of the massive amount of versions pre-caching was not feasible.” The team chose to use Caffeine Cache with Spring to implement a caching strategy. The lookup time for each one dropped from 1.6ms to 0.3ms, a five-fold improvement. Those accumulated gains dropped the migration time from four days to half a day. The tool was now processing about as fast as it could write to MongoDB.

With its process of optimization, the team improved performance by a factor of roughly 27, going from 14 days for a migration to 12 hours.

Conclusions

What are the general lessons learned for optimizing and improving performance?

Understanding the data and data structures is key to optimizing. “Be sure to know the settings and limitations in the underlying technology,” says Vegard. “In this case, key parameters turned out to be the Oracle fetch size, IN query limit, and MongoDB saveAll vs insert.”

Using the right tool for the situation and combining multiple optimizations led to huge benefits. Analyzing required entity data that was needed for loading revealed ways to reduce overhead. Optimization can be a trial and error process that takes time.

Working on real data demonstrated how large datasets that grow over time tend to accumulate anomalies that have to be considered during migration. There’s nothing like real-world data to bring out corner cases.

By the end of the process, the team had delivered a migration tool that was showing significant improvement from its first iteration. The previous system could be migrated to an underlying MongoDB system without affecting the rest of the entimICE system, gaining significant performance improvements in the overall system and ensuring compatibility with future updates or service packs.

* Upper image by Wikimedia user Famberhorst under a Creative Commons Attribution-Share Alike 4.0 International license. Lower image released to the public domain by its creator.

In this article

  • The challenge
  • Building the tool
  • How does the tool work?
  • Improving performance
  • Conclusions

Technology for people

Discover how we build our work around people behind clinical data.

Contact us

More blogs

All articles

Details
Published: December 18, 2020
Male peacock displaying its colorful tail feathers – entimo CEO blog on how a small company serves big pharma

The most colorful peacock: how a small company serves big pharma

How does the CEO of a respectable IT company end up writing about birds, fairy tales and greek mythology in his very first company blog article?

By Marc Jantke

I will take you to the answer of this question, but we have to start at the beginning.

About feathers and ambitions

entimo is providing software products and professional services to the pharmaceutical industry since its beginning back in 2003. Our products are used in the areas of Statistical Computing, Clinical Data Management and Standards Management by some of the top 10 pharma and many other small to medium sized companies around the world.

How can a company with 50 employees serve the global leaders in their core business processes?

The answer is simple. entimo is a reliable business partner to all customers. Our relations are lasting for decades and are based on a solid foundation of trust. As the saying goes trust is something you earn and not something that is given. entimo is ambitious about constantly improving towards perfection. This applies to our processes, quality, transparency and our own culture. That may be one of the reasons which earned us trust from our employees. Many of them have been with us right from the start, many more joined us during the journey and most of them stayed with us until today. The consistency in our teams, the agility in our minds and the ambitions in our goals seem to have earned us the trust of our customers as well.

And what about the feathers?

We are in the middle of developing the next evolution of our products and it will for sure revolutionize the market. It has the power to change the vision of the pharmaceutical research and development process, away from individual and monolithic software products with clunky interfaces towards a holistic ecosystem that puts the clinical data and the people who work with it in the center. 

entimo is looking for additional ways to bring interested parties closer together to involve you into the development of our future. This is where the idea of this blog comes in, with the goal to establish a new communication channel between our company, its customers and any other interested party in the life sciences industry.. The thing with blogs is: they have to be attractive in order to be read.

So if you have never written a blog before and are striving for the creation of a perfect blog in order to attract your readers, you start reading blogs. Then you read blogs about blogging and you end up consuming questionable expertise about how a perfect blog should look like. You learn a lot about golden rules, do's and don'ts and the importance of titles. Finally you stumble across advice on how to find the perfect title for your blog posts and they tell you that the most colorful peacock draws the most attention.

And here we are now in the middle of this article and you followed along because the colorful peacock drew your attention. But there is more to discover.

The sun shines brightly on a duckling, not ugly, on a bank of some body of water amidst grass and clover blooms.

About the ugly duckling

Catching someone's attention with an attractive title is not enough to make a blog post worthwhile. It should tell you something of interest, inspire you or at least be entertaining. At best, all of the above.

Something inspiring

Now entimo is taking its first steps in the world of blogs. Thinking of a company full of software developers with lots of coding, coffee and pizza, certainly does not strike the mind with the picture of a colorful peacock. You might have rather a flock of pigeons or sparrows in mind.

The analogy that came to my mind, was "The Ugly Duckling", a fairy tale from the Danish author Hans Christian Andersen first published in 1843. It is a heart-warming story about a little bird, raised by a duck but rejected from siblings and other animals because it is such an ugly duckling. After a winter of loneliness and despair, the little ugly duck has grown but still feels miserable and tries to end its life by getting killed by a flock of beautiful swans. To its surprise the swans take the duckling in, as one of them and the bird finally realizes it has been a swan itself all the time. The fairy tale ends with the swan flying away with its new family.

We are the swan

How does this relate to a software company getting familiarized with the approach of writing a blog?  One might argue, writing software and writing blogs is technically close by and should be a natural fit. Having done both myself now, I can tell you: It is not. This feels like comparing a bookbinder with a poet. Nonetheless, there is a deeper truth in it, as we at entimo have stories to share with you and those stories come from our professional lives in a software company. So, in the greater sense are we the swan of this story, recognizing at the end that we have been bloggers for a long time, but simply never realized this and thus never lived up to it, sharing the stories which could have been shared long ago.

About phoenix and the ashes

Until this point I have been talking about the most colorful peacock and the ugly duckling and explained how these birds relate to entimo as a company, entering new ways to get engaged more closely with our customers. And if you think, this is enough of questionable analogies between birds and software specialists... there is one more:

We are the phoenix

This bird from the ancient Greek mythology that was told to die in flames and combustion just to be reborn from its own ashes, has made its appearance in all kinds of art over all times.

A fantastical image of a phoenix rising above a silhouetted landscape into a fiery sky at sunset.

Shakespeare wrote about it in Henry VIII,  as well as J. K. Rowling in Harry Potter. Queen honored the bird in their band logo and Conchita Wurst won the Eurovision Song Contest 2014 with the song "Rise Like a Phoenix". And even modern computer games like Starcraft 2 and DOTA 2 make use of this mythological figure.

The phoenix appears often as symbol of renewal and reinvention. And this applies to entimo just as much. With the switch to agile methodologies a few years ago, we adopted a mindset that constantly strives for renewal and reinvention in order to keep the wheels of improvement turning. The development of our future platform as a natural evolution of our existing products is just another example for this.

The rebirth of the phoenix is associated with an end and a new beginning, giving room for new ideas, making things better than yesterday. This is the path to remain successful in any business and our customers are adopting this philosophy just as we are.

And this is it. When you want to hook people on your blog, find catchy titles, appealing pictures and a worthwhile topic. Fill it with some random facts and the information you really want to share. Of course, all of this has to be polished by a conclusive summary and a call to action. 

TL;DR

Our goal for the first entimo blog article is not to teach you a lesson about birds. It is to share some insights about who we are as a company and where we would like to go in the future. And most importantly, how we hope that this blog can help us to take you along on this journey.

If you are interested in reading more about what we are doing here at entimo, besides ornithology, you are invited to drop a comment and make suggestions for any future topics you would like to hear more about.

Less bird-oriented proposals are most welcome.

May the upcurrents carry your safely through these stormy times,

Marc Jantke (CEO)

In this article

  • About feathers and ambitions
  • How can a company with 50 employees serve the global leaders in their core business processes?
  • And what about the feathers?
  • About the ugly duckling
  • Something inspiring
  • We are the swan
  • About phoenix and the ashes
  • We are the phoenix
  • TL;DR

Technology for people

Discover how we build our work around people behind clinical data.

Contact us

More blogs

All articles

Details
Published: August 18, 2020