42 IBM Watson Health: Managing Partnerships to Improve Personalized Medicine

By: Lauren Jagus

Watson, an artificial intelligence machine developed by International Business Machines (IBM) Corporation, was heavily promoted by the company as being able to revolutionize the medical industry.¹ However, approaching on its ten year anniversary, Watson has failed to deliver on this claim. In September 2016, the partnership between a leading cancer institute, MD Anderson Cancer Center, and IBM that was created with the mission of eradicating cancer through improving personalized medicine has been put on hold.^2,3 This decision was made public prior to the release of an audit report of MD Anderson, indicating that the Cancer Centre had paid IBM $62 million for the project that was originally estimated to cost $2.4 million.⁴ Speculation within the industry formed stating that Watson was recommending “unsafe and incorrect” cancer treatments.⁵ This ultimately led to the general manager of IBM’s health division, Deborah DiSanzo, stepping down from her position.⁶ Despite this failure, the technology and medical industry continue to form partnerships on the basis that artificial intelligence can deliver on its claims.⁶ Companies need to create transparent partnerships and advertisements to improve the outcome of successive projects.

International Business Machines Corporation

Charles Flint and Herman Hollerith founded the Computing-Tabulating-Recording Company (C-T-R), on June 16th, 1911 (Exhibit A).⁷ The C-T-R emerged with the development of their tabulating machine that could count and track information.⁷ In 1914, Flin and Hollerith recruited Thomas J. Watson Sr. to work as the general manager of the company.⁷ Ten years later, Watson renamed the C-T-R to IBM to redefine and expand the company’s potential applications within information technology.⁷ Watson, an important employee for IBM, said, “The future of this business, gentlemen, and our future success, depends to a great extent on the progress we make along the lines of development, and along the lines of expansion into new fields and into lines we are not touching today… We must progress and we can’t do it any other way. We can not stand still on development work”.⁷

IBM continues to be a prominent producer of hardware and software services.⁷ Their most notable products include the automated teller machine (ATM), the magnetic-core memory and the floppy disk.⁷ In addition, Arthur Samuel, a computer scientist at IBM developed, helped develop machine learning and artificial intelligence.⁸ It was in 1959 that Samuel demonstrated that computers could be programmed to play checkers and learn from their past games to improve their abilities.⁸ This led to the company developing the chess-playing machine Deep Blue.⁹ The programming abilities of Deep Blue was successful in defeating the chess master, Garry Kasparov, in 1997.⁹ Since then, IBM has devoted many resources to machine learning.^{9, 10}

In 2004, IBM developed a new challenge to improve information technology.¹⁰ This challenge was question-answering (QA) technology, which would benefit many industries.¹⁰ Similar to Deep Blue, IBM gained worldwide attention as their computer scientists worked on creating a computer system tackling QA technology to compete on the game show Jeopardy!.^{10, 11} David Ferrucci began developing a computer system called Watson in 2006.^{10, 11} After undergoing initial tests that consisted of reviewing previous Jeopardy! episodes, Watson debut on the show in 2011 where it defeated long-time Jeopardy! Winner, Ken Jennings.¹⁰

Ferrucci programmed Watson to have the ability to process natural language of unstructured data, which is known as DeepQA, formulate and evaluate hypotheses, and learn from previous interactions (Exhibit B).¹¹ This allows Watson to be an accurate and precise machine.¹¹

IBM Watson Health

In April 2015, IBM Watson Health was created to provide solutions to data analytics within the healthcare industry.¹² At the time, physicians and researchers were burdened by the amount of time required to clean and normalize data in order to draw conclusions.¹² Physicians spend approximately 6 hours of their working day interacting with health data and it is estimated that the amount of health data will double every two years, furthering this burden.^{12, 13} Watson and its DeepQA were positioned to decipher health data, genomic data and social determinants of health to improve personalized medicine through increasing the accuracy and precision of treatment options.¹²

The ability of Watson to improve personalized medicine was demonstrated in a study conducted in 2018 at the Manipal Comprehensive Cancer Centre in India.¹⁴ In addition to the quantity of health data, oncologists struggle with keeping up to date with the rapid changes in science that affect treatment guidelines and new drug approvals.¹⁴ Personalized medicine has been shown to improve treatment options for individuals with cancer, as an individual’s genome can provide insights on therapies that will be most effective to reduce tumour growth.^{14, 15} However, this requires oncologist to sort through health data, including recent publications in scientific literature and case studies.^{14, 15} Researchers programmed IBM Watson for Oncology to acquire knowledge from scientific literature, protocols, patient charts, and test cases from the Memorial Sloan Kettering Cancer Centre.¹⁵ Within this study, IBM Watson analyzed 638 breast cancer patients between 2014 and 2016 and recommended personalized treatment options.¹⁴ These recommendations were compared with the oncologist at the Manipal Comprehensive Cancer Centre and the results of the study showed IBM Watson was able to recommend a concording treatment with the oncologist in 93% of the cases.¹⁴ In addition to the recommendation, Watson provides journal articles from scientific literature to support its decision.¹⁴

Hospital and physicians have been quick to pick up and utilize Watson within their practices.¹³ Hospitals that have fewer oncologists rely more heavily on Watson for recommendations and can alleviate the health inequalities by providing the best practice to individuals who do not have access to highly trained specialists.¹⁶ The cost of Watson varies between $200 and $1000 per patient, depending on the additional products and services.¹³ For example, a hospital can purchase additional consulting help and allow access to electronic medical records.¹³

IBM Watson Health has continued to conduct additional studies to support the potential of artificial intelligence in revolutionizing medicine.¹² Applications expand beyond genomics, including improving drug discovery, clinical trial matching and insurance coverage.¹²

MD Anderson Cancer Center and IBM Watson Health

The partnership between the two organizations, MD Anderson Cancer Center and IBM Watson Health, was developed to create an Oncology Expert Advisor (OEA).⁴ The goal was to help community patients, who are treated by community oncologists, receive the same high-quality care as the specialists at MD Anderson would deliver.⁴ Waston would provide community oncologist with scientific literature and personalized treatment options for their patients.^{4, 12} In addition, Watson would provide clinical trial matching for patients if applicable.⁴ As stated by the project leader, Dr. Lynda Chin, the project was designed to “‘transform how medicine will be practiced, by leveraging artificial intelligence’ to elevate the standard of cancer care world-wide”.⁴

What Went Wrong? The Downfall of MD Anderson Cancer Centre and Watson Health

The project between MD Anderson Cancer Centre and Watson came to a close due to delays, mismanagement and overspending on the budget (Exhibit C).¹³ Shortly after MD Anderson announced the closing of this project, an audit conducted by the University of Texas revealed many procurement violations involving outside consultants (Exhibit D).^{4, 13} The Cancer Centre was working with the consulting firm, PricewaterhouseCoopers (PwC), however, many of the services, such as supply chain management and information technology governance, required for the OEA did not follow the hospital protocol or were not within the scope of the project.⁴

Other reports indicated that the problem was inherent to Watson and its artificial intelligence ability.¹³ Specifically, Chin said that her team had difficulties with integrating Watson with the idiosyncrasies found in a medical record, such as acronyms, shorthand phrases, and different styles of writing.¹³ This required the doctors to more spend time teaching Watson to read a medical record and extract important information.¹³ They also faced struggles with employing Watson into clinical and routine practices of nurses and physicians as this was a new technology.¹³ Lastly, Watson did not have access to every medical institution and this created a relatively small patient pool.¹³ This affected Waston’s ability to generalize trends and to determine patterns, as a small patient pool does not accurately represent or allow for inference of all patients types.¹³ These patterns are critical for improving personalized medicine.¹³

IBM also experienced drawbacks from the closing of this partnership.¹³ This is represented through their declining revenue within its cognitive services, where Watson Health is classified, for 21 consecutive quarters.¹³ IBM has since implemented strategies to support their cognitive services as they view Watson Health as a growing $200 million market.^{13, 17}

Disadvantages of Artificial Intelligence

The disadvantages and limitations of artificial intelligence were brought to focus after MD Anderson Cancer Centre and IBM Watson Health terminated their partnership. Many criticize IBM for advertising false capabilities of Watson that were not developed yet.^{1, 3, 6, 13} Essentially, Watson and other artificial intelligence systems are only able to perform tasks they are programmed or trained to do.¹³ Reports indicated that Watson was struggling with learning the different forms of cancer and this resulted in the recommendations of inappropriate treatment options.¹³ However, for Watson to recommend the best treatment options, its operating system must be updated.¹³ As mentioned earlier, treatment guidelines and drug approvals can be changed rapidly prior to an overwhelming amount of published scientific literature that would allow Watson to update autonomously.^{18, 13} Instead, this would require hospital staff to be familiar with updating Watson, which is not likely as machine learning is a highly complex field.¹³ This ultimately creates a challenge for artificial intelligence to be used in the medical industry.

In addition, IBM was criticized for not conducting clinical trials prior to the launch of Watson Health to show scientific evidence of its capabilities.¹³ This is largely due to regulatory agencies not requiring clinical trials for software applications.¹³ Customers were incentivized to purchase Watson Health due to the value it provides through helping physicians manage a large amount of health data per patient.¹³ Watson Health was launched in 2015, however, IBM did not start publishing scientific literature on the application until 2017.¹² To date, studies have predominately been conducted by paying customers of IBM and there have been no randomized clinical trials.¹³

Physicians have also indicated that Watson is biased towards the United States and their clinical practices, as it was trained by physicians at the Memorial Sloan Kettering Cancer Center in New York.¹³ Despite the many physicians in other countries indicated that states they were hesitant to use Watson, it has been widely accepted.¹³ Currently, more than fifty hospitals across five continents have partnerships with IBM to use Watson for genomics, oncology and clinical trials.¹³

Future Partnerships

IBM has been a high-profile information technology company for over a century.⁷ Their recent involvement in artificial intelligence and the development of Watson Health has the potential to contribute to their legacy by providing benefits to many industries, including medicine.^{1, 12} However, IBM has lost hospital clients due to the news surrounding the failure of IBM Watson Health and the MD Anderson Cancer Centre.⁶ This is occurring at a problematic time for IBM as they face high competition from other information technology companies, including Amazon, Apple, Microsoft and Google who have formed partnerships with biotechnology and pharmaceutical companies.⁶

While there is still a debate on whether artificial intelligence has a place in the medical industry, this case highlights how two companies did not operate transparently and this ultimately led to the demise of their partnership. Interdisciplinary partnerships are becoming more common, especially within health and technology, and companies need to develop strategies to form and maintain successful partnerships. What strategies could IBM and MD Anderson Cancer Centre implement to avoid repeating this mistake? What can emerging partnership between health and technology companies learn from this case?

Exhibit A. A summarized timeline of IBM.

Exhibit B. A step-by-step depiction of how IBM Watson processes a question to determine an answer.¹¹

Exhibit C. Financial statements from the OEA project between MD Anderson Cancer Centre and IBM.⁴

Exhibit D. The procurement and vendor violations listed in the University of Texas Audit Report.⁴

References:

1. Whelan, D. (2018, October 24). Do IBM Watson health’s leadership change and layoffs mean A.I. is bad medicine?. Forbes. Retrieved from https://www.forbes.com/sites/davidwhelan/2018/10/24/do-ibm-watson-healths-leadership-change-and-layoffs-mean-a-i-is-bad-medicine/#4c76fe13227d
2. MD Anderson Cancer Centre. (2013, October 18). MD Anderson taps IBM Watson to power “moon shots” missions. Retrieved from https://www.mdanderson.org/newsroom/md-anderson–ibm-watson-work-together-to-fight-cancer.h00-158833590.html
3. Al Idrus, A. (2017, February 21). Clouded by MD Anderson audit, IBM Watson unveils new partnerships. FierceBiotech. Retrieved from https://www.fiercebiotech.com/medical-devices/clouded-by-md-anderson-audit-ibm-watson-unveils-new-partnerships
4. The University of Texas Audit Office. (2017). Special review of procurement procedures related to the M.D. Anderson Cancer Centre oncology expert advisor project. Retrieved from https://www.utsystem.edu/sites/default/files/documents/UT%20System%20Administration%20Special%20Review%20of%20Procurement%20Procedures%20Related%20to%20UTMDACC%20Oncology%20Expert%20Advisor%20Project/ut-system-administration-special-review-procurement-procedures-related-utmdacc-oncology-expert-advis.pdf
5. Flam, F. (2018, August 24). IBM”s Watson hasn’t beaten cancer, but A.I. still has promise. Bloomberg Opinion. Retrieved from https://www.bloomberg.com/opinion/articles/2018-08-24/ibm-s-watson-failed-aganst-cancer-but-a-i-still-has-promise
6. Swetlitz, I., & Ross, C. (2018, October, 19). Head of IBM Watson health leaves post after company stumbles, growing criticism. STAT News. Retrieved from https://www.statnews.com/2018/10/19/head-of-ibm-watson-health-leaving-post/
7. IBM 100. (n.d.). IBM is founded. Retrieved from https://www.ibm.com/ibm/history/ibm100/us/en/icons/founded/
8. Samuel, A. L. (1959). Some studies in machine learning using the game of checkers. IBM Journal, 3, 210-229.
9. IBM 100. (n.d.). Deep blue. Retrieved from https://www.ibm.com/ibm/history/ibm100/us/en/icons/deepblue/
10. Ferrucci, D., Brown, E., Chu-Carroll, J., Fan, J., Gondek, D., Kalyanpur, A. A., Lally, A., Murdock, J. W., Nyberg, E., Prager, J., Schlaefer, N., & Welty, C. (2010). Building Watson: An Overview of the DeepQA project. AI Magazine, 31(3), 59-79.
11. High, R. (2012). The era of cognitive systems: An inside look at IBM Watson and how it works. IBM Corporation, Redbooks. 1-16.
12. Watson Health Perspectives. (2018, November 19). Watson health: Get the facts. Retrieved from https://www.ibm.com/blogs/watson-health/watson-health-get-facts/
13. Ross, C., & Swetlitz, I. (2017, September 5). IBM pitched its Watson supercomputer as a revolution in cancer care. It’s nowhere close. STAT News. Retrieved from https://www.statnews.com/2017/09/05/watson-ibm-cancer/
14. Somashekhar, S. P., Sepúlveda, M. J., Puglielli, S., Norden, A. D., Shortliffe, E. H., Rohit Kumar, C., Patil, P., Rhee, K., & Ramya, Y. (2018). Watson for Oncology and breast cancer treatment recommendations: agreement with an expert multidisciplinary tumor board. Annals of Oncology, 29(2), 418-423.
15. IBM. (2019b.) Watson Oncology Suite. Retrieved from https://www.ibm.com/watson-health/oncology-and-genomics
16. Cha, A. E. (2016, June 27). Watson’s next feat? Taking on cancer. The Washington Post. Retrieved from https://www.washingtonpost.com/sf/national/2015/06/27/watsons-next-feat-taking-on-cancer/?utm_term=.3e6a1bb321ba
17. Schroeter, M. (2017). Financial discussion: IBM investor briefing 2017. Retrieved from https://www.ibm.com/investor/att/pdf/2017_Investor_Briefing_Financial_Discussion_charts.pdf