Leveraging Randomized Clinical Trials to Generate RWE for Regulatory Purposes – Day 2

– All right, good morning everyone Good morning, nice to see you this morning I’m Mark McClellan, I’m the director of the Duke Margolis Center for Health Policy and I, on behalf of our Center faculty and staff I’d like to welcome everyone in person and on our webcast back to day two of our public workshop on “Leveraging Randomized Clinical Trials “to Generate Real-World Evidence for Regulatory Purposes.” We are hosting this at the Duke Margolis Center for Health Policy with support through a cooperative agreement with the FDA And I’m pleased to be with you all after yesterday’s productive start to this dialogue on the considerations for using randomized approaches within clinical settings to develop real-world evidence on effectiveness for regulatory decision-making You all had a chance I think to recap a bit of the conversations yesterday already I do want to spend a moment before we get going again today on some of the key themes and takeaways just to reground us a bit and set the stage for today’s discussion First and foremost we heard a lot from speakers and many of you participating in this event that the type of research question a sponsor is pursuing will drive everything else Design and feasibility specifics depend on what you are trying to answer and for whom, the issue of fit for purpose, data methods and other features of a study In a regulatory context some designs may be more appropriate than others for specific regulatory questions And here’s where I want to be sure to remind everyone that for purposes of this workshop and in keeping with the congressional mandate behind this work at FDA and throughout the industry, we are largely speaking about regulatory questions that are post-approval for a drug or biologic And that means you can assume that some typically randomized controlled trial derived level of safety and efficacy has been established in the scenarios that we’re going to be continuing to discuss today The pivotal trials for those approvals will be a sound foundation for the further real-world evidence studies that use randomization at the point of clinical care So the randomized approaches that we’re considering today and throughout this conference and in the further work that will come out of it, should be seen as supplemental and additive to the body of evidence on drug products and biologic products that will in most cases also include these previous pivotal randomized controlled trials for approval Now a good part of the discussion yesterday focused on endpoints derived from real-world data use solely or in combination with more traditional randomized control trial data We heard a number of comments about the quality and reliability of real-world data and it’s use for endpoints And whether you’re defining a common set of data elements that are systematically captured within a therapeutic area and captured at the point of care or mapping large data sets into a common model, this combinations, electronic records may be supplemented by special therapeutic area information relevant to the research question at hand There’s wide recognition that these data or combinations of data offer great potential and insights into how patients are using medical products in the post-market setting And it was noted yesterday that even if you do randomize it may not change or impact the biases associated with the underlying data Additional challenges relate to issues like ascertainment, the availability of accurate mortality data and that sometimes burdensome data linkage processes that are needed to bridge across these different sources of data and create a more detailed and complete data set Also the fact that the systems from which these data are derived may be changing during the course of the research, may be evolving So these are not insignificant challenges and you heard from some of our panelists that a sensible approach maybe one where a sponsor and researchers clearly established what they think they need from the data or combination of data at the outset then identify what they think they can accurately obtain from existing already populated real-world resources like electronic health records or the like and then have a potential for add-on data collection modules in which a provider enters additional relevant information So a lot of discussion of these data issues But we all know that the data themselves are not evidence and that methods in analytic approaches need to be applied

to the data to turn it into conclusions, evidence, about a medical product question We discussed a lot of design and analysis issues and I’m not gonna try to cover all of them in just this short summary These issues included outcome measures, so turning the data into reliable outcome measures, minimizing bias and maximizing likelihood of a meaningful and precise enough result and the ability of researchers to correct issues that come from the messiness of real-world data Much of these considerations are caught up in patient recruitment, blinding, statistical analysis for causal inference and there are a variety of approaches that we covered and have been addressed outside this context for these kinds of problems The ultimate design and statistical analysis plan though, still hinges on that core question that the study is trying to answer and the intended regulatory context I do want to spend another moment on blinding which was the focus of an entire session yesterday as you recall Blinding is still a key part of randomized controlled trials that power an initial approval Remember we’re talking about post approval setting based on a foundation of pre-approval pivotal studies Blinding and follow-on in real world studies shouldn’t necessarily be seen as an all-or-nothing proposition Based on the underlying evidence about the already approved drug or biologic and the practicability of blinding in the real-world setting, there may be situations in which blinding is either infeasible or not necessary for the particular post market research question and clinical context at hand And we heard examples of some of these situations and how researchers have been able to mitigate biases that could arise as a result of issues related to blinding And this brings me to the concept of a hybrid approach to these types of real-world studies and that was mentioned several times yesterday Leslie Curtis and others described how there’s a continuum for many of these study designs A continuum for how much or how many real-world data sources are used or for how many different types of design features from real-world evidence methods are included And so we need to understand that again the question itself, the research question itself will drive how real-world data sources are used, what methods are applied to generate real-world evidence through randomization and what hybrid study design should best address those needs And finally a reminder that we heard many times yesterday, is that the real world is by its definition messier then you might like it to be And the ways to deal with these imperfections and data and the application of methods to these data are through pre-specification, research design, analytic approaches that we hope to further explore as this whole area of real-world evidence is developed in support with, in conjunction with, further guidance from the FDA There are many potential biases and other concerns yesterday that arose which are not unique to randomized real-world evidence studies but also reflect issues that have long been known to arise and messiness in traditional randomized controlled trials Perhaps to a lesser extent, perhaps different in nature, in some cases but not completely new in the real-world evidence context And we often talk about how real-world data and real-world evidence can make studies more efficient or simpler but I think it’s important to view that result of what appears to be a simpler and complete study design as reflecting a lot of attention going into it in terms of investigator and data oversight, patient adherence, methodologic challenges, infrastructure and resource constraints So these are issues that need to be addressed and planned for ideally with the inclusion of stakeholders involved in the clinical area to enable more widespread use of these types of studies So those are some of the themes that came up and it’s exactly these practical and operational issues involved going from the concept of using real-world data and real-world evidence through randomized approach to augment post-market evidence It’s exactly these kinds of practical and operational issues that arise when you try to do that in any specific context that we’d like to turn to in our sessions today So we’re gonna have two main sessions The first one is focused on important regulatory considerations to monitor trials embedded in clinical settings

This is gonna be followed by session to discuss key infrastructure issues and other important enablers or barriers of these trials that are critical to their success and how this might fit into FDA’s development and an expansion of their real-world evidence framework and program Then at the end of the day we’ll have another open comment period just as we did yesterday where we want to hear from you to make sure we haven’t missed anything on what you feel like the FDA should be considering when reviewing and using evidence generated by trials through randomization in clinical settings Now before we get started I have a couple of housekeeping items to review Just a reminder that this is a public meeting It’s being webcast online We appreciate all the people who joined us yesterday and many and all the ones who are here with us today We are going to be taking questions during the sessions We’ll have some time for that from our online audience So please think of your questions as we’re going along In addition to having that extra time at the end for comments and for those of you who are here in the room, please feel free to help yourself to coffee and beverages throughout the morning, they’re located right outside Okay so with that I’d like to start our first session on monitoring randomized clinical trials that generate real-world evidence In this session and for those of you who are participating this session this is kind of your cue to come on up In this session we’re going to dig into some of the key regulatory and oversight considerations and some potentially unique or distinct challenges that result from implementing trials in clinical settings These are issues such as the role and responsibilities of principal investigators overseeing the trial considerations for identifying and reporting safety events and assessing data integrity using real-world data So we’re gonna have three discussion topic areas teed up for the session which will start with a presentation using the Salford Lung Study as an example This is a widely cited study in the area of generating real-world evidence We think we can learn a lot from it that we’re not holding us up necessarily as the gold standard for everything but really is a concrete real-world effort to develop a trial integrated in the clinical settings that was used to inform a regulatory decision So for focus area one on sponsored conduct and monitoring challenges we’re gonna begin by introducing the speakers on sponsor conduct and marketing and monitoring challenges And they include Elaine Irving, the senior director and head of real-world study delivery at GlaxoSmithKline Loretta Jacques who’s clinical development director and Salford Lung Study project leader at GSK Adrian Hernandez, professor of medicine and vice dean for clinical research at Duke University School of Medicine And Leanne Larsen who is a corporate vice president worldwide head for real world-evidence at Paraxel So I want to hand over now to our two presenters, Elaine who is with us in person and Loretta who will be dialing in remotely And I’m hoping that technology is working just fine And so Leanne over to you first, thanks – Thanks, Mark Okay, I’m about to use, is remote, is it, okay So yeah, thank you again everyone for the opportunity to talk to you this morning As Mark said I’ve got to start by introducing the Salford Lung Studies which I guess was GSK’s first venture into this more real-world trial space It was, the concept for the Salford Lung Studies was born a number of years ago, long before probably most of us in this room But it’s considered conducting randomized control trials in the routine care setting So it is unique in a number of ways and that it was pioneering in its time but also a nothing remains to date it’s still the only first Phase III randomized control trial that to the scale to be conducted in the routine care setting But as we’ve heard most of the conference that we’ve been part of for the last two days has really been focused in that post-marketing phase So I just wanted to draw attention to the fact that this is a little bit different in that it was a phase or it started prior

to the launch of the molecule itself But however the the objective of the Salford Lung Study when we say tight at the beginning we’re pretty much the same as we’ve all been discussing for the last two days in that it was put in place to really help us better understand how Relvar which is a dual therapy for the treatment of COPD and for asthma would behave when we came from the very constrained routine randomized controlled trial setting and took the molecules out into the routine care setting So looking at that to really embrace the heterogeneity in the patient population that would eventually receive Relvar, use endpoints that were of relevance to the routine care of the patients So really focus on endpoints that the physicians use for day-to-day decisions and really try and keep the patients as close to routine care as possible And also the physicians The idea was to try and ensure the study was designed in a way that it fitted into routine care without hopefully increasing too much the burden on the investigators and also definitely not to impact on the patient’s day-to-day care The act of randomization obviously helped us to maintain the scientific rigor and internal validity that we discussed yesterday It was an open-label trial, again for a number of reasons that we described and spoke about yesterday, mainly again because like the Intrepid Study I described yesterday big piece here is really understanding the benefit of a once-daily administered drug versus the use of multiple inhalers multiple times a day which really can’t be done in a blinded scenario And as I said a big aim was to try and reduced the burden on the physicians participating in the study So we did hope that we could use the integrated care record as much as we possibly could to extract data that already existed in the patients to minimize the data entry into the eCRF And I should say that’s really why the study ended up being in Salford because of the need to access that integrated care record and Salford in the UK was, it was pretty unique in having fully integrated care records that captures the major hospitals and all the GP surgeries in that area So that was really why we ended up in the region of the UK that we did, hence the name of the studies So just to look at the first study, the asthma study And again very similar to so some of the points that were raised yesterday The inclusion exclusion criteria were very minimal for both of these studies Inclusion was really based on GP diagnosis and for asthma, for example, there was no smokers were included and there was no lung function tests as part of the eligibility criteria Patients were then randomized to Relvar or they continued on their usual care And the primary endpoint for this study was the asthma control test that was conducted at six months but part of the reason for that was that we wanted to try and time the delivery of the data with the launch of the molecule so that we had the data at the time of launch to support a lot of the decisions that the health technology, (groans) assessment agencies need to make decisions about the value of the medicine and also physicians for using the medicine as well The COPD study looked very similar Again entry criteria based on GP diagnosis, randomized at the start of the study and then a final review at the end of the study So, sorry I should have said, in both studies there was a protocol dictated visit beginning on the study and at the end of the study In between time the patients were treated in accordance to their routine care so if they needed to attend their physician they would do so Just this slide just summarizes the data collection for the study so our primary on endpoints were collected through an eCRF

So the primary end point for the COPD study was exacerbations They were collected as they would be in a normal routine randomized control trial And the questionnaires that were used in both studies So for the ACT in the asthma and CAT in the COPD study as well as some of the other PROs that were collected through site paths And then we had access to the integrated care record which enabled us to do extensive safety data collection which now we’ll talk more about later And also access to healthcare utilization core medications, that type of information So think to the point that Mark made earlier here, is thinking about where is the best place to gather your data depending on the end point of interest Just to give you an idea of the scale, I think as well Mark alludes to, these studies can be complex This seems simple in concept but I think actually as we do more of things what we’ve realized is the simpler we make the concept the more complicated it is behind the scenes and the longer it takes to get one of these studies actually started because of the amount of planning and the number of stakeholders that are involved to make these a success But it just gives you an idea of the scale of the number of individuals that were involved to actually make this, one of these studies happen So same again for the COPD study And I think just to emphasize again and as we’ve heard again as a common theme, the buy-in from the stakeholders for these studies is essential These types of studies can’t be done unless you have a partnership with all the different parties So for the Salford and studies it was essential that we engaged a lot of thought leading principle investigators right at the beginning of the study to really help champion and support the need for this type of research and help us engage with the physician community in the area and help them understand the value of the research Not only, obviously, with respect to the value for the medicine and its future but for them in their actual day-to-day care of their patients We also had a lot of help from the local commissioning groups where we engaged with GP committees and groups to promote the study There was a need to engage with the hospitals to help them understand the study the pharmacies, obviously were engaged as sites because we were trying to supply or we did supply the drug through the local community pharmacies Obviously this was investigational product so they all had to be trained in GCP and we set up to manage trial supplies That’s not something a community pharmacy would normally do It was through this entire infrastructure really this external infrastructure that really helped make it a success And then that with regular discussions as well with the MHRA and a nice around the, again their thoughts on the study design and the way that we were setting up the study And again this was one of the first studies to go through that joint advice process, you know just again reflecting the novelty and the innovation of the team at that time So I think as I said, the collaborations with the key stakeholder groups are key and I think as a sponsor, you can’t underestimate the time that that takes It takes an enormous amount of time I think and Loretta will correct me if I’m wrong but I think this study actually took about probably about four years of discussion and planning before we even hit the first subject to be recruited And that was just all educating and bringing all these stakeholders together Obviously we’re dealing with a number of research naive sites which is not unusual We do that in other clinical trial settings as well But I think it’s more about the scale in a study like this, the number of research naive sites that you tend to engage is a lot greater than you would in a randomized controlled trial And I think also what we were very mindful of is that we were trying to slot this study into routine care so if we had the design right then really we were asking GPs to do their jobs We weren’t asking them to do much else So what we were trying to do was, yes, everyone has to be GCP trained, it’s an interventional trial but really work to try and adapt training to suit the purpose of the study We were aware that this was not something

that GPs were used to doing nor did they have the time to do it really so we did supply a large study nurse support team to the physicians who really helped with respect to finding patients So they would run searches across the EHRs to find patients and support in contacting and bringing those patients in to the study They would carry study visits and they would help enter data into the eCRFs as well Obviously the investigator was still accountable and responsible for those activities so that there was still the need for those investigators to oversee those activities The local pharmacies as they say, you know, they’re naive as well and we were asking them to supply study drug and that’s new for them They don’t necessarily have the facilities or the know-how or being used to complying with all the GMP regulations So again, a lot of training required there And also what we noticed was there was an awful lot of turnover to staff in the pharmacy as well so that there was a need to continually go back and revisit the training Obviously the requirement we’d placed on ourselves for the use of the integrated care record meant that we were, as I said, localized, a very small region of the UK and that actually really meant then that everybody needed to participate in the study So we needed as many of the GPs from the community as we could, we needed the hospitals engaged and we needed all the pharmacies engaged So again that and, as I said, we did use the PIs, we did use then GP champions as GPs came onboard but we also had to do a huge awareness campaign And I know there were big pink buses went Salford that advertised the study and try to raise awareness of this study So again a lot of thought needs to be put into how you’re actually going to recruit and maintain the infrastructure required As I said, you know, this is a GCP, a controlled trial and the monitoring that we had in this study obviously was in line with what you would probably expect for a randomized control trial Primarily because this was a Phase III study In subsequent studies we have adapted our monitoring approaches much more but I think and I think the take home for me is just really there is a risk based monitoring framework for us to use and I think it’s just really having the confidence to use that and really think through what are the real risks to your study, what needs to be monitored but also if you have findings how will you act on those findings ’cause they may be different for a real-world study as well because there may be things that you can’t change or you can’t interfere with So do you really have to monitor them to the degree that you would in a randomized controlled trail And I think with respect to what complications did the use of their real-world data bring, well I think a lot we’ve talked a lot about the external stakeholder engagement but actually internally you have to think very differently So our clinical operations teams that met this study were only used to running randomized controlled trials And bringing in the use of real-world data and trying to set up a study where you have limited visits brings a completely different mindset And actually now we have my team who are fully dedicated to this space and that was really off of the back of the learnings from this study and the challenges that this study team faced You do need to have epidemiologists and people who understand real-world data on board from the outset if you’re going to bring real-world data into your trial and that involves bringing two completely different worlds together ’cause you have your clinical statisticians that are only used to dealing with data and the eCRF and then you have your epidemiologists who are only used to dealing with real-world data The worlds I’ve never met before so there’s bit of education there So it can be used to reduce the amount of data collected in the eCRF I think the key message is collect it when it adds value It’s complicated so if you can collect your data easily without interfering with patient care then I would encourage you to do so and use the real-world data sources when it really adds value like accessing healthcare utilization data another data that you can’t access through the normal channels And because bringing all those different sources together as well adds complication with respect to integration because we have lovely standards that are very clear for eCRF data collection

That’s not the same for an EHR and I’m sure Martin will touch on the number of those later on Okay – Great job, thank you (audience applauding) Thanks Elaine, thanks to you and Loretta and next is Adrienne – Sure, so, I’ll just talk about a few reactions here and kind of going from yesterday to today as well and then setting up Salford Lung Study And so just a few comments here One is as Mark highlighted at the beginning, this is really about post-approval studies So it’s a different context when we’re thinking about a setting where there is a medical product that’s getting either an expanded indication or a new setting compared to what its original approval for And in that setting like we really need to reconsider how we do things regarding monitoring What has happened over the last number of years is we’ve similar to what has happened in healthcare where there’s defensive medicine, we have now gotten to defensive research So every little thing that does gets monitored and cited then that has ripples across the whole research ecosystem for which we will never have that happen again And so it doesn’t matter the context, it actually that’s what’s the reactionary So we need to recognize that and deal with that So let’s get away from defensive research to actually proactive research And those are the elements that people have talked about in terms of quality by design So consider the context, refine the approach and deliver on what is said The third thing is again refocusing on the guiding principles of these areas So what we’re aiming to do here is actually try to answer more questions for which there’s uncertainty around and in doing so like you know the key questions are who are we studying, what’s the intervention, was it delivered, what were the key outcomes, were there key safety outcomes there? So focusing that and having our monitoring focused on that The fourth thing is if we don’t change the system and get away from defensive research there are opportunity costs The direct cost is very clear So the amount of monitoring that a typical study takes it can be upwards of 40% of the costs People may quarrel about that but that is at least some of the estimates are there I know this wasn’t discussed here just now, the costs were more for the Salford Lung Study than initially anticipated and not by just a percent And so that needs to change The other thing is that there are opportunity costs that are indirect So how many questions can be answered if we don’t change it? And then also time So if you just think about just a simple monitoring plan where a monitor comes in every six weeks, that’s time away from participation and study, time away in terms of doing something else and time to answer a question and that all adds up And so I mean you really pose the last question do we want to answer more questions or do we want to have more data per question or more cost per questions and that is the challenge around monitoring And then when you reflect on that, think about the studies that Bob highlighted yesterday The GC study, the ISIS trials, those trials that were done decades ago Could you do them today? Now I’ve worked with sponsors trying to develop those concepts and that’s been incredibly hard and today the answer is no And I think that’s what we’re aiming to change here And so to get to, yes, how can you do that, answer more questions and this is a pivotal area that can be reformed So I’ll stop there – [Mark] Great, thanks very much (audience applauding) And next Leanne – Great, so to wrap up this conversation and to build upon what Adrian and Elaine have both said, I want to talk about monitoring in the larger context of quality and kind of bring this whole conversation together If I can make the slides work right Probably the big button with the arrow on it would work So (laughs) in any case when we’re thinking about monitoring, it’s really about quality How are we ensuring the quality, ensuring that what we’re doing in this study is gonna deliver the results that we need We’ve talked a lot about the fact that these studies have to be designed first and that’s where the quality starts and the monitoring is a part of that So you have to ensure that these studies are aligned with the needs, that the needs of the regulators

and the payers, of the clinicians and the patients are all very, very different They’re looking for different kinds of data They have different questions that they need to have answered and you have to consider that when you’re putting these studies together You know, we don’t have the luxury of running 100 different studies to answer every question individually So to try to get the most out of the study is important But the second point here I think is what I want to emphasize today which is not only do you have to have it in line with the objectives but you have to have it in line with the operational realities of doing these studies as well When you’re working in the real world, things operate differently and the requirements and the mandates are different and when you bring all those together then you can get study results that are both robust and relevant and that’s what we’re driving toward So as we’ve talked about today as many, many speakers have said, hybrid studies are really, probably what we’re seeing most commonly today You know when we’re answering these kinds of questions, we generally just do have multiple data sources that are coming together in various ways that you probably can’t read the teeny, teeny, tiny print on here which is fine The message though is that it does take a lot of different sources and different kinds of data to create the information and the evidence that we need to have to answer stakeholder questions today And there are technology issues around this There are privacy and policy issues around this There are patient issues around this So it is very complex, as Elaine had said, getting these things to work is not an easy endeavor It can be done and also every study, every single study is different and so it’s very hard to put far-reaching kind of umbrella suggestions and arrangements around this You really do have to look at each study individually and when you, and don’t hold me to where these check boxes are, they’re just kind of an overview But you can see when you’re looking at different kinds of data source You have to look at all the different sources and see what the best source is for those data So it’s very important to design it in that context and that will drive your monitoring strategy So the last two slides what are some of the challenges that we’re seeing As I said, they really revolve around data operations, drug supply when that’s important and then the monitoring around that So in the data we do generally require multiple data inputs and that requires, as Elaine mentioned, challenges around mapping and the integration of those, the deduping, reducing the duplications, and also the ability to ingest large volumes of data in many cases You have to deal with the drugs, the access to the drug and in the real world if you are providing drug that can be challenging On the site management side, and Elaine touched on this as well, these are often research naive sites If we’re dealing with the real world we want to have sites involved that reflect the actual patient population We have to go to where the patients are and unless it’s a very rare disease those patients are generally not centered solely in a treatment, a tertiary care center an academic treatment center, a clinical trial site So we have to go find the patients And operating within that environment, in monitoring a study within that environment means that we have to be very sensitive to their workflow, their patient flow, their staffing, the amount of time and effort that they can put into this So we have to be very careful around that So how do we address these challenges? You have to get the specialized teams involved in this, medical informatics epidemiologists who understand how to work with the data, who understand real-world data is very critical The monitoring the quality begins with the the systems design as well The EDC systems that are using for the the data collection have to be designed differently We use more edit checks in a real world study than we do in a clinical trial ’cause we’re trying to drive first-time quality Get the data into the system accurately before it needs to be monitored The data review, the monitoring begins with the site teams and data management not just relying on people going out to the sites So there’s a pretty intensive review of the data as it’s coming in, going back to the site questioning them, asking them for clarification long before it gets to a CRA or someone going out to the site You have to really support these sites The site management is a really critical factor here It’s about holding their hands through this process in many cases So it’s a much broader question, much more frequent contact in that sense but trying to minimize the burden for the site overall One way to do it, as Elaine had mentioned, they sent nurse teams out to the sites In the 20 years that I’ve been doing this, we don’t do that as often It’s very complex as they know, but there are ways to work with the sites and support them so that the site staff themselves, even if it’s not a study coordinator, can actually do that effectively as well and there’s a lot of experience in doing that And finally on the remote side, talking about risk-based monitoring we also do look at a combined remote

and on-site monitoring mix and a really heavy reliance on triggers We know what to watch for so that we can see when things are starting to go awry Is it over enrollment or under enrollment? Are we seeing discrepancies in the data? So there are a lot of things that we look at in the data that are coming through that provide triggers and allow us to know when to employ the monitoring most effectively So hopefully that gives you an overview of how we can monitor these things again effectively It’s not the same as a clinical trial in any way but there are ways to do this and I think as we start to bring that into this model, it can help support the the overall quality of these studies and help deliver the results that we need – Thank you (audience applauding) All right, a couple of minutes for discussions You all covered a lot of ground from the standpoint of sponsor conduct and monitoring issues and needs It struck me that the foundations, like quality by design, risk base monitoring, those are generalizable principles but some important additional considerations around alignment with stakeholders in the community, operational issues that come up in these real-world settings, the site monitoring issues that Leanne was just talking about That is extra work I want to go back to Adrienne’s comment about defensive research and based on all your experiences where do you see sort of the biggest gap between, yes, there are a lot of additional considerations for sponsors in these settings But I take it from the experience you’ve had and, Adrienne, your comments that there may be some particular areas where there’s a relatively big gap between like how much effort seems to be going into it versus the payoff If we were gonna try to make it easier to use these methods for more questions, what are the, pick one maybe that you, issue they’d like to focus on more – So just as an example, if say we’re going to leverage electronic health data for the ground truth and saying that this is characterizing the population, knowing that it may be imperfect et cetera, but in a randomized trial we’re aiming to say what’s the general population that we’re enrolling and then what’s the outcome and especially for outcomes that are hard, like what that is The questions that come up are then when you see things in electronic health data that may vary because practice varies, what do you monitor? And we’ve gotten that question So how do you monitor the electronic health record and if you found something that’s different, do you change it? Well someone saw an ophthalmologist and someone saw a cardiologist that’s not surprising that they may document differently And so what’s the ground truth there? So that’s one example of just taking advantage of the data But the second thing is that there are things that may be the focus of monitors and or sponsors because of something else that happened in a completely other therapeutic area in a completely different phase of an indication And so what someone does for a Phase II study or a Phase III study may be very different than focusing on sodium levels or potassium levels in a Phase IV study that’s an adjacent population ending with an aim for either expanding an indication or informing greater clarity and a label – We have time maybe for one question or if there’s a quick one, yeah – [Jay] Jay and I’m from Pfizer I’m very, very thankful for example that Elaine and Leanne and your comments Adrienne I’m very motivated to do something like that I’m just saying right up front I’m very motivated Have you discussed this kind of design with FDA and following that what does that mean when the results come? Is it supportive of an indication or a change? What do you think the value of doing this would be? – I can’t really comment on them, on detailed discussions but I think it comes down to again the the key point that we’ve been saying the whole way through is it depends on what question you’ve asked with the design and what the intent of the the study was This study wasn’t designed to direct any kind of regulatory change or request for a label expansion in the outset And the EME did ask that we use this study to support additional post-marketing safety data collection

which we were able to accommodate in the study – I’ll say, I’ve had positive conversations about that with our FDA colleagues and then also folks in with sponsors where it comes down to though like when there is actually a clear use case that this is really relevant, it’s important there are elements here that can be highly streamlined here There are concerns of what happens three years from now or four years from now And so, and I can review cycle like, is there any thing that could come up, so-called unknown unknowns that people have to mitigate against And so then that’s when it kind of unravels And so sometimes I hear from our colleagues at FDA, it’s like, hey, no one just comes up and ask Like you know put it, send a proposal, the context matters, so forth, and then on the industry side we’ll hear that they were worried about something coming up because of something unrelated And so that’s what comes up – Yeah does seem like that FDA has made clear that, I’m sorry Elaine, did you have a comment? – No, no, no, go ahead – No, go ahead, we’re gonna have to wrap up momentarily – The only thing I was gonna add was that the, with regard to RWE the majority of the agency interactions that we’ve had have been and with regard to safety mandates Lots of studies that being done around the world that are in response to the mandates being given to the products More recently in synthetic control arms as they’re attached to clinical trials as well – Yeah, I do think that as this meeting reflects, the FDA interest in early engagement is definitely expanding to issues beyond post-market safety questions and as recurrent theme of the context matters, a specific question matters, hopefully coming out of this will be some more relevant guidance to different kinds of contacts but I think FDA thing, I’m not speaking for FDA, but I think they’re very interested in hearing from you about how to design your particular study And I want to thank our panel for a great discussion of some very important issues in sponsor conduct and monitoring for real-world evidence development Thank you all very much (audience applauding) And we’re gonna zip right on to focus area two on safety monitoring that will look at the key issues and processes involved when identifying safety events with real-world data Our speakers are coming up to the stage now are Nawar Bakerly who’s clinical director for the Department of Regulatory Medicine and divisions chief and clinical information officer at Salford Royal Foundation Trust Greg Ball, principal senior statistician at Merck and Ellis Unger the director of the Office of Drug Evaluation one in the Office of New Drugs at Cedar And we’re gonna hear first from Noir – Good morning everyone Thank you very much for inviting me and thank you very much for asking me to join you in this great city of yours It was quite enjoyable until it started raining yesterday and it felt like I’ve never left home.(laughs) My name is Nawar Bakerly I’m a pulmonologist, I work in Salford For those of you who don’t know where Salford is, it’s an area in Greater Manchester If you don’t know where Manchester is, Manchester United should help you out (audience laughing) What I am, within the next 10 minutes or so, is to try and describe to you the process of safety monitoring at the time when we’ve implemented the Salford Lung Study There are two things that I’d like you to remember First of all my slides are complex Apologies about that They’ve gone through many editions of recreation to try and simplify them And second thing is that Salford Lung Study when we started talking about this was about 10 years ago So technology has moved on quite a bit of time And obviously things have moved forward as well in relation to our understanding of real-world medicine This is my conflict of interest, I’m working on it This is the design of the Salford Lung Study for the COPD study I’m not gonna go through that again ’cause Elaine already mentioned this The number I’d like you to have a look at is around 2,800 patients This is the asthma study, the number is a little bit over 4,000 So there is about 7,000 patients between asthma and COPD and that really the two things here, first one is this is a safety monitoring nightmare in a study that started about 10 years ago, first-of-its-kind pre-market authorization So the bar had to be set really quite high And second thing is this was a recruitment nightmare because we were aiming to recruit about 40 to 50%

of the population, the target population, or the population that suffered from the conditions within the area that we’re recruiting from So really had a lot of challenges and if I’m just gonna focus on three main challenges, try and set the scene a little bit, the first one is that how we ensure that we don’t interfere with normal care and that is really quite difficult because obviously what is the definition of normal care and that could be a moving target The other thing which I think it came out in the previous conversation is about how do you train whole army of clinicians, primary care physicians, pharmacists to participate in a study, free-market authorization, remember because that’s really quite important So from dispensing of medication, stocking, accountability of medications, good clinical practice, PI responsibilities, all of this If I remember correctly, we trained in a little bit over 2,000 people over the period of the study At any one time we’ve had about 3,000 patients recruited One of the essential things, when we’ve set the Salford Lung Studies to ensure that drugs are accessed through the usual pharmacies There’s no point of doing a real-world study and patients are going to central pharmacy to pick up their medications That has to be set up with people picking up their medication through their own pharmacies But then the next challenge which is more relevant to my talk is that how we ensure robust safety monitoring is maintained without affecting routine clinical care And perhaps that’s maybe the reason to why our colleagues within GSK came to Salford ’cause for those of you who know the UK’s healthcare system, we kind of the mirror image in terms of digital maturity Our primary care is very well mature, had been digitally mature since the late ’80s Our secondary care is not The unique thing about Salford is primary, secondary care were very digitally mature and they were connected So there was a what we call Salford integrated record which really helped to create that sort of environment to enable the electronic, following patients electronically and I’ve described that in my few slides that are coming up next And we had to develop a dedicated safety team and I’ll obviously also describe what that looked like We also needed to ensure that there is a robust collection of study endpoints I’m not going to touch on this, but obviously the use of electronic medical record had helped immensely We’ve created an electronic CRF and perhaps maybe my colleague Martin Gibson will talk about that in his talk So this is my first complicated slide and I do apologize about this But the way I want you to think about it if you look at the linked database that the center of operations, that’s the headquarters, and everything else is pretty much like coming as a tentacle out of that So think about this as a big octopus, a big good octopus not a bad one, a big octopus and the head of the octopus is the center of operations That’s the linked database The tentacles are reaching out everywhere Some of those are fiber-optic, so the data is coming very quickly Some of those are broadband data coming quickly but not as quickly and some of those are landline, coming a bit slowly So if we are going to apply that analogy into what I’ve described, the fiber-optic ones are the data that are coming from the local hospital and the local GP practices The sort of the broadband type of data, that’s the data that comes in a little bit slower That data that takes about 24 hours, some primary care organizations also The slower data is the national data So what does that mean? That means if patient is recruited into the study, ends up being admitted in a hospital down in London whilst they were on a day out, that their data is not immediately available ’cause that comes through the national extraction of data In addition to that there is data that’s coming through from the ERT The question is that we’ll be using and those are sort of the data that comes instantaneously into the system And what that big head of the octopus does is that kind of processes all the data and sends it to GSK center of operations in an anonymized fashion My next complex slide The thing to try and explain in here which helped us in doing the safety monitoring which is something that’s unique perhaps to the UK is that every patient in the UK, or every citizen in the UK, has NHS number, National Health Service number It’s a unique identifier of each individual

and that helped us because that enabled us to put an electronic tag on the NHS number of every recruited patient And that electronic tag would allow us to track where patients are within the healthcare system So if you look at the left side, you’ve got a patient interacting with a variety of different healthcare providers whether it was out of our service, hospitals, primary care, research nurses, other hospitals, pharmacies Every time that patient interacts, the NHS number is entered into the system and as the NHS number is entered into the system, we’ll be able to extract that signal and analyze that signal So the safety team which is sitting right in the middle of that slide every day they would go into that big linked database that I’ve described to you in the slide before and they would open that up within the appropriate interface They have an interface for safety monitoring And they look at the alert that comes through and they analyze these alerts Some of these get discarded Some of these get investigated As I told you it was pre-market authorizations So at the beginning we had really set the bar very, very high so nothing was discarded but as the drug then sort of got market authorization we’re able to turn the dial now it down a little bit to try and reduce the noise that’s coming through So for example, if a patient turns up and gets a blood test and the blood test is normal, you know, we’re not interested in that But if they turn up in accident, in an emergency and they fractured a bone, we’re interested in that So once the safety team gets the signal, they analyze it, they look into the record, they contact the GP, they contact the hospital, if it’s another hospital, they get the history, speak to the PI who would be the GP of that particular patient and just gonna move to this slide, if you look at the purple or the pink colored square in the middle safety team will review the event then they will complete a serious adverse event on behalf of the PI Then the principal investigator will be asked to allocate causality and severity and then all of this gets submitted to the GSK The structure of the clinical safety team, please don’t judge us on that It was big but it was big for a reason Because we set the bar really quite high, ’cause we didn’t really know what to anticipate We had a pulmonologist and that was myself We had clinical research medics and they wear my clinical research fellows There were three of them There were 13 monitoring nurses There is a GSK medic, a dedicated technical support through Northwest eHealth, my colleague Martin Gibson gonna talk about in a second and we had a dedicated GSK staff within GSK Global Safety Office and we have one whole time equivalent who was looking at all the safety alerts that were coming through They were bombarded with that So it was what it was and obviously as time went on the volume of data that’s going through started to drop down but Elaine had already explained to you the amount of data that we eventually collected through the COPD and asthma This is my last slide and this is the publication of how we’ve conducted the safety monitoring, if you’re interested in learning a little bit more about that Within that publication there is a graph that describes pretty much what I’ve described to you in the previous slides Thank you very much – Thank you very much (audience applauding) Greg, you’re on – I hope my slides have got on here Oh, was I going backwards? – Yeah – Oh that’s the big arrow forward you were talking about – Yeah that seems to be tripping people up this morning – Okay but I do not have slides – [Mark] Oh, okay Do you all have anything? – It’s very prescient yesterday Somebody was talking about, oh, I need to always make sure I have a copy of my slides with me, so I learned from that yesterday – [Mark] Well please go ahead I apologize, I’ll see what we can do while you’re – So I want to talk about some of the complex challenges that exist for evaluating the relationship between a study drug and adverse events and other safety information This is true for real-world data and real-world evidence

as well as randomized controlled clinical trials So just because it’s real-world data, real-world evidence, they don’t get a free pass The same principles apply And in the FDA world there’s a big emphasis on aggregate reviews of safety data This goes all the way back to at least 1988 when the final guidance for the ISS, the Integrated Summary of Safety came out But also you can see it more recently with the FDA IND Safety Reporting Final Rule for ongoing aggregate safety evaluations and this provides us with many opportunities for sponsors and academia to partner with regulatory authorities for developing interdisciplinary program level safety evaluation procedures And this is just one example here Implementing randomization in a clinical care setting could help bridge the evidentiary gap between clinical research and practice This workshop is a great forum, a lot of good conversations going on, not just during the presentations and the Q&A but in between I had a couple interesting conversations with Bob Temple yesterday and he explained to me that the gap is not as big as I think it is or everybody else So I hope he asks a question ’cause I’d like some more discussion on this But regardless there is a lot of room, a lot of opportunities, for us to work together to develop these procedures Now, there’s been a lot of discussion recently about the difference of safety analysis, safety assessments compared to efficacy The rules for efficacy are very well understood yet studies are designed around efficacy and safety follows the same process But there’s been a lot of recognition recently that we need to reimagine how we do safety evaluation So we have developed a framework for safety evaluation Oh you did find them Sorry, next one, thank you And I’ve been involved in a lot of presentations with people from other sponsor organizations, from academic institutions, from the FDA and we have become very much aligned on a framework for safety evaluation The first and most important part of this, our component, is to engage critical team members You need a multidisciplinary approach even more than efficacy analysis You need a dynamic interactive collaboration and unlike efficacy where you have a single primary and a few key secondary efficacy endpoints and if you look at them multiple times you have to adjust for this and safety, you don’t know what you’re going to be looking for So it’s not a testing and confirming environment it’s a learning and decision-making environment So we have to reimagine how we look at safety So we don’t want tests or thresholds, we want quantitative frameworks to help the whole multidisciplinary safety team to measure the level of evidence so that the assessments can be product specific and the decisions can be driven by medical judgment And in the phrase, the catchphrase, that we’ve developed to codify this is, “Medical judgment within a quantitative framework.” So we don’t want statistical decision rules The other thing is that, the next thing is that we need to support the iterative nature of the process It emphasizes the continuum both throughout the whole program but also within individual studies And the last component is that we need to recognize different data sources The data sources are very different during clinical trial safety monitoring compared

to post-marketing safety surveillance but it’s also true for real-world data, in real-world evidence So my focus here in responding to what Nawar presented about the Salford study, which is an amazing undertaking And as they explained to me earlier today they over engineered the process and that’s a good thing because as the first one, there’s going to be things that develop that you didn’t anticipate There’s no previous example So how are they able to do this? Well the key to this and if you’ve been listening yesterday and today, there’s a lot of conversation about the data that are developed Whenever there’s health care going on, whenever there’s a healthcare procedure going on, there’s information that’s available But if we don’t capture that and get it entered into the the database we can’t analyze it So the key to the success of Salford and other studies like this, using real-world data to provide real-world evidence to enable regulatory decision-making is creating high quality interoperable networks of data that can be seamlessly leveraged for clinical and research purposes And if you have the data and not just data for individuals, right, you have an array of data sets that make up the whole database, you need to be able to connect what is happening for each patient’s experience with the healthcare that they received You gotta connect the the electronic health records with the claims and also the exposure, so the pharmacy data And you get that here Because they have a dedicated safety team that engages all of the critical team members Although I did notice that I didn’t see statistician on there somewhere So I think that that should be added, statistician or data analysts should be a core team member of the safety team And I’m sure that they were very involved because you can’t organize this kind of information for the safety team to evaluate on an ongoing basis like they did with the Salford study So you need a dedicated safety team They had access to the full EMR There’s a huge breadth and depth to that data They had ability to collect the HRU data which Nawar explained to me is the same as our claims data And prescribing data is also available, what is prescribed, dispensed and collected And so using the multidisciplinary safety team, they’re able to not only do the scientific evaluation of the integrated safety data at the end of the study, they’re able to do ongoing aggregate safety evaluation throughout the study So you can maintain the ongoing safety of the study throughout its lifecycle The other thing is, need to support the iterative nature of the process and this of course was handled because they had constant real-time data collection of all healthcare interventions And something to point out here, I mean we’re talking about a specific example with the Salford Lung Study This is one example and it worked very well in Salford because they had a number for each patient so they can connect all of the data together Now that doesn’t mean that we could do the exact same thing in the U.S But there’s other approaches, right, we can use the Sentinel System here There’s been several studies already that have been undertaken using the Sentinel System and there’s other ways of doing it And the last component of the framework which even though it’s real-world data, real-world evidence,

they need to follow the the same framework, is it recognizes different data sources used across the lifecycle of the product And this was done with the integrated electronic patient records, with real-time access So even though it’s a new way of collecting and analyzing data, it still needs to follow the the same framework And I think they they did a good job with the Salford Lung Study – [Mark] Thanks very much (audience applauding) Next is Ellis – I’m just– – There may be just loading the slides – Loading the slides, yeah I want to thank the organizers for inviting me to speak this morning I have to say I had some apprehension about speaking because most of people in this room I think have been thinking about real-world data, real-world evidence for a while And I’m firmly ensconced in the unreal world and really had not encountered much in the way of of real-world evidence or data And I was struck by this 65 million lines of data in the Salford Lung Study And I’m sure I’ve analyzed 65 million lines of data in my career but not in a study It’s really mind-blowing So with that introduction let’s see if I can get the slides out Okay, here we go Okay, so I wanted to give you some some quick points about safety monitoring and these are my thoughts, basically being here yesterday and thinking about it and I jotted these down last night First I’ll talk about some bad news which is the nature of safety analysis They are non-inferiority analysis and messiness biases towards the null and can obscure findings but there’s a lot of good news And again as a number of people this morning have said, we’re talking about post-approval studies, we’re talking about drugs that are labeled at least in one country And the good news is that once a drug has been approved, its risks and harms should be well characterized And so the safety monitoring in studies for new indications can be less intensive in many circumstances, may be less way less intensive So let’s talk a little bit about the bad news So the safety analysis are indeed non-inferior analysis So the messiness of patients discontinuing the drug or switching or the wrong diagnosis or you lose data, all that is messy And Bob Temple told you yesterday that for real-world evidence if you’re trying to show non-inferiority, the messiness biases towards proving that the drug is not inferior to the control and because safety analysis are inherently non-inferiority analysis the same thing apply So if everybody in the study just flushes their drug down the toilet on the first day you will invariably find that the drug is safe So the messiness is a problem in that regard But there’s good news which is if a drugs have been approved then you know the risks and harms should be well characterized So the safety monitoring can be less intensive And so one may not need to collect or analyze the non-serious adverse events All these things, routine physical exam, data, vital signs, weight, routine labs, EKGs, concomitant drugs and what we do need to collect are probably the serious adverse events and maybe adverse events of special interest if there are any So many of the serious adverse events will involve hospitalization and they’ll be captured in claims data Some serious adverse events, so I’m talking about an MI or heart attack A fracture may not involve hospitalization but it could involve a trip to the doctor and that’ll generate a claim You’re gonna miss some serious adverse events that don’t require a trip to the doctor I don’t know how they will be collected in claims data I was planning last night to take some large outcome trials and go through the data and look at the number, look at all the SAEs and show you a plot or a table of the serious adverse events in total and those that required, involved a hospitalization, and show you for the different serious adverse events

which ones involved a hospitalization, which one didn’t Unfortunately when I got home from the conference yesterday, I live locally, my power was out and so my router was down and I couldn’t do it But I think it’s pretty self-evident that the things like MI and stroke are captured in hospitalization and you’re gonna miss some serious adverse events and that’s just how it is The other thing you need special treatment for would be adverse events of special interest If there are any, you know, sometimes you’re interested in particular things, many times you’re not So this in part is relevant to real-world data but it’s also highly relevant to controlled trial data and this dovetails with Adrienne’s comment half an hour ago about I don’t know where you are but about this defensive research which is really I think a huge problem which is let’s collect everything because some regulator might ask us in a year, “Oh, well, what were the CBCs?” You know, what, who, you know all these nitpicking questions that require mountains of data We put out a guidance in 2016 It’s called, “Determining the Extent “of Safety Data Collection Needed in Late-Stage Premarket “and Postapproval Clinical Investigations.” And what it basically talks about is collecting less than usual safety data in those situations where the safety data are well characterized And my particular office oversees the Division of Cardiovascular and Renal Products and we’ve told many sponsors, “Look, don’t collect all this stuff in your next trial “You want to do a cardiovascular outcome trial, “that’s great, but we already know how many headaches “and hangnails the drug causes “Just collect the serious adverse events.” And they thank us and they shake their heads and they walk out the door and then when they send in the final protocol, they’re collecting everything And that happened time and time again And it dawned on us, well maybe the problem is regulators across the pond And so about four years ago, I think, we proposed this topic to ICH collecting less than full safety data collection and it was adopted And I’m glad to say I’m on the working group I’m glad to say that the Step 2 Guideline is out and open for public comment It’s called “Optimization of Safety Data Collection.” And the interesting thing was that a lot of the resistance in this approach was not coming from the regulators across the pond or in Asia, it was coming from companies who were just firmly entrenched in the idea that well they better collect it because they may need it And I would love it if someone during the comment period who works for a company has done one of these outcome studies say for an anti-diabetic drug, could tell me exactly what they learned by getting vital signs every lab, every concomitant drug, every non-serious adverse event Tell me what you learned and tell me what it cost you And I suspect you didn’t get much bang for your buck So this guideline’s out and I really do hope most of the people in the room would have a vested interest in this, you take a look at it And there’s a docket in the U.S., there are also dockets in other countries where you can submit your comments through official pathways to get them considered by the expert working group Thank you very much – Thank you (audience applauding) All right I want to thank all our presenters If there are any comments or questions especially like this discussion about efficient safety monitoring collection in the post market setting focusing on serious adverse events And those were some supplemental data collection may be needed for special interest adverse events We’ve just got time for maybe one or two but, Adrienne, I’ll start with you and then go over here – [Adrienne] Sure, so I’ve been involved in several diabetes CV outcome trials so I totally appreciate your comments So in– – So you’re contributing to defensive research – [Adrienne] Yeah well so yeah we did contribute to that So I have to acknowledge that and it’s really interesting, I mean those trials I think really changed the field ’cause it’s really changed the possibility in terms of improving cardiovascular health Now that came from a few pages of the CSR Now there was a really long list of pages of the CSR that didn’t change anything because the safety profile of different things, hypoglycemia, et cetera, are relatively known There were some questions that were asked for a lot of the GLP-1 trials like the question about medullary thyroid cancer as an example,

for which there are huge expenses for gathering calcitonin data So we have one of the largest calcitonin databases ever and if there’s anyone interested in that email me (audience laughing) And the number of cases of medullary thyroid cancer extremely small You couldn’t actually do, answer the question in any individual trial and actually even across trials And so I think that’s where we really need help with kind of again getting to kind of fit for purpose and this is a classic case so – You mentioned hypoglycemia, something you were monitoring for So this may sound revolutionary, but if you know that an anti-diabetic drug can cause hypoglycemia it doesn’t mean that in your safety study you need to collect information about hypoglycemia You already know it You need to monitor the patient, okay But you don’t need to write it down and send it to regulators – [Adrienne] That you’re serious adverse event comment – We already know that Just take care of the patient Don’t put it in the CRF – Amen (audience laughing) – Next. (laughs) Over here – [Man] After Elaine’s presentation and all the stakeholders that were involved in Salford Study I realized that all of them were national stakeholders and the particular situation in Europe is that we have one super national regulator which is the EMA And I was wondering if orders for the Salford Study you had some interactions in these regards and because that brings to another question that I think it’s very relevant for safety monitoring also Which is the the heterogeneity of health care systems in between countries and the ability and the feasibility of internationalizing this type of design And I don’t know if this is, I mean coming back to yesterday, a comment by Jesse Berlin, around the acknowledging the uncertainty Maybe it’s also relevant to acknowledge the complexity of health care in worldwide, for example, in the European Union and whether we need to embrace this complexity and heterogeneity and whether it’s feasible to embrace it and to make it part of our studies in this type of design – Sounds like Ellis is trying to embrace that with the the ICH alignment you talked about Nawar, do you want to comment on that or? – I think– – And I’m gonna try to get in two more questions where we need – The difference in between different health care systems, that’s inevitable, that’s gonna apply to any study that you do whether it was real world or not Obviously the impact on real-world studies, it’ll be different So it depends again on what we’ve heard yesterday in my view which is the question you’re trying to answer If the question that you’re trying to answer is relevant to, for example, healthcare resource utilization which you’ll have slightly different definition in different countries then you may need to take into account the fact that real-world evidence need to look differently in different countries Whilst if you look at effectiveness then I don’t think that’s gonna be a major problem That’s my take on this – We are gonna have to move this along There will be more time at the end again for comments I want to get to the other two questions quickly if we can – [Anika] I’m Anika Shonik My question really builds on Dr. Ellis Unger and Dr. Hernandez’s viewpoint which is regarding severities and whether we can think even more innovatively about pragmatic control arms derived from databases now that so many GLPs and other diabetic drugs are already in clinical use If we can use those kind of synthetic control arms or pragmatic control arms from EMR databases? – You’re talking about for, we’re, this is a discussion about safety – [Anika] Yes, to collect safety data if that can be used as supplemental information – Well I mean it would be difficult to compare to a control And safety as was mentioned a few minutes ago you’re not hypothesis testing anyway You’re exploring, but obviously you get less information than you would if you had a control That’s about all I can say – Thank you – Thank you – [Niklas] Hi, Niklas Berglind, AstraZeneca in Sweden And I wanted to also touch upon the ICH guidance on safety reporting and we are engaged in doing randomized clinical trials using registries and there obviously there’s a limitation of what data is available and I for one would, I mean we really welcome to see this guidance that recommends that you can for established drugs you don’t need to collect everything

But I think that one thing that you didn’t mention in your talk was discontinuation AEs Because that in the guidance that’s actually part of the mandatory collection and if we’re using data directly from the registry that’s not, you know, AE is leading to this continuation of drugs it’s not going to be possible to collect so And the openness to on a case-by-case basis to not do that and just hear your reflects on that – I think we have to think about these things on a case-by-case basis The reality is for a lot of these things, yes, it’s not gonna be a perfect study but you’re able to do the study because of the cost Because if you can reduce the cost, you can do more studies, you can answer more public health questions and so you have to be willing to make some compromises, I think – Yeah, it does seem like you’re in that ICH guidance is very relevant to the context here I want to thank our panel for a great discussion of these safety monitoring issues Thank you very much (audience applauding) All right For our last focus area before the break on maintaining data integrity I’d like to ask our speakers for this segment to come up We’re gonna consider the challenges of complying with good clinical practices and regulations related to data integrity and audits Our speakers for this session include Martin Gibson, the chief executive of Northwest eHealth, and consultant physician at Salford Royal and HS Foundation Trust Michael O’Neal, chief medical officer at Bio Clinica and Paul Harris, professor of biomedical informatics and biomedical engineering and the director of the Office of Research Informatics at Vanderbilt Martin, you’re gonna kick us off – Thanks very much for the introduction and for inviting us here It’s been an amazingly interesting meeting Thank you These are my conflicts of interests So the great news from your point of view is that I’m the third speaker in a series of three and whilst I was able to review the slides of my colleagues what I wasn’t able to understand is what they were going to say to them (audience laughing) So they’ve pretty much said everything I was gonna say So the good news from your point of view is that I won’t take quite as long So when we were discussing how we go through this particular session these were the kind of topics that people thought that we should be talking about So keeping it real, so trying to maximize external validity what are the things around GCP, you’ve heard a little bit about that already Monitoring, what’s different and what’s the same and what are the opportunities here for change What about data quality, what about data integrity, how on earth can you actually review or inspect all this kind of system And then there was a question about patient privacy and how is that handled So I’m gonna deal with that last time ’cause I don’t have a slide on that last one The answer to that last one is pretty straightforward It was dealt with in exactly the same way it is for every other randomized control trial that you’ve ever heard of So that one’s easy And I think Elaine mentions I’m not sure it was four years beforehand but it was certainly quite a long time beforehand that we started to think about how we do this study and this is really quite important because if you get this wrong you’re in serious trouble and feasibility is everything with every trial but in a real-world trial you’ve got to think about lots of other things So what kind of safety endpoints you gonna look at? How you’re gonna capture them? how do you know that the quality of the data is gonna be sufficient? How do you even know what an exacerbation of COPD is in a data set? How can you describe that? How can you validate that? How can you be sure that the regulators are gonna be okay with what you’re thinking about? And then it’s gonna provide data for payers which was important for this particular study And again Elaine’s mentioned that this was one of the first joint forums that we went to And we went there, I have to say on that particular occasion we were very nervous because nobody had ever tried to do anything like this before And as we’ve said, we probably over engineered it which may have added a little bit to the cost And then there’s the operationalization, how do you do this? And Elaine again is mentioned where you have to work with all of the different providers and stakeholders in the system of which there were very many I won’t go through the list So I think there was a conversation yesterday, that lays a case about a trial where it’s real-world

but by the time you get down to the people that actually volunteered for it it was only about 1 in 40 people that would have actually made it So there’s the opportunity there for selection bias So one of the things we wanted to do with this study upfront was to make sure that we were actually trying to get nearly everybody we could That would meet the ending criteria in our local population We’re having the same issue as yesterday It’s great (audience laughing) – [Mark] Was that it? – I’m never speaking on behalf of the hotel (audience laughing) So in the Salford and surrounding areas there were as you can see on this which was published in the “European Respiratory Journal” last year, 11,700 people met the entry criteria Sorry there were 11,000 people with with COPD of which 50% met the entry criteria for the Salford Lung Studies COPD Trial of which we recruited 50% And I think that’s a really important thing because we can actually say when we’re talking about a real-world trial that this really did recruit the real people that represent the population And how many of those that have actually got into a standard kind of exacerbation study Well that’s the one over right at the right hand side which is actually only 841 of those people would have got into a typical COPD Phase III Study This is my version of the picture that everybody else has shown but I think mine’s prettier (audience laughing) And the real reason is just to tell you about GCP and it’s really to mention again that there were a vast number of people involved in this, you know, in each GP surgery we trained half a dozen people In the hospitals there were several people, in the pharmacies, over 130 pharmacies with all that churn of people going through So again over 2,000 people needed to be trained Now you have to ask yourself was that truly required and necessary and we again, it’s one of the things that we probably wouldn’t do again but we over engineered it So what about the data and monitoring the data? So this is actually, and I should preface this by saying that people over the last day or two have said, “There’s two real aspects to this “There’s a technology aspect and there’s a people aspect “And the far more difficult bit is the people aspect.” And I would agree with that But don’t underestimate the technology aspect either because it’s pretty bad So we had all of the different systems that were available out there and although we have this integrated data system it doesn’t mean that there weren’t lots of legacy systems It doesn’t mean they weren’t changing all the time as we went through One of the big providers of GP software systems in our neighborhood is produced by an organization called EMOS and they decided to move from just the information being in the GP surgery to a web-based system halfway through the trial Didn’t bother telling us they were gonna do that They didn’t roll it out across all the GP surgeries simultaneously They did it in dribs and drabs So you have to have a system that sits on top of that that can make sure that you’re pulling all the data in and you’re not losing anything So you know if your data to drop out, you need a feed alert, which you can see here Now that’s easy But what if the feed alert has dropped out and how do you interpret that? Does that mean there’s no data or does it mean they’ve switched the computer off? So you have to have that constant refreshing system that you can see here with those little blue arrows and if one of those goes down and that feed is no longer live, you have to alert for that and you have to look into that And our technical team had to do this all the time Similarly these are all the people in your trial and you should be getting data on them, pretty much all the time So if somebody disappears from your trial that’s not a good thing So you need to know that that’s happened and you need to have a system that can say what happened, where is it happened, how do we maintain that data flow for that individual, where have they gone And then there’s the data itself You’re expecting it to arrive in a particular format but what if it doesn’t? And that was not an infrequent occurrence So if the data arrives in a incorrect format or you get too much of it or too little of it, you’re gonna have a system that allows you to look into that as well This is my version of Nawar’s slide and I just have one, this is the 24-hour, so real time at the top from the hospitals 24 hour from the middle bit which is primary care and less frequent for the national systems down at the bottom All that feeds into our activity summary system

which is much more beautiful than it was for when we did it for the Salford Lung Study but it basically allows you to drill into those alerts I’ll give you one anecdote of how good this is One morning an orderly on the team got an alert that one of the patients in the COPD trial might be pregnant This is a bit of a surprise seeing as the average age is 61 (audience laughing) So what we’re able to do is to drill into that alert and say, well how did this come about? And when you do that you find that it was an administrator in the hospital who instead, accidentally, of making a patient a follow-up appointment for the respiratory clinic, made them by mistake a follow-up appointment for an antenatal clinic The system picked up it was antenatal clinic and it said this could be a pregnancy So it tells you how good it is and you can make it very sophisticated but you have a price to pay for that So here’s some other good stuff and some bad stuff and we’ve talked about that So if you take some similar Phase III COPD trials and you look at the Salford Lung Studies COPD Trial, we had a lower rate of withdrawals, a good thing A higher rate of SAEs What does that mean? A higher rate of pneumonia SAEs doesn’t sound good but a very similar rate of adverse drug reactions Now this is actually a sort of victims of the population that we’ve recruited We’ve recruited an all-comers population They’re much sicker, they’re all smoking, well not all of ’em We don’t recommend that But there’s a lot of people with cancer in there All sorts of other stuff And so they’re a sicker population So you might expect to see that kind of higher level of serious adverse events And again that shows up here in if you compare on the left-hand side the SLS with some of the similar kinds of traditional COPD trials But this is exacerbations This is your outcome This is the thing you’re interested in So you’re seeing about double that risk of exacerbation and again it’s representing the sicker population I’ve heard people say that we don’t want to put our drug into this all-comers population ’cause we might dilute the result or we might not see what we want to see But actually you might see, you might be pleasantly surprised because what you’re seeing is the the actual size of the effect is probably the same if not greater in those sicker patients And that’s something that these real-world trials can bring out This is to remind you that we get a lot of data from a lot of different sources Now the bottom right and over the right hand side you can see nowadays we’re being asked to bring all these other data in So around ‘omics and wearables and so on, huge amounts of data flow And of course the nice thing about that is it’s lots of data The really bad thing about that is, I think somebody said it a meeting like this, I came to you before? The great thing about standards, they’re like toothbrushes, you know everyone’s got one but nobody wants to share (audience laughing) So we have to have a system that brings all those different data standards together and this is our system So top left you’ve got the recoding system for GPs in the pink The top right is ICD-10 codes from the hospital We bring all those into the middle which is the orange concepts which is SNOMED CT and we can now output those in to SDTM or OMAP We didn’t have that when we did the Salford Lung Study We sent them raw data, sorry about that But we’ve improved it since Inspection, we had multiple inspections throughout from the GSK internal audit system team I have to say both they and us learned a lot from that process because, as Elaine said, it was very different We had to check on system validity We have to be able to prove that we could track the data We had to be able to do site level inspections in the same way that you would any others and then there was the small but very significant issue of archiving How do you archive a real-world data set? It’s enormous and you have to archive it for decades possibly And so if you do that and you stick it into a current system how do you know that when you get it out you’ve still got compatibility with any kind of system that could open it up and show you what it was in there? So we had to develop a system for doing that as well So last slide, I think what it gives me confidence in is that you can actually use these technologies and you can link EHRs to do pre-licensed trials It’s possible, we’ve done it This was a particularly special one but you could do it for any, somebody’s been talking a lot about hybrid trials This was a hybrid trial Actually all trials are hybrid trials if you think about it because you take the data from a bit of paper and you put it into an eCRF and so it’s a hybrid

All that we’re changing is the different pots of hybrid-ness, if that’s a word Safety monitoring of this type of study, I think it’s effective, it’s different In some ways it’s more challenging, in some ways the closer to real time and the fact it’s highly configurable gives you some major pluses And I think when we start to do this in a more widespread way we should be able to improve this further and it should actually lend itself to adaptive trial designs which is again something people talk about a lot Thank you very much – Thank you (audience applauding) All right, Paul, go ahead – So I’m Paul Harris from Vanderbilt and just sort of following on to that last talk or actually the last several talks, I believe that a key to data integrity is getting as close to the source system as you can in as timely a manner as you can And so really want to talk a little bit about some of the work that we’ve been doing around connecting EHR systems to EDC systems Just start with a really, really busy slide here We’ll spend about 14 seconds on it Just to tell you that we’re gonna be talking about or I’m gonna be talking about a platform called REDCap that is a configurable system, it’s disease neutral, it’s sort of intent neutral We built it at Vanderbilt about 15 years ago to support all sorts of research studies and clinical and translational studies and trials The one graphic I’d point out there is the Swiss Army knife People, a lot of folks don’t need all of those modules and so the deal is you sort of put the modules in that makes sense for your study We make it available at no cost to academic, nonprofit and government organizations and a lot of people, a lot of institutions have a license and are using it regularly for a lot of studies and trials with this combination of a Swiss Army knife and a lot of really, really smart users around the world We see a couple of things Number one they use it in all sorts of ways that we never envisioned that they would use it which is great That drives new use cases and they use it for all sorts of scientific studies and projects And so here you see a bibliometric analysis of some of those publications that have cited REDCap The thing I’d point you to though is the bottom right, lots of great ideas for new functionality Put smart people with configurable tools and they’ll tell you what’s great about it and they’ll tell you what would be really, really great if we could do this This has often been, can we connect it to the electronic health record system? That would that would help us presumably with accuracy, timeliness of data and efficiency So we’ve been working for the last two years, primarily with Epic but we’re starting to work with Cerner as well to create a module that can be plugged into a local REDCap Epic ecosystem to allow the two systems to work well together There’s two elements here that are really, really important Workflow, we can get the REDCap screens, the EDC system screens inside of normal workflow for the providers working on that for patients as well now But getting real estate and then getting the data flow so mapping fields from the EHR system to the local EDC system that’s sort of done on a project-by-project basis But once you have it set up then the data just flows in in near or actual real time The really, really important piece here is that Health IT is required for initial setup That Health IT is not required for each individual study afterwards And if you work in sort of academic medicine and particularly informatics you know that that is a huge deal because it’s really hard to get prioritization from Health IT to do anything other than clinical clinical system work So getting that module up and working and I’ll show you some statistics on that in just a moment We’ve now sort of pivoted to, well how could we create efficiency for multicenter studies and trials and the model’s pretty simple If you can get a local REDCap or a local EDC system talking to a local EHR system which the Health IT folks and the security and privacy folks seem to be comfortable with If you can get that working and you can get data flow and whatever workflow you need for an individual study working then it’s just a short hop for being able

to de-identify that data and push it or pull it to a data coordinating center version, sorry a DCC, for centralized collection and reporting So as I mentioned we’ve been working on this for about a couple of years with Epic We’ve built it out in a way that allows, standards-based way so that it actually works in Cerner pretty well also now But we’ve got more traction with Epic because of a collaboratory that we have with their engineers We have about 24 institutions, academic institutions, that are Epic partners that are also using REDCap that have signed on and they’re in the process, various processes for getting it set up We’ve been using it for some time at Vanderbilt and you can see the statistics on some of our single site projects there in the top right corner We’ve also got, thank you, we’ve also got a study, sort of a comparison study, where we’re looking at data from an existing NIH trial called CLOVERS we’re looking at, we’ve got a platform study going on using the technology We’re calling it Flowerbed for that particular project But seeing if we can sort of get equivalence between what we were able to do in traditional manners by in CLOVERS with what we can do using just EHR methods It’s working quite well and I’ll give you a few of those lessons learned in just a moment And then finally looking forward Building this and sort of thinking about it in this generalized and abstracted way we think we can put it into a lot more studies and trials And so we’re working with the CTSA network, the trial innovation network within that NIH funded initiative to see if we can put this in a sort of a disruptive innovation into more multicenter studies and trials A few lessons learned so far This has been said several times, including in the last talk, then Dr. Gibson’s comments that the easiest part is technical As always it’s the governance and the governance first and then just again building in the integration pieces even if they’re light takes a lot of prioritization, a lot of committee meetings and that sort of thing Technical piece in this part, in this case is not trivial but again is the easiest part All Health IT groups are busy and they have competing priorities I found for individual studies going to my local Health IT, it’s an uphill battle, even if it’s the most important research trial coming to Vanderbilt in the next year It’s just hard to get that prioritization But one of the things we’re finding is that we can convince the Health IT groups, if you’ll just do this once then it’ll actually be a pressure relief valve and you won’t need to do it many, many times So that philosophy is actually working well You saw that with the different groups that are signing up in the last slide This is kind of surprising It shouldn’t be but in the real world I wondered about it When you do the mappings from a local EDC system to a local EHR system for the data, you know, what do we call labs at Stanford versus Vanderbilt versus Mayo Clinic et cetera Do they map or do they transfer? Surprisingly they transfer pretty well, not perfect but they transfer pretty well And so we do, that increases our enthusiasm for building an efficiency of setup and so forth We’ve done some anecdotal work Some people have done some actual rigorous work around the gains for getting that closed system, the EHR to the EDC system and in fact we see those increases of efficiency, data quality and timeliness that one would expect, Early indicators for our multicenter study model indicate that sites are gonna be, prefer to push data to the DCC rather than having automated systems that pull it I think that’s more privacy folks that are a little bit nervous and want to go walk before they run And then finally and this is probably the most important part, just getting in and working the first view of these studies where we’ve got everything wired up It’s amazing, you know, once you’ve got the real estate, once you’ve got the data flow, the use cases just start piling on it So we’re really looking forward to future evolution here Thank you very much (audience applauding) – [Mark] You ready to come now? – Hi, my name is Mike O’Neill I come from an imaging core lab We have a tremendous database of validated oncology trials So that really gives us a gold standard that can be used when developing these real-world trials We’re working with several sponsors right now to look at

both retrospective data and also helping to design prospective trials using the data that we have The advantage of the data is that it’s, you know, it was done under a controlled basis It’s an unbiased evaluation and we use response criteria So our thinking is that to move this into the the real-world setting we can take lessons learned from our trials and put the required, basically the required operational procedures and the forms needed to come up with appropriate responses Generally we work with RECIST and other normal response criteria for oncology So these are very easy to enforce into an eCRF and also into the electronic health record and it can be transferred directly to your imaging core lab And you could basically have a site read and essential read as opposed to the standard two central readers with an adjudication So again we’re looking at this data retrospectively and also we’re designing some prospective trials to see if it’s effective We’re very excited about the use of real-world data and the electronic medical record data We struggle to import data from the sites We don’t just look at the radiology which is very easy to get from the sites, very objective data, but to give a complete overview of response or progression We need to incorporate the clinical data that the clinicians are seeing at this site and that data is very difficult to get It requires a lot of transcription from the sites or study coordinators and there’s a big lag So if we can directly link the medical record to the clinical trial or to the independent reviews facility doing the clinical trial it saves a lot of time I think the data would be much cleaner We would get it much faster And there’s data that we request that we don’t even know if we really need but we do request it and because we work in a blinded fashion, we don’t have access to any prior records or radiology data So we ask for things like benign metastatic mimics which could be measured as metastatic lesions The good example would be a hemangioma liver which is very difficult to distinguish from a liver metastasis So all this information should be included during the history and physical in their record or in some prior radiology report So this is something we could get, would be clean, we would know that it was accurate and it would facilitate the review Also eligibility data would be very important Some of these patients have to failed multiple therapies and if that data was transferred to us we would know that they were actually eligible for the trial we could continue to read them without any delays So again we’re looking forward to having that data available to us We feel that it will be reliable, attributable, accessible and much more efficient for running our trials in oncology Data collection, aside from the radiology part, is very very challenging So next slide, please Oh, I do it myself, sorry. (laughs) Okay so those are the advantages We think they’re tremendous amount advantages We’re really excited about working with the sponsors now to develop these new trials that are hybrid trials and also looking at some of the investigator initiated studies that were never centrally a read, bringing them in-house, reading them and comparing them to the data that they have from the real-world data to see where the gaps are and how we can improve the process going forward Our challenges are similar to what everyone’s mentioned today When is the EHR clean and suitable for the clinical trial use endpoint Again that’s something we’re not that familiar with Data privacy is another issue And some variability criteria used for analysis and that’s important Not everyone is trained in it but again we feel looking at the retrospective data that we have, if we can incorporate the requirements into the electronic health record and the CRF, we can actually enforce the rules and have that filled out at the site Systems talking each other, it sounds like it’s not as big a problem as we’ve run into so far But again we’re just early in the process And then the radiology and oncology training and background can be different So standard of care can differ across sites It’s a little bit easier I think to standardize care in an oncology trial because there’s a routine schedule of events that have to happen with respect to imaging and the good clinical practices So we’re really excited to work with all of you

and figuring out the best way to move forward with this There’s multiple options, multiple lessons learned and we do have huge databases that I think can be very helpful in supporting this effort Thank you – Thank you (audience applauding) If you look at the clock, we’re running a little bit behind These have been really good presentation If there is a question or two now, we’ll try to fit it in And since this public comment, we’ll have a chance for also addressing these later on I would just like to say that it seems like a big theme has been making sure that all features of a Rule 11 study are fit for purpose, for the particular study And that certainly goes for data But what I took away from several of you was that while there is a focus now on getting some initial studies going, you all seem to feel like there’s a capacity to really build an infrastructure at the same time, using these early cases I mean, Paul you were saying, the costs seem pile on which I think if we pay explicit attention, to using these early studies to build out those capabilities that the cost should come down So time for maybe one question – [Luke] Hi, Mark Hi, Mark, I’m Luke from the Food and Drug Administration I have a question about data integrity in the context of laboratory tests So as you know, I guess in the UK you may not have the problem of different labs using very various testing For example, Theracoasts probably could not have happened in UK, that kind of issue But with specific certain specific laboratory test there’s quite a bit of variability in the United States with regards to how they’re done and data standards Are there ways we can ensure that the data that we get from real-world data are correlated For example, sending standards to the labs to test and double-check or sending duplicate samples to see that we get the same value, from the same test sample from a specific laboratory Do you advocate for any of those for data integrity testing? Thank you – Sorry, from a UK perspective, all of the labs, their internal quality audits which are national And again it really, it’s about that front piece of work It’s understanding what lab tests you’re actually interested in and actually doing the spade work and working out who’s doing what, how are they analyzing it, and making sure you know that And making sure that you know that that’s not gonna change or if it does change throughout your study, understanding what that change is If there are specific things that are important to your trial that are lab based you might still want to use central labs Yeah, you’ve still got that option with a hybrid approach – [Luke] Yeah, I was wondering how you would integrate a core lab into a real-world study That’s an additional expense perhaps for that kind of– – It is, but it it’s super important, maybe you have to do that I mean, we talked about practical pragmatic trials That’s just one of the decisions you have to make it from But if it’s something like an A1c for example, they’re all DCCT aligned so it’s not really gonna be a problem – [Luke] It depends on the test and how specific it is – We’re gonna have to – Thank you – Any other final comments from the other panelists on this? All right I want to thank our panel for an excellent discussion of data integrity issue Thank you (audience applauding) So we are scheduled to take a break now I’d like to ask you to make that break 10 minutes There are drinks coffee in the back and we’ll have our panelists for our final session come up I appreciate their dealing with our running a few minutes late here and then we’ll have some time for public comment on everything else after that Thank you all very much All right thank you, thank you all for, again for being with us today and in this final session we’re gonna try to bring some issues together The sessions on building a framework for randomized clinical trials, barriers, enablers and infrastructures and in this session we want to consider the key mechanisms for developing the infrastructure needed to leverage real-world data in the kinds of studies that we’ve been talking about in this workshop So what we want to do in particular is zero in on the data infrastructure needs and the other key enablers or barriers to get to the infrastructure that can implement the trials And this is I think, a nice follow-on to the last session and to some of the points that were are made about how while each individual study needs to focus on fit for purpose issues and needs to focus on minimizing the cost, maximizing the efficiency of getting that study done, these are all part of the larger shift taking place towards greater use of real-world evidence somewhere on this continuum of studies

And some attention to those larger implications as well, should also enable more progress with the use of real-world evidence in the post-market setting So I’d like to introduce our speakers for this We’re gonna hear first from Lesley Curtis, the chair and professor Department of Population Health Sciences and interim executive director of the Duke Clinical Research Institutes Her list of titles keeps getting longer and Joanne Waldstreicher, the chief medical officer at Johnson & Johnson and Peter Stein from Office of New Drugs at CDER will be joining us shortly So Lesley, I’d like to turn over to you – Great thanks, thanks, Mark And thanks for really what has been a terrific day and a half I’ve certainly enjoyed the discussions and think that we’ve managed to move a lot of issues to the fore in important ways As I’ve reflected on the conversations, a couple of things have certainly come to mind And the first is we began, I would say, with some maybe more philosophical discussions about real-world data and real-world evidence And I think we can all agree that there are situations and examples in which real-world data, real-world evidence, are not appropriate And I would urge us to agree to agree on that and then dig in to identify those questions, circumstances, in which it is appropriate and then develop the evidence base that moves this forward So rather than again, rather than focus on those examples, we all know where it doesn’t make sense Let’s dive down and identify where it does and then define how to do that best An example was raised about structured data capture Great example, right So can we begin to really talk about when that matters, when that is cost-efficient, when that helps and when it may not be the solution And there are many other examples of these So that’s just one thought I think there’s also been an undercurrent or maybe an unstated assumption that high quality and high cost are somehow the same thing when it comes to clinical trials Only a few of us spoke about or even mentioned the resource constraints and cost of clinical trials I’d like us to really explore, investigate that assumption I’m not sure that high cost and/or lower cost necessarily means lower quality And let’s be sure that we push in that area We could probably do that through some retrospective analysis of retrospective case studies if you will of trials that have been completed and we could probably also do that with some empirical work But really important for us to push that forward One of my colleagues, Lisa Berdan from the DCRI, I noted that there’s probably what we may see in terms of what a sponsor does in the next trial, may be a direct reflection of the most negative feedback that they received in the prior trial when it went before regulators And so just making sure that we’re clear about where we’re driving costs in the system And then the only the other point that I think is important to raise and this is one that I’m sure that Peter Stein will touch on is more specific guidance and clarity about what is actually needed in the context of real-world evidence, real-world data for those post-approval studies The closer we get to that, the more we move down that area or that, yes, that the more we move down in that, we will I think help people move the ball forward much more quickly So those are just three sort of general thoughts – Great, thanks very much Lesley, and next we turn to Joann – Great, thank you so much and really I want to applaud FDA and Duke Margolis for bringing us all together And also want to thank those who shared the specific examples I think it’s a great example of what we need to do forward, an abler as you said in you’re opening, for helping us all advance the field I want to get back to a few points that were made yesterday which I thought were really critical And the first one from Jacqueline from the FDA is really we want to bring clinical research and practice together for a joint goal And of course for all of these types of studies there can be two goals

One is for regulatory purposes and the other is for generating real-world data for important reasons but for non-regulatory purposes And I think it’s important as several of the other speakers have done to really separate that out because for regulatory purposes as Peter mentioned, we need the believability, we need to isolate the efficacy and the safety for public health reasons And so that is I think what the critical part that I would like to focus on And that I think we and industry need to focus much more on as well Although I’ll add of course, generating real-world data is also extremely important for all stakeholders but I want to put that aside for a second and really just focus on generating data for regulatory purposes And the goal I think that we all have by bringing together the clinical research and practice together is by getting much more efficient, not just from the forum perspective but also from the patient perspective Making sure all the patient data is in one place, one electronic place let’s say So patient doesn’t have two isolated records of his or her medical history floating around that never get connected It’s also better for investigators so that they only have to enter the data once And I think one of the speaker said and doing it with high quality the first time and doing it very completely maybe adding some more things to the EHR is, it would be best And then of course it’s better for the companies I think we’ve talked and Adrienne really talked about this the opportunity cost especially for monitoring visits and for comparing source documents to clinical research forms That’s a huge opportunity cost that none of us, none of the stakeholders can really afford at this point in the cycle where there are so many important questions that we have to answer and just not enough resources, not just from the companies but investigators, patients, et cetera, to really go around And so there’s really an opportunity to put these and to start with these hybrid approaches to leverage health systems data such as was mentioned yesterday from the researcher from the Brigham, Dr. Piantedosi and also from O’Neal today, really using electronic health records and getting more data in those electronic health records so that they can be of high quality which is critical As you said we have to have that high quality because we have to generate the the excellent data for the health authorities to make good decisions for public health but also in a much more cost-efficient and single way that feeds into the learning healthcare system that helps everyone and stops all of the double work and double efforts And I think what maybe we went the next time we have this meeting we can include even other stakeholders, other groups that are working on bringing those together like we heard an example today we heard from PCORnet, PCORI, maybe we could hear from Apple next time or Google or Hugo or some of these other companies that are trying to do this in a very high-quality way We also need to learn from experiences when we see a new approach Peter, you mentioned about adjudication, there have been many publications about adjudication maybe not being necessary Has that translated into regulatory requirements yet I don’t know, I mean it’s probably a case-by-case basis but it’s a good example of a situation where there’s a lot of data in the literature and maybe we can just try to start moving forward and getting those into regulatory guidelines Another example is in new endpoints Endpoints that are relevant from the real-world data perspective but that could also be evaluated from the regulatory perspective And there this would be I think a real enabler is a group like Friends of Cancer Research that is really a neutral third party, not a company, not a regulator, working to validate novel endpoints that are in electronic health record data for clinical research purposes And I really think that moving forward, having groups like that like Friends of Cancer Research or groups that have infrastructure like PCORI, PCORnet, I think bringing forward new endpoints, new infrastructure, new ways of doing things and then getting those within the guidelines or getting some sort of validation or some sort of documentation from FDA so we can move from the case-by-case basis where I think we are in now where I think we have to be but move to getting it more specific and more documented Maybe even by therapeutic area or by study type I think that that would help everyone and I recognize that we’re in early days I want to say one thing about the barriers and I think several barriers have been brought up today and sometimes I read and see that people look to FDA

as a barrier they might look to industry as a barrier It’s yes, and yes I think there is reality The reality is for a company if you’re looking at a post-approval study, let’s say a second or a third or a fourth or a seventh indication the company wants to be sure that the only variable in terms of the success of the efficacy and the safety readout of the study, the only variable should be the drug and whether it works or not or whether it’s safe or not Not the quality of the study, not the fact that it wasn’t monitored appropriately or there was some problem with the database and it’s not worth taking any risk The risk adversity is just extremely high when you’re going for a new indication And also it’s a timeline issue The FDA has been very open to discussing these kinds of studies but of course it’s gonna take time to go through, to figure out We heard that today about the Salford Study It’ll take time It’ll take time going back and forth negotiating or working it through and I’m not criticizing I think that’s just a reality And so for a company that time may not be worth it if they can just go back to a traditional study even if it costs much more money But these are opportunities I think and really reinforce the importance of when something is validated, when something will work, when it’s possible to put pen to paper and get new guidance on new approaches and validation with new endpoints let’s be sure we put those down Let’s be sure we try to share them We have neutral third parties like we do here today with Duke Margolis or maybe some other forum for fora that we can all share these learnings and move forward together – [Mark] Thank you, thanks next, Peter – Great, I think I have the opportunity just to echo what Lesley and Joanne just said because I certainly agree with their comments I think one thing that strikes me about today’s and yesterday’s discussions is that this is really a work in progress in many ways And I don’t think that the concept of sort of real-world evidence, randomized clinical trials is sort of a black-and-white one where there are it’s simply, you do a trial which is gonna be called a real-world evidence randomized trial and then there’s this traditional trial on the left and these are completely separate entities I think what we’re seeing is that there will be and I think maybe hybrid isn’t even the right term Because it implies I think something different I think there are a real-world elements that can be moved into traditional trials probably already and then there will be trials that may ultimately be what we might at some point consider a pure real-world trial with randomization being the one sort of more traditional component if you will So there’s a spectrum and I think what it will always evolve to, I hope it will evolve to is a setting in which we pick out elements that are the real-world elements that may improve efficiency cost and even improve the kind of data that we can obtain in a selective way focused on what the objectives of the trial in the first place were We’re really in an evolutionary stage as we’re thinking about, this isn’t just starting in last years I mean real-world types of trials, Bob mentioned the Jesse trial go back decades But I think we are in an era where the infrastructure support for these kinds of trials is rapidly improving which is a critical element in all of this So I think this is something that I think as we think about focusing on these different elements, we can begin I think to think about where different elements are appropriate I was interested in the Salford Lung discussion because there I would almost look at that as it as a traditional trial with a couple of real-world data elements that were introduced It was in real-world sites but those real-world sites were trained up to be really like traditional sites And I would say that if I ran a traditional trial that had that good safety monitoring for SAEs, I’d be very impressed I’ve read many trials that I can’t say that I had systems where hospitalization was, I would find about hospitalization five minutes after it occurred – [Mark] Daily – That’s fabulous but it’s but again obviously complex But as we design those kinds of systems the first time and I’m sure in that study was a very complex and expensive effort to design that safety system but once you have that model and the infrastructure is set up, the next study becomes much more feasible And if you want to call that real-world that’s great But it’s a wonderful way of assuring, I think, comprehensive safety monitoring for serious adverse events And as Ellis Unger mentioned those are the kinds of information that we often want later in the development of a drug as opposed to earlier when more detailed safety information would be necessary and that infrastructure probably isn’t quite right to obtain that kind of more detailed information on non-serious adverse events

or standardized laboratory types of studies So I guess the first thing I’d emphasize, it’s really about what are you trying to figure out What is the study objectives and is the real-world data elements that you’re interested capable of achieving those objectives We talked about the need for blinding Well sometimes you do need blinding and sometimes blinding may not be necessary and I won’t go back over some of those settings but I think that’s a question you could ask once you’re going to do a randomized trial What are the endpoints you’re using? Are those endpoints obtainable within the context of a real-world data set? If they are that may be that, and are robust in that data set, that’s great Many, many endpoints are not If it’s a subjective endpoint, a PRO, or other such measures it may not be obtainable Although I will say that with digital biomarkers and I don’t have an iWatch but you know with those kinds of endpoints that can be obtained, more and more things may be available through real-world data sources than are available now We may be able to validate those kinds of instruments which can come in through real-world data sources where right now we need a patient to fill out a form which is not in the context of real-world data There are a lot of challenges I mean we have the issue around data quality and certainly developing systems that can improve that but we do worry about converting data that was really generated for healthcare interactions into data that is research et reliable And that is improving over time but it will always be a challenge Physicians are obtaining data from patients that weren’t intended to answer questions regarding research and now we’re trying to rely on that data Is it reliable? Well we’ll find out more and more about its reliability and there’ll be ways of improving that, I hope, over time We certainly right now don’t really have an infrastructure, I think, that fully supports these efforts it’s highly fractionated We know patients move from one care system to another even in terms of who the payer is So the use of EHR claims, can we really have adequate follow-up of patients That’s something that we need to address Can we adequately link EHR and claim and laboratory data? Well, we’re trying, many efforts have been ongoing and as those efforts evolve it’ll give us much greater capabilities But right now those capabilities in most systems are pretty limited There have been efforts and we talked about one of those yesterday to introduce EHR research elements into EHR That’s great but you know most physicians who have three minutes to see a patient, probably aren’t gonna be too happy about filling in 10 more elements Maybe two more elements they might be, maybe one, but those are the kinds of efforts that we’ll have to at least explore to see if they have some opportunity for us I mean there clearly are opportunities here We can broaden the patient exposure, as I said I sort of bridled yesterday at thinking about the real-world evidence versus sort of fake-world evidence in terms of traditional trials I don’t quite view the world that way I think it’s an evolution But I certainly think that there’s value in the greater exposure, the greater number of patients finding out safety information that might be rare events or in populations that were not studied previously, particularly ones that are hypothesis based questions An example yesterday where, in a drug where it was not studied in smokers, it certainly is a relevant question for an asthma medication what is the response in smokers That’s a hypothesis based question Our labeling likely, if it wasn’t studied in smokers, would limit its use in that setting Well what is the response in those patients? Or if there’s another reason why the drug might not respond for example a drug study late in an oncological occasion might it work for prevention or might it work very early on? Our labeling would limit it to later use But certainly the very relevant questions to broaden the experience and real-world evidence might be a source for answering some of those very important questions There are other questions that aren’t really regulatory We talked about adherence, a very important issue Bob mentioned yesterday that one of the big health challenges we have is that patients don’t take their medication And so knowing the differences between an inhaler, for example, or a pill in adherence That’s a relevant question May not necessarily have regulatory impact or it might but it’s a clinical question of great moment and great importance And I think it needs to be studied and real-world evidence type studies could potentially address exactly that Certainly there are other issues I think real-world data, Bob mentioned also, the use for recruitment and retention which we heard other speakers comment on that I think that’s a terrific potential use to leverage these kinds of large data sets to bring more patients into studies I will say, I mean Joanne mentioned the role the FDA plays here And I’ll mention a couple things in that regard

One is that I hope we’re not a lagging indicator in general for things but we certainly want to understand where the field is going before you jump in and accept how new approaches are to be used But we’re trying to be part of the process Obviously that’s why we’re here in large numbers today and in other Duke Margolis and other meetings that have occurred We’re trying to learn this field better We have a real-world evidence subcommittee that’s actively looking at proposals and thinking about them and trying to understand how they would meet our regulatory decision needs And I think that provides an opportunity for companies to have interactions with us and to find out what our perspective is on different designs and I particularly encourage companies to come to us with randomized trials using world data elements or even entirely in a real-world data setting I can’t say we’ll accept it but you’ll hear a very interactive and a very, I think, full discussion of those elements and if you’ve really thought it through, I think you may find that we are receptive, particularly for studies that are randomized We are working on guidances and certainly this is part of an effort to help develop those guidances and I completely agree, Joanne, that the more we can be clear about providing guidance on our thinking the better it will be And of course we want to make sure that the guidances we put out really do reflect our best thinking But we’re actively working on that Of course part of that’s a mandate from 21st Century Cures We were told to do it We’re gonna do it But it also I think very much reflects our deep desire to try to make the entire infrastructure around clinical trials more efficient It’s inefficient, it’s costly, we can certainly do better and I think all of us at FDA are unified in agreeing that we need to be part of the solution That’s why we’re here today So I think I’d encourage these continuing discussions I’d encourage the industry to keep coming to us with proposals to use real-world data elements that are well thought out and if you find us to be inflexible, send us nasty emails and we’ll respond to that (audience laughing) and try to be as flexible as we possibly can But again thanks for the opportunity to discuss these things – Great, thanks very much, Peter and thanks to all of our panelists for some great comments (audience applauding) We do have just a few minutes for questions or comments on these issues of infrastructure development so I’ll try to get in a couple here And then remember whatever ones we don’t get in in this session we do have an open public comment session coming up, please that go ahead – [Carla] Hi, thank you so much I’m Carla Rodriguez Watson from the foundation for the FDA I’m also a scientific director for specifically their IMeds program which, little PR here, is a subset of the Sentinel Network that is there to partner with private industry researchers and academia for public health and safety questions So Dr. Stein I was very happy to hear you say that potentially, maybe we could be thinking about adherence as part of looking at it as a regulatory question I mean yesterday we heard a lot of this discussion, maybe not, maybe, maybe not But there was a lot of discussion about, you know, that we don’t want to be able to look at the drug if we can’t see the efficacy or describe the pharmacological effect of a drug if a patient’s not taking the drug But what if the drug directly affects adherence if it makes you foggy or it makes you nauseous or unable to work then you potentially might not take that drug And in that case, you know, it’s not something, I think Dr. Levin said yesterday we have to take it into consideration and perhaps address it in terms of fixing the randomization that got broken But if it is downstream from your primary exposure of interest which is the drug and it is then in the causal pathway because the drug is causing you, is intolerable, then it does affect the effectiveness and you don’t want to control for it in that way So how do we address that unless maybe as an adverse event? That you would consider – So maybe– – This is not just an issue in real-world evidence – No, and I’ll be brief It’s a great question And just to be clear, I think adherence is very important Sometimes it has a regulatory aspect to it and sometimes it’s really more of a clinical question, letting practitioners know about differences in adherence And particularly what predicts differences in adherence We have a range of patients We know if the patient doesn’t take the drug, it won’t work Straight forward But why might they not take the drug and what are the preferences? Some of them, sometimes it’s very obvious things like intolerance or safety, tolerability or safety issues and sometimes it’s just convenience issues and understanding those things I think in the trials that we expect, we expect to be able

to collect information that reflect on those things If the drug causes nausea or causes whatever adverse event that might lead to this continuation, we should be characterizing that, it’s incredibly important I also think that where adherence impacts outcome that’s very important There have been some nice studies done of oral versus long-acting injectable with very well described outcomes that were relevant and had regulatory implications with regard to labeling So I do think those things can be very important and we by no means would want to ignore them But I think adherence is a broader issue and sometimes trials can be done for adherence just to understand differences and predictors of that that may not have a regulatory implication but has a huge clinical impact which is relevant and an important reason to do the study – Great, than you – Agree, thanks for that distinction – Thank you and I’ll try to get in these last three questions before we move into our public comment phase – Thanks, I’m Beth Belluscio from Pfizer Clinical Development and my comments may be sort of a general one but it touches on something that Peter Stein was just talking about so I thought I’d speak I’m new to this area and so my comment sort of reflects the experience of someone coming in new and hearing a lot of awesome information but honestly feeling a little lost a lot of the time And I think one of the things that could really help is if we try to develop some terms that are really specific to what we’re talking about when we say real-world evidence I know I’ve heard in other venues, people don’t really like that term I’ve also heard people don’t like the term pragmatic trials but I think we still need some language that’s specific enough that we know what we’re talking about So for example, real-world evidence can mean conducting a study in a typical clinical setting, real-world evidence can mean conducting an interventional trial but using external controls, real-world evidence can you mean using a registry to identify patients, but those are very different things And so if you’re not sure what you’re talking about and then you start launching into a discussion of let’s say blinding or randomization but you’re not really sure what you’re talking about yet then it’s hard for everyone to communicate effectively and really come up with some guidances So I think from the standpoint of someone coming in new to this and wanting to really learn and absorb information, I think if the leaders, maybe the Margolis Center is a good place to kick this off, is to really come up with terms that, you know Peter was talking about this is a spectrum and I think that’s true and these just sort of depends whether you’re a lumper or a splitter how you would attack this But I do think it’s important to come up with some terms that then people really know sort of what aspect of real-world evidence we’re talking about and that will help us be sort of focused and ultimately communicate the important ideas – Thanks, thanks for that comment It is a general framework that we’re trying to fill out here Any, I see heads nodding, yeah – I like that I agree and I think that also ties into some of the other comments that we made about just being clearer and clearer about what we’re talking about, the situations in which what we are talking about makes the most sense So I think it’s a really important point – We’ll work on it So far the only term I think we’ve heard, new term I’ve heard today is defensive research but (audience laughing) We’re gonna expand out to others, yeah – [Dr. Peterson] I’m Dr. Peterson head of Cardiovascular Outcome Team at AstraZeneca So we are talking about barriers for bringing new medicines, safe medicines to patients and also at a reasonable cost And we put great faith in new technologies but I doubt that we, with the current technology, could design trials at a lower cost than some of the trials we’ve seen lately But I like the concept Peter Stein brought forward, looking at real-world data as modules that could be plugged into trials because another barrier is the time clinicians have to do clinical research And obviously if we could make it easier for clinicians to participate in clinical trials that would be very beneficial both to patients and to the science So I look forward to becoming discussion where we look critically both at how many objectives we’ll build into our study designs at least after the Pyrotinib Trials but also what kind of modules we’re ready to plug in into our trials to make clinical research more available

to more physicians Thank you – Any comments on that? I mean it does seem like an important point that touches on all of your initial comments about where, as you all said, we’re an early stage of either including real-world elements and randomized clinical trials in a more extensive way or in doing randomization in real-world evidence settings more effectively And obviously that all is gonna go faster if we find efficient ways to do it And you all touched on these issues in one way or another but maybe any follow-up thoughts about where the best opportunities are, either in how we’re building out the data infrastructure for this or in you know particular areas like guidance on the need to only focus on serious adverse events and special questions or cases for adjudication or blinding may not be necessary Any thoughts about how to accelerate this development? – I would just go back to and I think it’s a really important comment and I go back to something that Ellis Unger mentioned earlier which is again that the extent of safety reporting really is proportional to the questions that need to be answered And leader in development, even at the end of, even prior to even approval but certainly post approval, the amount of and the type of information can be, from a safety perspective, can be narrowed But I’d also say that even from the the issue of the primary endpoint, what do we want to make sure is robustly assessed? I mean do we need, do you need 10 endpoints or is one or two endpoints that are really key to answering the question sufficient? I don’t think we push sponsors to add a lot of unnecessary endpoints but very often we see protocols with 10 secondary objectives and 10 exploratory objectives and 10 other things That’s not generated from our interest, from regulatory, I mean we’re interested in it, but – [Dr. Peterson] I needed to plead guilty about that I think we aren’t many stakeholders in that conversation that leads to complex protocols And if we really want to bring more medicines to patients we need not only from industry but also from regulators and academy, strive for simpler study designs And simpler study is not only stripped of protocols but also using new technology certainly And I’m an optimist from that perspective – Perfectly agree and I guess the only comment that I’d make is that what we want to make sure is the most robust are the endpoints that will have regulatory impact I mean that’s really what we want to focus the quality of the data and the believability of the data If you’re 6th secondary endpoint isn’t terribly robust and you’re not asking for it in labeling, you know, that’s for you But for us we want to make sure that you’ve very carefully collected and robustly and rigorously collected the endpoints that really matter to us from a regulatory decision viewpoint – Thank you – Just to stress a couple of the points that were made, it all depends on the objective, the primary objective of the protocol and what’s critical And I would say that even if the primary objective is a lab and you think that’s a beautiful thing you can get from the electronic health record, if it’s your primary endpoint sometimes you might say no we’re gonna bite the bullet and at the beginning and the two-year time point we’re gonna send it to a central lab It doesn’t matter that we could get beautiful data from the EHR because that’s the priority for that study But you might say but everything in between and all the other visits we’re gonna let them go through the EHR and let patients be followed by usual care et cetera And maybe using the new technology will be, ’cause you want adherence, you’ll send out reminders to patients or something else, questionnaires to assess other secondary endpoints, et cetera So these are the kinds of hybrid approaches It all depends on prioritization, what your primary objectives and secondary objectives are and with the new technologies what can be gathered electronically or in a much more efficient way And you know putting that into perspective with the priorities from a regulatory perspective – Lesley any? – No, actually, just I think you’ve summarized that nicely, yeah – Great, thanks, Adrienne? – [Adrienne] So this question might be more for Joanne So there’s been a lot of discussion about different areas where you can focus on in terms of real-world evidence and randomized trials and so from blinding to kind of operations and then data And I worry sometimes about how we can focus on kind of what’s the simplest things to do now versus things that are evolving such as data and technology

I was struck by some of the slides that showed the complexity that people have put together for data systems here and I personally know some of the challenges there So any thoughts in terms of what’s the biggest bang for the buck ’cause you get to see the so-called checks that are written for these things and like what should people focus on? – Oh, it’s tough because I have to say, I’ll say it for this meeting I won’t even talk about internally within our company I really haven’t seen too many proposals for regulatory grade indication seeking studies within real-world data systems I really didn’t see any here We said that the Salford Study was not set up for regulatory purposes certainly not I’ve heard here and I didn’t know but it doesn’t look like it was set up with the FDA concurrence, it was set up for other reasons And so it’s a tough question because of the barriers that I mentioned People are very risk-averse Where I personally think the biggest opportunity is in the situations that I mentioned It may not even be the second indication It might be the sixth or seventh indication that you’re seeking It may not be the highest priority from a patient need perspective but something that’s a little bit of a lower priority from everyone’s perspective where time is a little bit more of a luxury So you can work things out, bring them to the FDA, go back and forth And also where the endpoints are amenable and the interim monitoring and the safety profile of the product is amenable to doing the kind of channeling Ellis’s discussion now where it’s amenable to not doing the intense safety monitoring I think that’s the biggest opportunity for now But even that it’s still a challenge to find the right pilot, the right program, to bring forward – [Adrienne] So it sounds like we need to push something forward so that we can have Bob update his slides (audience laughing) And not necessarily use trials from decades ago – In the next meeting make sure Joanne actually has something – [Adrienne] Okay, all right – I think we need to push things forward I think also we should try to see if there are any lessons we can learn from the post marketing studies or post marketing monitoring that’s done, post marketing requirements that are done which are using more real-world data elements to see if there’s any validation, anything that we can be learning from the studies that are done to get health economic data We heard some of those Is there any validation done in any of those studies? It would be good to kind of start that discussion and that lessons learning together – Yeah, great All right, I want to thank the panelists Thank all of you who participated in this session It was a great way, a great final session to end this meeting on Thank you all very much (audience applauding) – Thank you – And that was our last panel but we are not done As we’ve wrapped up our panel discussions, I want to challenge you all to think about what you’ve heard throughout the morning and yesterday and in terms of important points for FDA to consider as they build out their real-world evidence program As we’ve discussed, there are going to be forthcoming guidance documents from the FDA as well as, as you heard, a lot of interest in developing further activities in this real-world evidence space, making some sort of further strategic investments that the agency related to these topics So this is a chance to inform that process I think the comments we’ve heard so far during this meeting as well as the presentations have been very helpful in that regard but we want to make sure we’re not missing something So we want to open up the floor for questions and comments now If anyone has any please, please go ahead Yes – [Jay] I’m Jay Aram from Pfizer Oh I’m sorry, was I? – No, go ahead – Go ahead – [Jay] Thanks, thank you very much I learned a lot and if I got nothing contribution to the guidance, I definitely learned a lot But I would ask the FDA to consider, I didn’t see representation rare disease I saw a lot of great studies and ideas for cardiovascular and COPD which is wonderful and important I would just encourage involvement of rare disease We need to generate real-world data maybe in a different urgency than diseases like cardiovascular So I would encourage Duke and FDA in future to include rare disease And in the same regard when developing the guidance, I would hate to see a guidance that would just be a baby step I would encourage that it should really make a difference

and maybe by making a difference it’s not to just reflect what we discuss but rather think about the future and how will that impact Because as you know FDA does have an impact, I hope, on how we do clinical research One point also that I mentioned, more than once, is the registry Registries can be of high quality I can assure you that FDA as well as the AMA had provided Pfizer with an indication based in part on a registry data So I would say that some registries, not all, because it’s all voluntary So unless there is quality control, the quality will not be as high But I assure you that in mucormycosis for example, which affects one in a million population, it’s very hard for anybody on the planet to be able to collect data Registry is a lifesaver So we would like to know how far can we take it – Thank you very much Over here – [Steve] Thank you, Steve Piantidosi, Brigham and Women’s Hospital I’d like to pick up on the earlier comment regarding terminology And I know I’m fighting City Hall here but since my FDA colleagues have been good enough to hold in here until the end of the comment period, I thought would pick up on this – Please – [Steve] The term real world is really an awful term and I would appeal to the FDA to either define it or replace it To be very blunt, the term is smarmy And for those of you who don’t know that word, please google it And honestly to somebody like me, who’s done clinical trials for 40 years, it’s a little bit offensive And it’s often weaponized to say, “Oh well you’ve done trials for a long time “but I do real-world trials.” And I always say, “What the hell are those?” So in my very brief remarks, I mentioned a term that I think would apply to a large fraction of what we’re talking about which is point-of-care And I vowed a year ago, when I started working actively on some point-of-care initiatives to never use the term real-world And I’ve said it more in these few seconds than I have in the previous 18 months So this is really an appeal to fix this It not only interferes with the ability for new people to come into the field but when you use bad terminology it damages our ability to think And we have to think very clearly about this field It’s very complex There are controversial things There’s some things that are squarely in the domain of traditional randomized trials and single cohort trials and then there are other things that are new and extremely important as we go forward So an appeal to FDA Please either define it or replace it, thank you – And your positive suggestion on that is point-of-care data, point-of-care studies, point-of-care evidence – [Steve] Point-of-care is a term that I think would pertain in many cases but then there there are also well-defined terms for many of the other data sources and things that we’re about, whether they’re derived or were curated or whatever But let’s try to fall back on things that we all understand rather than using a term that not only is poorly defined but actually I don’t think is defined at all And quite honestly I’ve heard one person recently trying to define it at the cancer meetings and the definition that was offered essentially included everything So, I mean, every source of data, every kind of study and so on We all live in the real world I mean go to Cochrane, I’m sorry I’m preaching a little bit now, but it is public comment right – It absolutely is – [Steve] Go to Cochrane and look at the traditional randomized clinical trials that fall outside of what lots of people mean when they talk about real-world and you’ll see an astronomical amount of evidence about stuff that pertains to things that we do every day and we care about All of that came not from the real world? It’s ridiculous So again for the third time to Jackie and Bob, please define it or replace it Thank you – All right, thank you Thanks – [Steve] All right Steve Grossman I’m a regulatory and policy consultant The backbone of real-world data is claims data,

electronic health records, mortality data and of course interoperability And I guess I have two points One is how do we systematically stiffen that backbone ’cause right now it’s, a mess is charitable And my particular question within that is how do we, I wrote achieve interoperative ability, but what I really mean is how do we enforce interoperability Right now we have a half-dozen vendors It’s not in their economic interest to be interoperable and if this system never gets better, unless that changes, I mean there’s a lot of stuff to stiffen it, that’s where I would start – Great – Thank you very much, Steve – Next – Hi, Elaine Irving from GlaxoSmithKline I just wanted to revisit one of the comments that Lesley made in their sum up really around is there other areas that the FDA could help us identify where there’s a real opportunity for some of these studies So, as we said, not try to think that this is an answer for everything but are there some real niche areas where we can get into more specifics about the disease areas Some of those are very well, we’ve heard diabetes, COPD over the last couple of days These areas for me seem to be primed for this kind of research so can we– – These areas meaning diabetes, COPD because they’re common condition with somewhat heterogeneous populations, where there’s a lot to learn? – [Elaine] Well because we a lot of the challenges with restricted populations in the randomized control trials So I think there’s some very specific questions that probably need to be answered for assets that are moving into those indications and whether we can have some quite more clarification around the types of endpoints and that and so we can get into some of the, so more structure around some of those end point selections And then I think the other piece I just wanted to echo was the fact that we do have these new guidelines now around safety and data collection and minimizing that And I appreciate it a lot within industry is we cause a lot of the complexity because we are risk averse I think what’s been really great today is actually to hear directly from members of the FDA saying that actually we don’t want it, we don’t want to see it because that’s a great message for me to take back Because, while a few individuals in the organizations are starting to work that way, really hearing it firsthand gives great ammunition to force change within the companies as well And I think that for me has been the biggest value of this meeting is really bringing the opportunity to bring stakeholders together And I wonder if there’s a way that actually we can convene some of these meetings around when there has been perhaps a large pragmatic trial like Salford or something conducted so that we can really share the lessons learned ‘Cause of it I think we’re all in a journey together and it would be nice to be able to go forward from this meeting – That’s great, thank you and thanks for all the efforts to share your experiences, is very helpful for this meeting – [Niklas] Hi, Niklas Berglind, AstraZeneca And so first I’d like to chip in in the terminology discussion There’s actually a term that’s been used in Europe I think it’s Rob Hemmings, formerly of the MHRA, who always tried to push this term Instead of real-world data, He uses data generated in clinical practice, DGCP It may be a little bit more narrow than what we think of for real-world data but I think for many of the applications we talked about, I think it’s a lot more descriptive and better than then real-world data, so DGCP – Thank you – So the other thing I just wanted to highlight which I don’t think it has been touched upon so much and that’s one of the challenges that we see when we try to do large cardiovascular outcome trials in these kind of settings is that when you use real-world data whether it’s registry based or whether it is electronic health records, it’s very challenging to do global multi-country studies because the systems are very different We are currently working on registry based RCTs and from what we have found out, we can only really feasibly do this in a small handful of European countries And even though these studies might be a little bit cheaper than regular cardiovascular outcome trial,

they’re still not cheap, right So the fact that we’re getting data that, I mean generalizability to, for example, the U.S. population it’s going to be a challenge from a regulatory perspective So I think just wanted to highlight that as one of the challenges that we have before us – Thank you very much for the comments – [Martin] Hi, Martin Gibson, NorthWest Ehealth I have some sympathy with all of the terminology Is it real-world evidence, real-world data, is it real, surreal, what is it? (audience laughing) I have obviously had quite a long time to think about this And I think what we’re all actually struggling with is how we use up-to-date technology to deliver trials and that’s really all it is, it’s no more complicated than that It’s how do we use the information technology that’s now available which wasn’t available 20 years ago to do trials in a better way, with better data quality, with better safety monitoring There’s nothing new here really, apart from the technology And it’s the application of that that we need to get right – Thank you, go ahead – [Joanne] Hi, I’m Joanne Bonkensty with statistics from AstraZeneca So this is just a plea for our regulators is that to continue to engage with statisticians We’ve heard yesterday about statistical challenges but I think there are a lot of opportunities as well So please engage with us At the end of the day, it’s all about the data and how interpretable can the data be Thanks – Great, thank you I’ll rotate a little bit – [John] My name is John Birch and I’m a member of the Mid-America Angels, an investment group in Kansas City I’m an observer I have a couple quick comments First on what you call it I’ve taken to using the term transactional data Data that is just generated in the course of the various sorts of transactions that occur in healthcare And that differentiates it from experimental data which is essentially what you said a clinical trial is supposed to be So I just wanted to suggest that as a possible term Three other quick comments One, data latency I don’t believe has received quite enough attention here in this meeting and I would strongly suggest it get a lot of attention going forward What I mean by that is where an intervention or an event or of significance occurs at a certain point in time, the outcomes of interest can occur at multiple points of time or indeed over many, many years of time afterwards and I think that’s a weakness that is not addressed in the information infrastructure of healthcare generally So it certainly isn’t gonna be captured in the real-world transactional data And I know that’s something that has come up in your value-based payment group as well and it’s extremely important to do something about that problem in an infrastructural way to allow valid based valid care to go forward So I just wanted to throw that out as one suggestion Secondly, economics There’s been a lot of sort of parenthetic talk about the costs of real-world data, the costs of registries, the cost of this and that, the cost of, Lesley was making the point that high quality really high cost? And Adrienne referred to this as well And I think that that, I would love to see you get an economist up here, someone with really sure– – Besides me, but. (laughs) – [John] Well true, but I mean to really look at the cost because I also think that the costs of the real-world data today is not necessarily what it’s gonna be five years from now as things begin economies of scale, as new efforts, new procedures And so we might be making decisions on direction, directional decisions now that would be irrelevant, I know, in five years if, we could go in the wrong direction And related to the economics of it, I think Joanne was making a really wonderful point before that there are many, many uses of real-world data not just the regulatory uses that we’ve been focused on here And I would infer, and that’s going to help pay for it Whatever the costs are, the many other uses I would particularly focus, and this is not part of your subject here today but on consumer use of real-world data, of their own data There have been many, many attempts to develop personal health records Basically to, in one way or another, to collect a given patient’s data in one location for their own use as well as for many other uses

I think that there’s a huge amount of discussion of patient engagement, the need for better patient engagement has also been here, a lot of talk about the need for better adherence or compliance The fact is that underlying all of that is poor patient health literacy on the part of American consumers I think that there are a lot of ways that that real-world data, we’ll continue to use that term for the moment at least, could serve an educational purpose on a one-by-one basis, on an individual basis for people if we were to collect the data and make it available to them And I think that would address many of the other issues here and it would certainly, I think, help to bring down some of the costs of health care It would move us toward a world not just of defensive research but of preventive care – John, thanks for your comment – Thank you – Thank you very much Jesse – Jesse] Yeah, thanks Jesse Berlin from Johnson & Johnson I was gonna say I’m finally getting the last word in which I never do at home But now I have somebody behind me (audience laughing) – May not quite be the last one – [Jesse] It’s kind of a comment question to FDA and prompted by something Peter said I may take you up on your offer to send you a nasty email The question really is when – Not the only one (audience laughing) – [Jesse] Yeah, join the club We have some very different conversations at the reviewing division level than what I’m hearing in this room from more senior people and we struggle sometimes with where do we go ’cause there’s a real hesitance on the industry side to go above the head of the reviewing division, you know unless we really, really feel like we have to So at what point as we begin to have these more complex discussions, at what point do we involve the more senior people? Early on should we be coming to you guys? Or at the beginning? Should we start at the reviewing division and then when we’re frustrated escalate? And you don’t have to answer that now but (man speaking faintly) – If you are gonna answer it, go to a microphone, please (audience laughing) – [Steve] So, I, more than happy to get nasty emails, that’s part of my remit But what I would add is that and I mentioned this before, we do have real-world evidence subcommittee and Jacqueline really has used that to bring together the divisions, the group of us, it’s Bob and Jacqueline and statisticians and myself and others with interest and/or expertise in real-world evidence and from a variety of perspectives And we get together with the company, we get together with division, we get together with this group and so that’s really a forum for these kinds of discussions And that’s what I would encourage you if you have a real-world evidence and again, sorry, not to comment on terminology, but right now it is the terminology, proposal I think it’s an opportunity to reach out There’s a way to provide that directly to us with background information and that will engage us in a discussion that will be with division And we have a lot of robust discussions internally with division and with the real-world evidence group around these things – Okay, great, thank you – Yeah, thank you And yes please go ahead – [David] Hi, David Vizcaya, from Bayer Epidemiology And I just wanted to make one comment and what’s, at least for me, the main driving force for being engaged this type of research which is the ethical component and in the end improving patient’s life So there’s a wealth of data being generated and I think we have a mandate to use that and to push it forward And I think we cannot forget the ethical component that this type of research has in it and that comes also not in terms of having a mandate to use that data to generate better outcomes for the patients and to generate better medicines But also there are many other sub components associated to this So for example data privacy There is a completely different scenario of data privacy when using secondary data and that’s something that also should be part of the discussion, I think And for next instances we should be incorporating that, those aspects in in these discussions Thank you – Thank you very much for the comments I’d like to thank everyone for the kind of, do you have– – Sorry – Okay one more – [Woman] Yeah my question is what we have seen today and I think we are all facing it is that there is a rising interest in real-world evidence Sorry for using this term again The interest is raising and also the time point

when our stakeholders want to see this data, real-world data, real-world evidence is also at a much earlier time point So my question would be, as a consequence, the time frame when we have to perform this kind of studies is also getting earlier and earlier So what would there be a cut-off or a time point which would be too early maybe when there’s not enough in the box of our safety and efficacy database I’m a little bit puzzled on that question – It’s a good question I’m not sure I’m gonna get the answer now We have focused at this meeting on the post-market context, you know taking pre-market pivotal trials and the like as the foundation but as you heard from Salford, that was done sort of in the pre-market context, release accompanying approval, and I think that’s a good topic for further discussion, is how early and what kinds of questions come at different stages – [Woman] But we have seen Salford Lung Study has also been performed in a brief approval status So as a consequence – Yes, yes – Time is moving on in both directions – That’s right Yeah, well thank you for the comments And again thanks to all of you who have participated for the input both in the panel sessions and in this last open public comment session We are not quite done, Jesse does not get the last word I would like to ask Jacqueline Corrigan-Curay for any final comment she has And then I’ll wrap up – Thanks everyone, I really can’t add to all the wisdom that we heard yesterday and today, so I won’t try and go into that I think Bob started off and reminded us of the regulator’s skepticism and Peter talked about believability but what was really exciting was we heard a lot of innovative and very thoughtful work that’s going into trying to make this a reality and I think we are willing to work with you and figure out what makes sense We’ll figure out what we call it as well Leslie talked about it’s really complex these studies and engaging all the stakeholders And I think that paradigm is not only at the individual study but at how do we build that infrastructure that’s going to make this possible, the data quality, being able to get the data and all the other elements that go into this So we are hoping that this is the start of another conversation We have guidances that we need to get but we are willing to really engage on this and I just want to thank everyone who came, who spoke and everyone who’s listening online thank you – Thank you (audience applauding) – Yeah, just to second Jacqueline’s remarks, this is clearly at, this whole effort is at an evolutionary stage and hopefully through meetings like this with a lot of input and real work from different perspectives on the best ways to move it forward hopefully it can be something of an inflection point as well I would just echo that as FDA has repeatedly made clear during this meeting and all of the associated work and it’s been a lot of work that went into this meeting, they are going to be doing further guidance What you all have contributed is going to have an impact on it and I would say probably going beyond just the guidances as well And just to echo that there are mechanisms for people who have questions and are trying to design those next studies whether we call it real-world evidence or point-of-care studies or just high-value clinical trials, FDA does want to hear from you early about it I think we are hopefully getting there We’ve got some terminology issues among other things to work out and just you know I don’t want people to leave with the concept of defensive research in mind So sort of building on that notion of high-value health care, hopefully meetings like this and the FDA’s further efforts are going to help us get to high-value clinical trials as well There are a few people that I want to thank who made this event possible First that starts with all of the speakers and panelists from both days of the workshop Second I mentioned this before, FDA has put a lot of effort into this and related activities in this whole space of what’s currently called at least real-world evidence So a big thanks to Jacqueline, to Karen Alzarad, Diane Heroine, David Martin, Leonard Sacks, who were very helpful in our putting this together As well as on the outside, Rita Redberg, Adrienne Hernandez, Lesley Curtis,

David Madigan and Barbara Behrer all helped us develop the content and participants and work of this meeting And then finally thanks to the team at Duke Margolis, Greg Daniel, Morgan Romine, Adam Aten, Kara Marcone, Sara Subsiri and Elizabeth Murphy for all of their efforts too So thank you all very much and for everyone safe travels back and have a great weekend, take care (audience applauding)