Video and Image Analytics A Key Part of an Analytics Strategy

good afternoon welcome these webinar brought to you by place is in the business of breakthroughs practicing open innovation to provide custom rd services technology expertise best practices and intellectual property to companies worldwide nine sigma is proud to work with parts to bring you a gallery on our ninth based platform where you can view in directly engage park on any of the technologies present on that site for more information on any of that please visit the park out at a nine sites web site or visit the park website at depth pwr calm so today mike first director of parks video an image in linux lab will be presenting a video and image analytics a key part of an analytics drive and if a few moments i’ll let might take the floor and present all assassinated laser technology but just a few housekeeping notes before we start if you have any questions during the session please at any time submit your questions to the Q&A chat box in your dashboard we will be collecting these questions during the entire webinar and then part will follow up with you directly after this webinar but should we have time at the end of the webinar we may be able to answer those questions live additionally i wish to inform all of you listening live at this webinar is being recorded and the content will be made available after the webinar to view idle edit later date so without further hesitation let me present mike might go ahead oh yeah this is mike thanks for joining today i’m excited to talk about on an area where here at park we’re working every day so i hope you come away with both some some new facts and interested in the possibilities and excited about exploring it further so a couple quick words about park to put this presentation in context parks and innovation partner for companies and government agencies of many types park provides breakthrough innovation as an end-to-end service from problem to prototype we do ideation part can invent together with you focused on your problem producing novel and customized technological solutions for your strategic needs we help articulate a vision and a roadmap we’ve helped clients explore various technical technological options for their business and how to get there from here we also thrive on technology creation and development our core competencies are deep technical expertise we apply ongoing research to the cutting edge to invent in prototype custom solutions and we can also transfer IP and know-how to your team as part of that process such parks in the business of breakthroughs it has been for 45 years it was originally founded as zero park in 1970 when we were showered with to the future that let the many inventions a bunch of them are shown here including the the PC but more gently when combined with the others of late laser printing the graphical user interface networking which enabled distributed data sharing and move the information across people and devices and the approach of social science used in technology design to understand what technologies fit for purpose to engage the user and get the technology right for the job we’ve done this for years with xerox but in 2002 we were incorporated as an independent company which allowed us to do research for a broader set of client base and it’s really broaden the potential areas of impact that we followed many of which are shown on the bottom row of this slide and it’s also about a new business model for innovation yeah now today we work in many domains six are sort of highlighted here at some of our key focus areas they range from new materials and the new to vital and the way that manufacturing gets done to reinventing all the ways these new and existing connected systems communicate with content centric networking which is aimed at making the network more secure flexible scalable to the use of all the data generated by these systems and devices connected in in the world today in various forms of analytics to put it to work to solve today’s toughest problem so today I’m going to focus on the areas that my particular lab focuses on which is in the video analytics space and how we can put that to work to enable breakthroughs so for today here’s my three topics first a little bit about video and image analytics and how it will be a major part of any big data analytics strategy and video & imagery are perfect complements to the more traditional data analytics that people might be thinking

of when they hear big data and images unique attributes and rich content can enable very powerful insights if you know how to extract them next we’ll share some some real projects for real clients and discuss real world challenges that we’ve faced or that you will face as you get into this in doing so and as with any challenge having a good tool box and knowing how to choose and use the right tool for the job is a critical factor in project success so we’ll we’ll spend a little bit time on that and I finally wrap off of how to get started and perhaps I’ll Park can help in that process so big data big data is a broad term used for many used for innovations of all kinds today it’s a broad term used to describe data sets now available that are too big for traditional approaches of data mining big data is transforming the world in all kinds of ways at park we believe potential applications for video and image analytics pan a broad range of industries with data from surveillance cameras to cell phones the range and volume of this content is increasing exponentially and will to dominate many other forms for example in healthcare the quality of care versus the cost of providing it represents an ongoing struggle what is video analytics to provide healthcare delivery while reducing costs by augmenting the reach of human interaction without increasing the cost of labor cameras and patient rooms going to augment the attention of floor nurses letting them know when someone looks uncomfortable zor in need of help instead of a 24 by 7 nurse at home for healthcare video analytic system could monitor patients well-being for a portion of the day do they fall are they eating properly are they taking medicine on schedule how’s their behavior changed over time or perhaps in a municipality concerned with congestion the environment and the well-being of the constituents you can go beyond simple sensors like ground loops and parking sensors that simply indicate the vehicles presence and go beyond surveillance footage that’s gathered and stored and not referred to until after an infant is reported to something more increasingly video will play a role in smart cities by capturing the flow of passengers and pedestrians versus just vehicle dynamics speed limits we’ll just a crowds traffic analysis will be used to eliminate congestion through traffic light optimization and smart parking safety will prove by identifying patterns that lead to accidents computers will scour the video inputs and provide screen recommendations to what a Public Safety Officer should take a detailed look at saving time reducing fatigue and improving the quality of life in a city in other settings like education video cam video analytics can augment what a teacher already sees and can focus on each day and focus on every student throughout today it can provide insights into who’s involved and doing well who’s engaged in the learning activity or perhaps be an early warning system for teachers that are students having trouble losing focus are getting bored these additional behavioural insights can complement the data from assessments and testing that quickly help the teacher personalized tomorrow’s educational content for each student every day finally neat in retail today we derive insights from the point-of-sale transactions grocery store loyalty card tells us what people buy and when but video analytics can capture the customer experience how do people move throughout the store what items do they pick up what do they compare how do they interact with the sample state with sample stations how long do they wait in line video analytics systems combined with transaction data provides a dashboard of valuable insights including comparative data from other store locations enable managers to optimize store design product placement shoppers who wish to be recognized may enjoy personalized experience with recommendations and reminders tailors system so we see video and image video and images as the other big data overall video analytics completes the picture of an individual or business process process and can be instrumental in instrumenting the uninstall table computational analytics tends to utilize data that is structured specific and transaction oriented designed for a purpose with clear definitions video is a great complement it can fit many needs and is almost universal in terms of application space it’s very rich in content a picture is worth a thousand words and each frame and video multiplies that value video could be captured and processed or extract one kind of output today but then in the future new questions could arise in the same video re-examine and

new insect learned it’s also growing a tremendous rate in 2013 it’s reported that high-definition surveillance cameras produced 413 petabytes of data per day that’s the equivalent of 92 million DVDs worth of content each day and it’s expected to more than double to more than 800 petabytes per day in 2017 to build on the retail example shown here if you can fill out many gaps and analytics not just what you purchased or the coupon coupons you used but it can capture what you looked at what you’ve compared where you were you shopping alone or with others and more all data that retailers desperately want so what makes up video and image analytics video and image analytics can be described in several layers of capability that that I’ve shown here building up from fundamental scene analysis people and behavior and out this up to a semantic understanding of people and scenes cameras capture the world to their optics and change the light into ones and zeros those digitalize provide the wrong teria for fundamental layer of video analytics to go to work on these fundamentals includes a recognition of objects events and images and extracting important signals either visible or invisible from the subtle differences frame-to-frame in the videos these end up being the basic building blocks they’re useful on their own in many applications but they find even more use in the next level which we’re calling scene analysis this includes tracking and timing of events and objects so how they interact with each other the sequence of events understanding this variation in timing between them and it’s becomes the heart of many operations focused analytics projects for a process improvement it provides rich data that would be difficult to instrument with unique sensors or tedious or expensive for people to capture manually it also includes identifying something out of the ordinary or an anomaly in many cases it’s these unexpected events that can be much more important than the common ones they’re also much harder to teach a computer to find but are key and the scene analysis layer of video analytics eventually we want to understand more about the scenes and the people and objects within them this can lead to a better understanding of what is happening in a more human way not just what humans are doing in the scene but in a min computers to describe what they see in more human terms leading to better more appropriate action being taken as a result it’s not just a kind of processing that can be done that’s important another option to think about is the way in which the data is captured there are many many options but they larger fit into what i’m calling 3+1 categories of course we’re probably all familiar with the fixed camera systems is the traditional surveillance camera scenario field of view with steady and items people objects move in and out of the scene the background is typically constant beyond this theorem are more and more mobile options that complement the fixed installations wearables like body cameras GoPros the new microsoft hololens which capture a view from the wearer’s perspective and tend to be along for the ride offer a different more intimate perspective on a scene a lot and have their own strengths and weaknesses versus the fixed cameras another option for a gathering data is the mobile platform this essentially a roving set of eyes on the scene cameras that fall into this category are your smartphone or even UAVs or other vehicle mounted systems it can be used to gain unique vantage point a person might not easily be able to get to or a fixed camera may have occlusions that block it from seeing the last important perspective in the capture side of the video analytics is the +1 if you will the it is whether your analytics team will use isolated cameras doing their own analytics separately and unaware of other information or will the system be designed to have cameras work together as part of more comprehensive network getting a bigger perspective on the scene each of these has their own pluses and minuses and as one moves from explore to delivery of a solution it becomes more important to know how each can be used to develop an overall effective results this leads to a set of technical challenges to be to be navigated and thinking about video analytics of algorithms and cameras are just that I just described other starting point but there are some additional items to be considered first is the choice of technology and the system architecture as I mentioned there are many approaches to video and image analytics the area’s hot and many new technical options are being developed

and made available every day the state of the art is advancing daily for example recent computer vision and pattern recognition conference in Boston cvpr 2015 a couple weeks ago was one of the biggest and spanned a week of tutorials posters and presentations from a worldwide set of contributors all sharing reviewed and vetted approaches to teaching computers to see and understand videos and images keeping up with the technology can be a job on its own and is needed to make good design decisions when thinking about the application of the technology another decision to be made is where the analytics will be performed do you do it out near the cameras or on a mobile devices on the edge of the network so that less data needs to be shipped around the system where do you store the data in that case does it make sense to ship everything back to a central location for mining there with more powerful computing platforms next the world is a really messy place developing a system to work in the wild vs. in a laboratory setting can be a challenging activity simply put how do you get performance in the face of all kinds of noises somebody’s includes lighting are their shadows glare from the Sun differences between morning and afternoon reflection stray light seasonal changes and how the scene is lipped or does weather affect it snow fog rain puddles on the street missed or downpours movements in the in the platform itself the camera vibration drift of the field of view over time or contamination and dirt slowly degrading the from the cameras is the pose the pose of the items of interest is a face on side view and how accurately is the current scenario to ones that you’ve seen before no two sites are the same backgrounds configuration and installation variability all contribute to variation that systems need to be robust to think about tracking a vehicle down the street throughout the year and all the variability that we can deal with those humans you’ll see that as a car driving on the street but teaching a computer to do that amongst all those noises it’s not as easy performance and accuracy is also critical for such systems this is particularly true where a more more automation scale and volume of transactions are the norm take a tolling system Ian in a big city for example Manhattan has about 1.6 million commuters statistics show that about a 160 k drive from Brooklyn alone or through Brooklyn on their way into commuting let’s say about a quarter of the overall commuters drive in across any or all the bridges if only a three sigma performance on your automation or ninety three percent accuracy rate that would leave you with 66,000 defects in your reads defects per million or in our case with this scale it would be about 26,000 bad reads per day in order to get to a more workable level for the kind of automation that many clients are looking for the level of performance needs to be well above the 99 plus percent accuracy in order to get there 5 Sigma being about 93 bad read in our cases we have a ninety-nine point nine seven percent accurate a tough that’s a tough tough performance level and way beyond that many of the optus shelf technologies are capable of without special tuning and thought put into how the system will work in the environment you find it one option to consider is to determine with when and how humans collaborate with computers and in the operation of the system and the solution the so called SAR a human in the loop system they said this is a collaborative system using computers for what they’re good at scale speed precision and people for what they’re good at creativity problem formulation intuition action planning cross-domain recognition of objects and and characters humans can guide the attention of the computer to get quantified results and detailed measurements of systems that the computers can identify quicker or humans can assist the automated systems with particularly tough problem sets that the computer flax is requiring human intervention kids the key is to develop systems where the computer screens mostly inputs and only passes along what’s not confident up to the computer for expert review and finally one more category to consider is the real-time nature of the desired solution many interesting concepts using video analytics require results very quickly so the action can be taken in a time

frame suitable for the application as a result needed in the moment as a uterus you interacts with the system in largely real time does it require feedback within a minute in order to send someone over to help in a retail establishment or as an analytics on store operations to review at the end of the day or before the weekend all of these criteria or requirements affect your decisions on your system design so sorting out all these possibilities and options is a challenge we have clients that come to us in all different places in this process we like to begin with a focus deep dive to understand the challenge at hand to identify the opportunity access options and and because if this deal is moving rapidly and others are ruling out video and image analytics it’s always good to to jumpstart the learning process with some benchmarking of other solutions that are available we then understand and follow an iterative approach and do a quick proof of concept to determine and understand the opportunity the needs and what a proposed solution can hit the desired business impact the next steps beyond that are working with the client to demonstrate deliver and and scale this solution we found this approach to help navigate through all the options that we were just talking about effective and a nice structured way to do this so I want to transition now to talking about some examples of some of the innovations that we’ve seen most recently and participated in in the application of image and video analytics to real world problems the examples are going to span from concept to transfer you know through the workshop phase through execution of a point of a proof of concept and beyond in particular place to start is in transportation example transportation areas where we’ve worked with industry from top to bottom for a client transportation video analytics address as many many market needs we’ve learned a lot about the needs of the populace and the municipalities and local governments and transportation system providers have served them many opportunities we’ve discussed with them lend themselves to use of video & imagery as a robust and flexible source of information about what’s happening society in general needs to address several grand challenges congestion environmental impact of human activity or two of the largest they sort of fit into some major categories of efficiency safety and security and predictability all are very important to this domain but also many attributes come with means from other domains as well that will will soon talk about so to be to be specific with a particular example with an eye on improving traffic flow while continuing to use tolling to fund roadway maintenance many operating agencies are looking to go to open road transponder list all video based tolling system in addition all video based tolling enables interoperability interoperability between various tolling agencies because there’s no customized hardware to deploy to vehicles or to infrastructure along the road transponder live bidding talk video tolling can be a great solution but requires new levels of accuracy in the automated license plate recognition system currently people are often employed to read a plate from a captured image if for some reason this transponder doesn’t work the failsafe system this human in the loop system well effective one scale effectively to the volumes needed to be successful if all of the reads were done with a with an image or with video so we worked with a client to enable an all video based tolling at levels of accuracy where the human reviews drops you know 22 or below the current operating levels with transponders as the primary reading mechanism this enabled a kind of scale needed for this application to be broadly employed parts work from concept through delivery in partnership with the client in this case so the challenges some of which are shown here a little bit of hundreds of license plate designs growing every year the u.s. is particularly challenging with its variety and customization of license plates there’s all kinds of environmental conditions some examples of variations are shown over here on the right lighting pose shadows dirt and the accuracy required which mentioned a little bit in the in the tolling example about New York City the level required is is quite high 99% plus yield at high accuracy before

engaging a human to approach things so we took a couple of unique approaches in working through all the choices one of them is incorporating the plate rules there are formulas and ways that certain numbers follow certain letters and patterns which you can use this visual cues to help understand the plate you’re looking at we also used synthetic synthetic training data to feel speed the scaling and adaptation to the new plates from various states machine learning technologies employ a wide variety of learning mechanisms to teach the computer what it seemed and synthetic data allows us to speed that up we source continue to leverage human in the loop to reach the performance levels we want and we designed in feedback as the system operates to improve it over time right now the system is deployed reading millions of place a month and achieving a 99 plus percent accuracy at eighty percent automation levels which makes it one of the leading offerings in this space most video shown here about finding the license plate reading it digesting it and making read is shown here so another a quick topic in transportation congestion and effective managed lanes can improve overall traffic flow condition is a major concern and the economic impacts are significant high occupancy vehicle lanes are put in place to reward those who can transport several people in a vehicle a time however they can go under utilize and age have been converting the HOV Lane high octave eagle as the high occupancy tolling lanes the intent is to allow anyone to drive an h OT line h-o-t lane but at a price with two or more people you drive for free if you buy yourself you pay a toll these are currently operated using a combination of transponder settings indicating the number of passengers currently in the car and employ roadside enforcement these are proven challenging to operate do the difficulty enforcing tolling and the self reporting of rider compliance varies by time of day and traffic conditions and enforcement time consuming in addition drivers are paying a toll based on a agreement that the agency running the road will ensure a certain minimum speed will be maintained in the lane challenging this challenges many agencies to have better data in order to show that they are meeting that minimum speed requirement otherwise they’ll be unable to continue as an HOV Lane the application of video analytics account passengers and enable robust tolling it provide an early warning for enforcement’s become the strategy you can imagine if you’re looking at a vehicle going by at highway speeds many times a day and under many weather conditions choosing the right combination of technologies would be critical so some of the challenges are listed here similar in terms of lighting environment passenger compartment you illumination looking into the to the front of the car what you see here are a set of images of the front view of cars that were captured the screen when windscreen was localized the passenger seat was looked at and set that are characterized as being occupied in assets that are being characterized as unoccupied for example you can see some highlight examples of a person almost entirely occluded by a newspaper but still able to be characterized as occupied and one with a small dog sitting in the passenger seat the system trained well enough to understand that that’s not a qualified writer in the front of the car lift system is currently achieving about ninety six percent accuracy and automatically determining HOV plus to violators and enabling enforcement with minimal review of violations so moving on quickly to a couple other examples in retail we are working with clients again throughout the pipeline but I’ll talk about one that’s more in the proof of concept phase in retail video analytics has many places where it could be effective in that market segment from operational effectiveness in terms of running the business better to helping understand the situation going on in the retail establishment to help associates get the right information at the right time to the guests and in particular the connecting of clicks and bricks perhaps

one of the most important areas with online shopper with online shopping a retailer knows almost everything about you and what you looked at in their store they know what you looked at which put your basket what you move removed how long you spent on each item but contrast that with what they have about your in-store experience you know which purchased and that maybe you use the coupon but that might be about it video analytics is a powerful way to instrument a physical store and understand much more about the guest guest journey and their total experience their behaviors interaction with products interaction with signage and the effectiveness of smart signage video and image content therefore it’s a powerful addition to other sources of information retailer has about their guests and can drive valuable insights from that data leading to action as an example a client was interested in gathering an operational performance data to drive productivity in a quick service restaurant there is of course info from the point of sale system as I mentioned before but that didn’t provide the complete picture since the desired timing info can be gathered by someone in the store visually assessing the operations and taking notes that kind of an operation to great candidate for automated video analytics from the in-store sent cameras so this quick little video that I’m going to going to run here in a second so the good example of the combination of a person reification tracking and timing linked with event detection what you’ll see is that in in step 1 this particular guest has been a tagged as having placed an order I’m step till you’ll see him standing over here on the side waiting for his food to arrive while others pick up their items and move about the environment eventually his item comes up he walks over to the counter to get us food and the event is recognized of him coming up and the event of taking the trays recognized the timing information is then generated so here you can see him standing here eventually becomes his term his tray pops up he recognizes he walks over we have re identified as the same person we’ve seen the event of gathering the tray and been able to capture the time that elapsed between those various events it’s a retail example shown here but to be any kind of queue management or processed timing application could be parking management manufacturing processes safety or security applications this data would then be used in combination with other data from retailers to complete the operational dashboards that they can use to run their establishments better extensions of that include the use of video analyzed to determine the number of operational strategic decisions that need to be pursued every day for example is possible together not only where in the store did individuals go but how long they spent their what they looked at and video and image analytics can provide a lot of the missing information to complete the clicks to brix picture enabling more optimized retail experience one example from from xerox that i will include here real quickly of course we’re delivering technology to Xerox and much the same approach what and success there as well so I’ll show another example that emphasizes the application of video and image analytics on the edge of the network customized for user a smartphone and sort of all that that implies so safe courier is an application designed to automate paperwork transactions in this case for an insurer what it includes is the ability to capture documents photos of the damage combine them together in a way that automatic allows them to be submitted versus putting them in the mail and some kind of hard copy way which was the previous approach as the app is kicked off the user simply holds the phone over the documents to be captured and you can see it’s starting right here we’ll take a look I will submit a new claim and get the process started so going from a very manual process and hard copy mail to make it all work seamlessly on a smartphone so the user would simply hold the phone over the documents the system will do the video analytics to determine when the picture should be taken to get a great position for capture it automatically recognizes the document finds the edges adjust for distortions and prepare to for sending so that it could be used downstream this whole process enables great business process outsourcing back office work because

we’re ensuring that the content going into the process is suitable for work in the back office when it gets there all the other information was captured cleaned up and attached and a complete quality submission can be done in minutes with this app and the user experience is quite simple because of the automation around the video in real time locally on the smartphone in made that process very very simple some quick overview of some exploratory directions that we’re taking in health care and the industrial market segments more at the workshop level although we are working with clients again throughout this space when it comes right down to it they’re basically two major goals of the health care system deliver excellent healthcare outcomes and do it at a reasonable cost and so a video and image analytics shows great promise in enabling new ways of capturing ongoing health status and longitudinal data when a computer can become many additional eyes for the doctors and nurses especially when the content is billed effectively for them it allows them to take action as needed when the time is right now this can be in a clinic in a nursing home or even in a hospital or even better at patient’s home or they can be comfortable living at a lower cost with a higher morale and but still remain very connected and monitored by professionals since cameras and computers can extract information from video that is much more subtle than a human eye and brain can there’s an opportunity to gather richer information in new ways the vital signs can be monitored at any time throughout the day which can provide a more continuous look at a patient as possible during a typical 20 minute office visit that occurs once in a while in addition themes repeat themselves from other domains as well safety and security monitoring marring process steps to assure compliance like safety and washing or surgery prep work flows or all possibilities there are many many possibilities to explore this application video adulation health care and we’re exploring a variety of those in the domains shown here with this various clients and in the industrial segment similar operational safety goals are very aligned with what’s possible now in the future again the areas of people and object tracking and timing for process improvement come up here quite strongly but also ensuring human safety is a big scene the do the dot there’s a desire to put in place systems that monitor a video feed to provide feedback to the individuals involved as well as to the central management site for coaching and feedback can be applied to monitoring operations and to watch out for workers who are alone in isolated areas and it can also apply to health and safety policies other being followed very where is everybody wearing a helmet was there unauthorized access to secure location and all these are very amenable to a video analytics of various kinds so getting started how do you get started in this space of course we feel the best place to start is with the kinds of questions we address in the initial exploratory workshop in particular the definition of the opportunity and clear definition of the business outcomes the business impact should be articulated the rest of the steps follow naturally when supported by the right tool box and team so from our experience we’d recommend a tool boxes the following kinds of content access to and an understanding of the range of video and image analytics algorithms and architectural options that can be explored you need to know what the tools might be to be appropriate and we’re aware and how to look for them and how they work we believe rapid prototyping is very important as each solution can require unique customization and nothing beats seeing a system working early in the development process to know if you are on track this is particularly valuable in identifying those critical unknowns that can derail a project our experience also shows that the steps which follow the proof of concept and the validation of business impact are also very critical this is where you should be working with the development delivery team who understands the process for maturing and technology understanding robust design and the processes for the optimization of the solution and finally experience with deploying video and image analytic solutions to operating groups and all that is required to scale them effectively help sell head off potential problems early on with the right tool box used by internal teams or with external partners the outcomes really could be outstanding parking gauges with many different industries and can help you explore your opportunities for using video and image analytics for your

business and so some of the ones that covered are shown here but we have clientele from across this wide range working through this process with us hopefully of whom this talk you’ve been able to you’ve seen a bit of the toolbox employed in those real examples I showed you how and have ideas about how we might employ it together to navigate the landscape of the options space that could be involved in your video analytics opportunity in your business so just sort of wrapping up contact us let us know how we how we can help again this is our business model we do breakthrough innovation from from problem to prototype some of the links are shown here to reach us but either at par kaam or through the 9th sites gallery 9 sigma so please contact us let us know how we can help thanks for both your time and interest today and best of luck with your future projects so I think we have a you know maybe a minute or two I don’t know if there are any questions lined up now or we made this end here you know what Mike and we have a number of questions I think that it’s probably best to wait until afterwards just because in effort of time I don’t think we’d get through enough of them so well go make sure that we can send all of those over to the park team and you guys can follow up with everybody individually so I just wanted to again reiterate our thanks to the audience for for listening in this afternoon I also want to thank Mike for his time and expertise I’m like I said we’re going to go ahead and wrap up right now and like Mike said we want to encourage you to visit the park gallery on our nine sites website the link is right there and also visit the park website for any additional information you might have and when contact is of course through any of those means as well I just want to make a quick reminder that this webinar was recorded and will be made available for viewing at a later date any questions you have based off of those later viewings feel free again to contact us directly those links that we have up there so we have an additional I will I will leave this up while we let everybody else go well obviously I’m living it up for another five minutes or so just together any additional lesson inquiries we might have for the park team so they can answer them through follow-up after the fact again thank you so much to everybody and have a great afternoon