Sensors, sensors everywhere – Building blocks of the IoT

These days when the Internet of Things (IoT) is mentioned one of the first things that come to mind are sensors. This is actually “unfair” in a few different ways – first, the IoT is more than just sensors and second, if you do talk about that foundational layer you should also include actuators/controllers, which are especially relevant for the Industrial IoT. In this sense, it would be better to talk about transducers – superset of sensors and actuators. It is true that sensors are a major part of the IoT and therefore it is worth spending some time (and text) to talk about these components.Read more..



Source:Geospatial World
Sensors, sensors everywhere – Building blocks of the IoT

These days when the Internet of Things (IoT) is mentioned one of the first things that come to mind are sensors. This is actually “unfair” in a few different ways – first, the IoT is more than just sensors and second, if you do talk about that foundational layer you should also include actuators/controllers, which are especially relevant for the Industrial IoT. In this sense, it would be better to talk about transducers – superset of sensors and actuators. It is true that sensors are a major part of the IoT and therefore it is worth spending some time (and text) to talk about these components.

We all know that sensors are devices that can monitor and measure the physical world and everything in it (including ourselves) – how does this “magic” happen? The basic answer is Physics – physical phenomena happen all the time: electromagnetic radiation (i.e., light, RF, etc.), mechanical and kinematic (i.e., motion, force, etc.), thermal, acoustic (i.e., sound, ultrasonic), chemical reactions, biological and more. These phenomena reflect things and processes that are happening in world around us (and even in us), so that if we measure these physical quantities we learn about what is going on. This is not unlike what we do as humans – we sense and “measure” the world with our senses (our eyes are sophisticated light sensors, our ears are sound sensors, etc.) and we then use our brain to make sense of this data.

Sensors typically convert the raw physical phenomena being measured into electrical current or voltage, because these are easy to “manipulate”. For example, many light sensors (not just visible light) are constructed of an appropriate semiconductor material such that when a specific type of light illuminates it the photons (light particles) are absorbed by the semiconductor and transformed into electrons (and an aptly named photocurrent, that is an electrical current created by the absorbed photons). This electrical current is related in a known way to the amount of light; the electrical current can then be easily manipulated and processed by electrical circuitry (including, for example, analog-to-digital conversion). This processed signal can then be transmitted by another appropriate communication device (using an antenna for wireless transmission, a laser for optical transmission, etc.). Clearly, the sensor itself, that part of the device that transforms a physical phenomenon into an electric current, is just the very front end of a chain that produces a signal that can be transmitted. By the way, actuators work the other way around – typically an electrical signal is sent to the actuator, which causes the end device to produce a physical phenomenon (e.g., a loudspeaker that produces sound related to the electrical signal sent).

So let’s move to a more complex scenario – in many cases, the direct measurement in itself is not very interesting but by proper transformation can tell us something that is interesting. For example, a pyroelectric sensors are based on materials that create an electric voltage when there is a change in temperature around them. However, if you have ever dealt with motion sensors of security purposes then you know that the end objective of a motion sensor is to detect motion. The pyroelectric sensor (used in many security applications) senses the temperature change in the room associated with a (warm) object moving in front of the sensor. So the motion sensor not only does transforms a temperature change into a voltage and then performs processing of this voltage, it also needs one more level of processing to yield an indication whether there is motion in front of it (e.g., it has crossed some temperature change threshold).

Here is another, more complex, example –

Many wearables, specifically smart watches and bracelets, include a heart rate monitor (HRM). One of the most common approaches is to use an optical HRM and it goes beyond just a sensor. In fact, it requires an active element to work – the HRM includes a light emitting diode (LED) that shines light onto the wrist of the wearer, which penetrates the skin and reaches the blood vessels (turns out our skin is not as opaque as it looks). A light sensor (not unlike the one I described earlier) measures the reflected light. Because the blood absorbs more of this light than the surrounding tissue and the amount of blood at any given point changes with the heart rate, the (measured) reflected light is actually modulated by the heart rate. If you are really interested, check out photoplethysmography (after you have practiced saying it a few times; the common acronym is PPG, which is a lot easier to pronounce). This waveform needs to undergo some signal processing before a reliable and robust heart rate is extracted. So now we have the heart rate as a function of time, but really we are interested in things like intensity of physical activity, fatigue, calories burned, etc. In other words, we need to further analyze the heart rate waveform and how it changes and infer (or calculate) these quantities of interest. Quite often, other sensors embedded in the smart watch (like accelerometers, see this for example) are also monitored and fused with the HRM to get more accurate information.

What I am trying to highlight here is the fact the term “sensor” is sometimes used rather loosely and can range from the actual sensing element (measuring light or temperature) to a complete “sensing system” like the optical HRM (that actually senses reflected light but delivers the wearers heart rate, by applying signal processing and analytics). One of the reasons for the broadening of the term “sensor” is directly related to the commoditization of both the basic sensing element as well as the computational resources that can be attached to it. Advances in semiconductor manufacturing technology has driven the price of these components down by orders of magnitude over the last two decades (driven, in turn, by the demand created by smart phones and wearables). So these days, one can buy 3-axis accelerometer chips for significantly less than $1 (quantities of 1000) but on the other end of the spectrum you can get a system on chip (SoC) with multiple sensors and an ARM processor. In fact, in 2015, ST Microelectronics came out with iNEMO advanced inertial module, Invensense came out with Firefly and Bosch Sensortec came out with their sensor hub solutions – all similar in specs, single package (about 3x3x1 mm) SoCs that one would be hard pressed to call simply sensors. Maybe these should be called Smart Sensors (implying integrated compute capabilities) or Sensing Engines (implying more of a system rather than a discrete component); in any case these are some of the new building blocks of the IoT (especially the consumer or Human IoT), which are already being embedded in smart phones, smart devices and wearables.

Now what about the Industrial IoT sensors?

Turns out that here too, sensors as discrete components are also fading away. GE is using an additive technology (similar to 3D printing) called Direct Write. One application GE is developing is to build small devices such as sensors on (machine) parts to measure things like temperature or strain – the sensor is basically “sprayed” onto the part. The notion of printing sensors is not new and is certainly part of the trend of sensor price and size reduction. Admittedly, these printed or “written” sensors are usually the bare sensing elements (without the intelligent part, i.e., the ARM processor is not printed together with the sensor), on the other hand a lot of them can be produced and integrated inexpensively.

Smaller, cheaper, lower power, less obtrusive (quoting Mark Weiser – “The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.”) and more intelligent are the major trends associated with sensors. These also lead to new challenges like security issues, especially when your sensor looks more like a small system.

So where are else are sensors headed?

I think the already emerging frontier is related to the sensors that sense us – wearable sensors coupled with body area networks (BANs). The human individual as an intelligent network node – again, not a new concept (the military has been talking about such ideas as part of its network centric warfare concept for at least 2 decades), but one that can now be taken to new extremes. Here are some examples of the exotic (some already here and some not necessarily very far off…) for you to look up: the Biostamp, the ingestible sensored pill, the ingestible pill camera, smart contact lenses, implantable RFID chip (a little bit creepy for my taste…), to name a few. You can wear them, ingest them and implant them (let us not forget things like pacemakers, which have become incredibly sophisticated and continue to make advances by leveraging some of the same trends mentioned earlier).

Indeed, sensors, sensors everywhere…Read less


Structure, context, semantics – What does it all mean?

It seems that recently, driven by the immense coverage that Big Data is getting, some old terms are getting a lot of airplay and some of these terms are interrelated and interdependent. These terms include structured (vs. unstructured) data, metadata, context and semantics – none of these is new but they are all important because they are related to what the (raw) data means.Read more..


Structure, context, semantics – What does it all mean?

It seems that recently, driven by the immense coverage that Big Data is getting, some old terms are getting a lot of airplay and some of these terms are interrelated and interdependent. These terms include structured (vs. unstructured) data, metadata, context and semantics – none of these is new but they are all important because they are related to what the (raw) data means.

So what is raw data?

Basically, it is data that comes directly from a data source before any processing has been applied. Let’s look at a concrete example – a sensor produces a measurement of a physical quantity like temperature, light, acceleration, etc. Typically, this sensor produces an electrical signal proportional to this quantity. It is then digitized, and we essentially now have a stream of bits that represents the measured quantity over time (e.g., the temperature every second). But if we transmit these bits to some recipient who needs to do something with this data, how will they know these bits are not just random bits but actually mean the temperature at a certain place updated every second? A stream of bits without additional information is not very useful (I apologize to all the Information Theory geeks out there for the gross over simplification).

So we need some data about the (raw) data – metadata. How does this work? At the start of each data packet we add a Header consisting of a known number of bits that would encode the time of transmission, the location where the data was created, the number of raw data points in the data packet, the type of data (e.g., temperature) and any other required information. The recipient of this information can now easily store the data in a spreadsheet or relational database and Voila we have introduced structure in what was originally unstructured. But this is not the end of the story – all we have done was to add some basic information about the data.

So far, we have looked at a single sensor (or data source) without looking at related/nearby sensors, the environment around the sensor, relevant historical data, etc. In other words, we have not looked at the context (although one could argue that a timestamp and a geographical location are part of the context). For example, adverse weather can influence the reliability of the sensor and therefore of the data it produces; if we have multiple sensors of different types in the same area we can apply sensor fusion to improve performance and accuracy. Context is therefore an additional layer that when used with processing and analysis (e.g., sensor fusion), enriches the data.

Some of us may find simple time series data (like temperature) pretty boring and “simple” – what about some more complex data sources such as a video camera. As an aside, in 2014 there were an estimated 210 million video surveillance cameras and a whopping 2 billion smartphones (with implied built in video camera). Video cameras are based on sensor arrays that convert light into an electrical signal – there millions of sensors in the array (hence megapixel), which produce one frame every fraction of a second (e.g., Live in the US implies 30 frames per second). The video camera digitizes the signals, encodes the data (implying massive compression) and formatting the data in packets. Here too, there is some form of header describing things like the resolution (number of pixels), encoding format, GPS coordinates and many more parameters. So we get a video clip with some basic metadata, but we have no idea what the clip is about – OK, you say, we could just look at it and know. Sure, but suppose you would like a machine (i.e., a computer) to automatically extract portions of a clip that are interesting or significant (because sifting through 3 months of a zillion cameras way too labor intensive, not to mention boring and therefore error prone).

This brings us to semantics, the study of meaning, and it is the meaningful that is interesting and significant (a synonym of meaningful). What we need to do therefore is to apply video analytics to the clips to extract some meaningful description of what happens in the clip (see one of my prior posts on the topic of what we can learn from video). Things like a person entering a restricted area, the facial expression of a person looking at a specific ad, etc.

The common thread of this whole discussion is: “What does the data mean?” and extracting meaning from the data is a hierarchical process. Data processing and analytics are successively applied to get higher levels of abstraction – lower level (raw) data is transformed into meaningful information. A model that captures this sort of hierarchy is the DIKW (Data, Information, Knowledge, Wisdom) model and a good and relevant graphic can be found here. As you climb the DIKW pyramid, simple features are transformed into complex symbols (not unlike many processes in the brain) and as you get closer to the summit more sophisticated tools are needed such as knowledge-based systems, which originally grew out of early Artificial Intelligence (AI) efforts.

To those among you with philosophical tendencies you should look at the field of epistemology (i.e., the theory of knowledge), which deals with what knowledge is and how it can be acquired and other related topics. Specifically, Empiricism that emphasizes sensory experience (or sense data) and brings us back to the beginning, namely how we gain knowledge from simple, raw data.Read less


Indoor Positioning – Getting There

It is hard to imagine anyone not knowing what GPS is these days (as an aside GPS, is the US Global Positioning System, which is a subset of the overarching Global Navigation Satellite System, GNSS) – over the past 20 years or so it has become part of our everyday life. Read more..


Indoor Positioning – Getting There

It is hard to imagine anyone not knowing what GPS is these days (as an aside GPS, is the US Global Positioning System, which is a subset of the overarching Global Navigation Satellite System, GNSS) – over the past 20 years or so it has become part of our everyday life. GPS receivers are now included in most Smart Phones, Smart Tablets, consumer cameras, many vehicles, etc. Furthermore, because of this ubiquitous technology it is difficult to get lost in most of the world – when your GPS receiver can “see” a few satellites you (or rather your smart device) will be able to determine your position on the globe with fantastic accuracy. Even when you do not have a good view of the sky (e.g., you are in an “urban canyon” surrounded by view-blocking skyscrapers), in many cases other technologies will be able to compensate – because your smart device usually includes additional sensors (like accelerometers, electronic compasses, gyroscopes) that enable things like dead reckoning. Oh yeah, let us not forget that your smart phone, at least, can also determine location based on signals coming from a number of cellular base stations (but usually not with the same accuracy as GPS).

So this is great when you are outdoors, but the minute you walk into a large building, mall, hospital, etc., your GPS receiver becomes useless. I don’t know about you, but getting lost in a large mall is not an unusual scenario. Indoor navigation (for humans and robots) is an obvious use case, but there are others, such as –

  • Highly accurate positioning – less than 1 meter accuracy. This enables, for example, a retailer to know exactly in front of which item her customer is standing; knowing exactly where a specific, tagged asset is in a large warehouse.
  • Positioning in 3D – knowing whether you are standing or in distress and lying on the floor.
  • Tracking – for example, in shopping centers it may be of interest to understand what paths people take as they explore the place; in times of emergency it could be important to know where people are and where they are heading.

There are already quite a few solutions that assume that you have your smart phone with you at all times and that WiFi and/or Bluetooth are turned on (e.g., check out Wifarer), however that is not always the case (e.g., when you are home on the weekend you may want to be away from your phone…). Additionally, some of these solutions require a lot of infrastructure (e.g., many WiFi access points and associated networking hardware) and complex installation. Ideally, one would like minimal infrastructure and nothing on your person, but with the capability of scaling to locating and tracking of many entities simultaneously in a privacy preserving way (i.e., avoid using cameras) – this is a pretty tall order…as usual, tradeoffs need to be made and to understand them let’s go over some basics.

So how does positioning work?

There are a few ways – the simplest one involves transmitting a pulse (RF, light, ultrasound), which then gets reflected; this reflected pulse is picked up by a receiver and the time between transmission and reception is measured. Since the pulse speed is known (speed of light for RF and light, speed of sound in air for ultrasonic) the distance can be calculated. Seems simple…but…Let’s look at Radar (and for the moment oversimplify): if the transmission is in all directions (omnidirectional antenna) then your reflected pulse can come from anywhere on a circle (all points on a circle are the same distance from its center, where the radar is located) – we need to do something else to determine direction. This can be achieved by making the antenna directional – a narrow beam is pointed at the direction of interest or the beam is “swept” across a section of interest. Here is another complication – almost everything reflects the radar pulse and most of it is not interesting; how do you track entities of interest? In most cases, a lot of what you are looking at is stationary and this “background” can be accounted for; alternatively moving objects “change” the pulse (check out the Doppler effect if you are really interested) and identifying this change allows one to immediately identify the moving things. So not so simple after all…

Note that if the object you are locating and tracking happens to have a device (with some unique identifier) attached to it, like some sort of tag or Smart Phone, things become significantly easier. Now you can have many fixed transmitters sending out pulses, getting received by the device that can then send out a “reply” (rather than the reflected pulse) that can also contain its unique identifier. The transmitters can be simple and omnidirectional, but then you need a few of them (remember each one defines a circle; in the plane, i.e., in 2D, at least 3 transmitters are needed to determine a unique position) – the determination of a location from measuring distances to a few fixed points is known as Trilateration (check out Multilateration while you’re at it). This still requires some calculations that need to be performed somewhere – basically solving for the intersection point of multiple circles. Additionally, in many cases, the object is moving and we would like to track it accurately and in real time.

There are additional methods for real time positioning specifically for indoor environments, but this is not meant to be a comprehensive survey (Wikipedia’s IPS entry is a pretty good starting point) – suffice it to say that it is non-trivial. What else do we need? We know from location-based applications and services for outdoor solutions that usually we want our location to be displayed on a map of some sort. Similarly, when we determine the position of an entity indoors we would like to see it on some sort of schematic or (CAD) drawing – this is also more involved than outdoor positioning where high quality maps are readily available. Many new buildings may have used sophisticated 3D architectural tools when designing the building and therefore good accurate 3D models of that building are available, that is not the case with older buildings, which, in turn, may imply creating new accurate drawings (based on accurate measurements). These drawings, just like maps, need to be kept up to date.

The technology and many of the standards are pretty much there, but the ultimate indoor positioning solution that is inexpensive, has high accuracy, has good privacy protection, etc., is still not quite there.

At the end of the day, where we are in space and time is tightly coupled with what we are doing and also provides some of our context. Outdoor location is part of the story (when did we leave home, where did we go, etc.) but many of us spend a lot of their time indoors – at home, in the office, in shopping malls, in restaurants, etc. – providing much more information on our activities, behavior, preferences, and more. Whether we want to share this data and let others use it, is a different story…

I am quite sure that indoor positioning systems will become ubiquitous (just as GPS is a given today), enabling an ever-growing number of services to make our lives easier (as well as getting much more targeted advertisements…sigh). Read less


The Terminology Zoo – IoT and all of its cousins

A few weeks ago I attended the IEEE Pervasive Computing 2015 and one of the interesting panel discussions revolved around the difference between Pervasive Computing and Cyber-Physical Systems (CPS).Read more..


The Terminology Zoo – IoT and all of its cousins

A few weeks ago I attended the IEEE Pervasive Computing 2015 and one of the interesting panel discussions revolved around the difference between Pervasive Computing and Cyber-Physical Systems (CPS). Clearly, there is overlap but there was no agreement on the extent of it and when someone in the audience asked about where Ubiquitous Computing stood (given that there is another annual conference called UbiComp, not an unreasonable question), some people started making annoyed noises and rolling there eyes…

The discussion got me to thinking that there is actually a long list of terms and associated disciplines related to IoT with varying degrees of overlap. Here is a non-exhaustive list:

  • Internet of Things (IoT)
  • Internet of Everything (IoE)
  • Industrial Internet
  • Industrial IoT (IIoT)
  • Pervasive Computing (or Sensing)
  • Ubiquitous Computing (or Sensing)
  • Cyber-Physical Systems (CPS)
  • Wireless Sensor Networks (WSN)
  • Smart Objects
  • Cooperating Objects
  • Machine-to-Machine (M2M)
  • Ambient Intelligence (AmI)
  • Operational Technology (OT) / Information Technology (IT) convergence

Many of these have associated standards bodies and industry consortia such as oneM2M, IEEE IoT related standards, IPSO, AllSeen and MANY others. Obviously, there has been an enormous amount of research in these areas in many dedicated peer-reviewed journals and presented in a large variety of conferences, sponsored by a correspondingly large variety of governmental and industrial entities (such as the NSF, the EU FP7 funded IERC and many others).

Even for the specialist, this terminology zoo is daunting and any definition you put out there is bound to be challenged and argued about, so I will refrain from offering any definition of the above items (Wikipedia is always a good starting point). What I will try to do is to point out some of the key attributes/dimensions where these terms differ (and do my best to avoid landmines).

Let me start with what most people agree on –

All of the above involve smart and connected devices, which implies

  • Sensing and actuating (or control)
  • Communication (hence connected) – with each other and/or with a hub or aggregation point
  • Computation (hence smart) – on the device (or close by) ranging from simple signal processing to sophisticated analytics (“embedded systems” is a term that comes up in this context)

These are the basic building blocks, but since in many applications a direct connection to a power “outlet” is not available to support the sensing/actuating, communicating and computing, a power source (and associated power management) becomes the fourth building block.

Beyond this basic set, there is a hierarchy of systems that extract “meaning” from the raw collected data and/or trigger actions by actuators, controllers, etc. This achieved by processing and analyzing the data in a variety of ways known collectively as Analytics (an incredibly overloaded term that could mean anything from fancy dashboards with some relevant numbers to machine learning based predictive capabilities).

So now that we have the basics, let me point out a few ways to slice and dice the terminology zoo and hopefully offer some insights.

First, some of the terms are actually “subsets” or enablers of others – for example, M2M deals primarily with communication and is an enabler and “part” of IoT, since IoT requires machine-to-machine communications; should you want a lengthier discussion and additional perspectives on this topic take a look at this Quora thread. In general, it is important to remember that IoT is a technology stack that includes devices, communication, processing and analytics and even IoT Applications.

Second, some of these terms are really synonymous – pervasive and ubiquitous are synonyms, so pervasive sensing/computing is pretty much the same ubiquitous sensing/computing. If you look at the programs of Pervasive Computing and Ubicomp, you will find a high degree of similarity.

Third, some of these are associated with certain domains – CPS has a lot of emphasis on control (or actuation) and therefore targeting the industrial world (e.g., manufacturing), which requires not only sensing but also controlling machinery, for example. Clearly, CPS is a large part of the Industrial Internet.

Finally, some of these represent the extension or evolution of earlier terms – the Internet of Everything (IoE) is a superset of IoT as it adds Data, People and Processes to the original Things. IoT was coined before the Smart Phones/Tablets/Devices (as we know them today) existed; the Things were initially associated with RFID tags and later with fixed sensors. Adding Smart Mobile Devices – chock full of sensors, always connected, with more compute power than all of NASA in 1969, and attached to a human that can add context and meaning to the captured data – has literally added new dimensions to IoT.

Coming back to that panel discussion at IEEE PerCom 2015, one of the perspectives I really liked was the following –

CPS is all about Control, Automation, Machinery, Systems, the Human as an Operator and very tight real time requirements

Pervasive Computing is all about Mobile, Cognition and Semantics, Recognition (e.g., of behaviors and activities) and the Human as a User

So CPS is closely related to industrial applications and hence to the Industrial Internet, while Pervasive Computing is more people centric and therefore closely related to personal health and wellness, elderly care, etc. Clearly, there are applications where the two disciplines meet like Smart Cities, where you have both traffic control and energy management, on the one hand, as well as crowdsourcing (e.g., Waze) and augmented reality, on the other.

This distinction reminded me of another interesting partitioning of the IoT (somewhat reminiscent of the above) offered by a research paper from Moor Insights and Strategy – highlighting the Industrial IoT (IIoT) vs. the Human IoT (HIoT). Quoting from this paper:

“This division between systems that operate continuously by a set of rules and systems that interact dynamically with humans creates a useful distinction between autonomous “Industrial” systems and reactive “Human” systems.”

There are many more worthwhile insights in this and associated papers from Moor. Note that others have also pointed out this separation, for example, the Economist in a blog post that (amusingly) was titled “The Internet of Nothings”.

A final thought –

Trying to lump something as complex as the systems (and systems of systems) mentioned here into a single term – IoT, IoE, Smart & Connected Objects, etc. – is not very useful from a practical point of view, in my opinion, for two reasons: 1. It encompasses too much – everything that connects to the physical world is IoT. 2. It is a source of confusion for the end user – when a company says it is in the IoT business, is it sensors, networking, embedded systems, analytics…

Obviously, the above-mentioned zoo of terms is too large and a single term is not enough – it would be nice if the industry could agree on maybe a very small number of terms…one can always dream…Read less


Edge Analytics – a necessary ingredient in the Internet of Things (IoT)

Depending on whose market research you believe and how you count, in a few years there will be tens of billions of smart and connected devices (“Things”), Read more..

Edge Analytics – a necessary ingredient in the Internet of Things (IoT)

Depending on whose market research you believe and how you count, in a few years there will be tens of billions of smart and connected devices (“Things”), maybe more. The expectation is that it will not stop there and the growth will continue fueled by the appetite for expanding the IoT – after all, the migration to IPv6 will increase the number of available IP addresses to a whopping  (enough to address every single grain of sand on Earth and still have a LOT left over). These devices contain sensors and/or actuators, some computational capabilities, a power source and a connection (direct or indirect) to the Internet; a smart and connected device can therefore include everything from a real nifty thermostat to a smart phone and even a car (if it is connected, which is becoming more prevalent).

Until very recently, the focus was on making the devices, working out how they connect to the Internet, collecting the data they produce and making this data available for someone to use – a sort of “if you build it (the IoT) they will come”. While collecting the data and transporting it somewhere (for example, “The Cloud”) is a basic requirement, the real value comes from analyzing the data to gain some understanding as to what it all means. A stream of temperature measurements from a sensor in a house or on the street, has limited usefulness – we need to process and analyze this data so that we can decide if we need to turn on the air conditioning, for example. Hence, the IoT requires analytics and so the emerging zoo of analytics that has emerged from the Big Data market – predictive analytics, streaming analytics, real time analytics, etc. – are now being adopted by the IoT industry. Indeed the IoT is producing Bigger and Bigger Data covering the same 3 V’s – Volume, Variety and Velocity – as all other forms of Big Data.

So, we have a LOT of smart devices collecting a LOT of data and therefore requiring a LOT of analytics (and corresponding compute, storage and networking infrastructure). The traditional, centralized approach where all the data comes to a relatively small number of data centers where the data gets analyzed (“Big Data Analytics in the Cloud), will simply not scale and will also be very costly (e.g., transporting bits from here to there actually costs money). What is the alternative? Distribute the analytics! Consider, the smart devices have compute capabilities built into them (that’s why they’re smart); in fact, compute hardware is cheap – for literally a few dollars you can get significant computational power, which, in turn, can run analytics on the device itself. Furthermore, multiple devices are usually connected to a local hub/router/gateway where potentially more compute power is available (like Cisco’s IOx), enabling more complex multi-device analytics. Analytics are therefore running on or near the edge (of the network) and delivering Distributed Intelligence.

We can now think of a hierarchy of analytics corresponding to the available compute capabilities – “simple” analytics on the smart device itself, more complex multi-device analytics on the IoT gateways and finally the heavy lifting, the Big Data Analytics, running in the cloud. This distribution of analytics, offloads the network and the data centers creating a model that scales. Related terms have come up recently, such as Fog Computing and Cloudlets (related to mobile computing), also introducing the notion of a hierarchy of computing from the edge to the Cloud.

Edge Analytics is not only about creating a distributed, scalable model for IoT analytics – it has additional benefits:

  • Many business processes do not require “heavy duty” analytics and therefore the data collected, processed and analyzed by Edge Analytics on or near the edge can drive (automatic) “decisions” without ever needing to travel to a remote data center. For example, a local valve can be turned off when Edge Analytics have indicated a leak. This is a sort of “closing the circle” on the edge.
  • Some actions need to be taken in real time (i.e., they can not tolerate any latency between the sensor-registered event and the reaction to that event) – a situation that arises in industrial control systems – so there is literally no time to transmit the data to a remote Cloud.
  • Some sensors create a LOT of data – video being the primary one. Even when compressed, a typical video camera can produce a few megabits of data every second. Transporting these video streams requires bandwidth and bandwidth costs money and, if in addition, you want some quality of service guarantees it becomes even more expensive. Thus, performing video analytics on the edge and transporting only the “results” is much cheaper. If some of the video needs to be stored, depending on the volume, it could be stored near the edge or in a local (on premise) data center.

Since I am a firm believer in “No Free Lunch”, there has to be some price to pay…

So here is what we are trading off –

Ideally, you would want ALL of the data to process and analyze, as there may be relationships, patterns and correlations that could only be seen when using ALL of the data. Edge Analytics is all about processing and analyzing subsets of all the data collected and then only transmitting the results – we are essentially discarding some of the raw data and potentially missing some insights. The question is can we live with this “loss” and if so how should we choose which pieces we are willing to “discard”.

Before answering this question, let me point out that this topic is related to Big Data in general – given that sense making cannot keep up with the amount of data collected (the so called Big Data Gap), should we collect/store/analyze EVERYTHING? Or maybe we should be prudent and deal with Smart and Relevant Data. Edge Analytics is one way of making decisions about what we keep and analyze, but here too, the question is how do we decide what needs to be kept and analyzed? What is Relevant?

The answer to this question is highly domain dependent – some organizations (e.g., militaries) may not be willing to lose ANY data and for them Relevant Data is ALL data. This leads me to conclude that domain expertise is required to make to decision what can be handled at the edge with the associated loss of underlying data. Domain experts will be those who know a priori that some pieces of data from different locations are unlikely to be related and therefore can be dealt with locally and possibly acted on locally. Sure, you can always argue that there may be new, unknown, surprising correlations that even domain experts are unaware of and, yes, we may lose these insights. Hence, no free lunch

I should also mention that distributed systems have a long and rich history and we should keep in mind some of the lessons learned when dealing with such systems. For example, when many “islands” are analyzing and acting on the edge, it may be important to have somewhere a single “up-to-date view”, which in turn, may impose various constraints (for those interested in some of the more technical details and not faint of heart, see the CAP Theorem). The fact that many of the edge devices are also mobile complicates the situation even more. After all, most of us carry an edge analytics device on us (smart phone).

Here is the upshot of this discussion –

If you believe that the IoT will expand into every nook and cranny of our lives, where every conceivable object we interact with will become smart and connected, in the home, at work, in industry, indoors and outdoors, then distributing the analytics and the intelligence is inevitable. That’s a good thing – it will help us in dealing with Big Data and releasing bottlenecks in the networks and in the data centers. However, it will require new tools when developing analytic-rich IoT applications – what is the best way of allocating the analytics between the edge and the cloud? How do we keep data consistent? How do we handle mobile edge analytic devices traveling in a highly dynamic environment?

Some domains already employ solutions that are essentially doing some analytics on the edge (e.g., adaptive traffic signal control), but not on the massive scale that will result from the rapid expansion of the IoT. While some people may associate the term “Distributed Intelligence” with apocalyptic visions a la Skynet in “The Terminator” movies, I believe that some real cool and exciting stuff is ahead of us and (excuse the cliché) it will change our lives for the better. Read less


Visual Data – still hard to analyze

Here is an interesting observation: Ask a child to describe what she sees around her and she will immediately tell you something like “I see a tall man talking to a woman in the driveway in front of a yellow house”. Read more..

Visual Data – still hard to analyze

Here is an interesting observation: Ask a child to describe what she sees around her and she will immediately tell you something like “I see a tall man talking to a woman in the driveway in front of a yellow house”. The same task is beyond current computer technology – specifically, feeding a “raw” video clip to a machine and getting back (reasonably quickly) a short textual description of what happens in the clip, is currently pretty much impossible. Images and video are rich sources of information consisting of many different objects (with different shapes and colors) with some relationship to each other, in some environment, possibly moving (in the case of video), etc. – there is a reason that a picture is worth a thousand words. Analyzing images and video to facilitate automatic insights and associated decisions is still incredibly difficult (even offline; doing it in real time is much harder). A further complication is the fact that most of the visual content we view is actually a 2D projection of the real (3D) world. Remarkably, humans are really good at these types of tasks, so one approach could be “Hey, let’s just copy the human visual system” – we’ll get back to this later.

So, what can we do in the area of Video Analytics or Video Content Analysis? – Actually, quite a bit (but nothing like you may have seen in some popular movies) and here are some examples (certainly not an exhaustive list):

  • Driven by security and surveillance use cases, many “suspicious” behaviors can be recognized automatically (i.e., with no human in the loop) such as an object that has been left behind, someone crossing a virtual line, people counting, loitering and many others (but probably no more than about 20). Similarly, in the vehicular traffic area, behaviors such as stopped vehicle, someone driving on the shoulder, etc., can be identified.
  • Some very specific objects can be recognized – faces, vehicles, license plates and probably a few more. Although some only under limited conditions – controlled lighting, controlled pose, minimal occlusions, etc.
  • Tracking of specific objects in the camera’s field of view (tracking across multiple cameras, even when there is overlap in successive cameras, is very difficult)

If your interest is in some specific items on this limited list – no problem, you can buy them from numerous vendors. However, if you are looking for a different behavior or a different object, you will need some computer vision people to develop a new analytic – the generic object recognizer or the generic “tell me if anything unusual happens in this area” do not exist yet.

But don’t despair – Machine Learning approaches are starting to appear in some commercial products. Basically, the machine is trained, for example, on video that represents normal vehicular traffic flow and once the learning phase is over, the machine can indicate that something abnormal has happened such as traffic slowdown due to some sort of incident further down the road. When I say “machine”, by the way, I mean the computer that ingests the video stream and runs the anomaly detection algorithm (which could, in principle, run in the camera itself or very near to it (see my previous post on Edge Analytics).

At this point, you are probably saying, “so what about copying the human visual system?” Well, it turns out the HVS is quite complex and we have not figured out how all of it works yet. A lot of progress has been made over the years and a lot of good research is going on (for example, work at MIT, Penn State and others). One of the exciting developments related to this area are associated with Deep Learning (which really deserves it’s own post…), which is a Machine Learning approach that does really well with tasks where humans are usually better than machines – for example, object recognition in images. DL usually requires a lot of computational resources, which is slowly evaporating as a real hurdle and as a result there have been some really exciting results! Google has recently shown an image with a caption that was caption that was created automatically – this is actually getting us closer to the target of automatic video summarization at the beginning of this post. Another one comes from Microsoft where they managed to outdo humans in an image classification task.

As an aside, some of the world’s top academics in the area of Deep Learning have joined Google, Facebook and Baidu in the last year or two – that should tell you something.

Let me also make the following point – humans are equipped with visual hardware (eyes) that can see in 3D. A lot of stuff gets easier when you also have depth information (e.g., which of two visible objects is in front and which is in back) and there actually are cameras that can record depth information – from stereoscopic cameras to Kinect-like sensors to time-of-flight cameras. In fact, if you have a number of cameras covering the same area from different vantage points you can do on the fly 3D reconstruction (check out FreeD for some real cool clips). Now you can run some advanced video analytics on real time 3D streams to do things that were simply not possible before (or required ridiculous amounts of computational power).

A final thought –

The estimated number of surveillance cameras in the world is about 210 million (obviously, not counting consumer cameras, smart phone cameras, etc.) producing an obscene amount stored video, most of which has never been viewed by anyone and most likely will never be viewed by anyone – there is just too much of it. Only advanced video analytics will be able “to watch the video for us”, letting us know when there is something interesting there.  Read less