Social Physics

Social Physics is a revolutionary new science which uses big data analysis and the mathematical laws of biology to understand the behavior of human crowds, enabling Endor to overcome traditional Machine Learning limitations. This new science originated at MIT through research by Prof. Alex “Sandy” Pentland and Dr. Yaniv Altshuler. It was further developed by Endor using proprietary technology, resulting in a powerful engine that is able to explain and predict any sort of human behavior – which by nature changes and evolves rapidly.

Simply put, Social Physics is based on the premise that every event-data representing human activity (e.g. phone call records, credit card purchases, taxi rides, web activity) is guaranteed to contain a set of mathematical patterns that are embedded within that data. These mathematical invariances, which are common to all human data-types, across all demographics, can then serve as a filter for detecting emerging behavioral patterns before they can be observed by any other technique.

A revolutionary concept and a truly technological breakthrough. The results they presented are unmatched by any competing tool

CIO of Israeli Intelligence Corps

Who is going to get a mortgage in the next 6 months?

The marketing department of a large bank is looking to identify customers who are likely to seek a mortgage in the next 6 months, and to start a marketing campaign targeting this group.

They use two tools to predict who these customers are:
A Machine Learning model developed in-house by the bank’s data science team; and Endor’s engine.

Here is a simplified representation of what each tool concluded:

Comparing Social Physics with Machine Learning and Deep Learning

Machine Learning model
A segment of 5,650 customers who:
Are married
Have a household income of >$100k
Have a credit score > 650
Have been customers for at least two years
Have at least one bank credit card

Among these customers, the vast majority of those who will indeed seek a mortgage will also be identified by Endor.

endor logo
Powered by
Social Physics

Accuracy and performance

The customer group that was detected by the Machine Learning model comprised of customers who will indeed respond positively to a marketing offer by the bank (“True Positives”), and those who will not (“False Positives”).

By way of illustration, let’s assume that the True Positives comprise 10% of the model’s results. Extensive experiments reveal that we can expect the vast majority of those 10% to also be detected by Endor’s Social Physics engine, with two main differences: (a) many of the False Positives of the Machine Learning model will not be reported by the Endor engine; (b) the Endor results list will contain many additional True Positives, which are not detected by the traditional model. The implication of this is a significant improvement in sales efforts (thanks to the better precision / recall trade-off).

Example:

In a recent test 15 million Tweets’ meta-data were provided to Endor as raw data for analysis. In addition, the customer revealed the identity of 50 Twitter accounts known to be ISIS activists that were contained in the input data, and tested Endor’s ability to detect an additional 74 accounts that were hidden within the data.

Endor’s engine completed the task on a single laptop in only 24 minutes (measured from the time the raw data was introduced into the system until the final results were available), identifying 80 Twitter accounts as “lookalikes” to the provided example, 45 of which (56%) turned out to be part of the list of the 74 hidden accounts. Importantly, this provided an extremely low false alarm rate (35 False Positive results), so that the customer could easily afford to have human experts investigate the identified targets

Why?

Human reality is composed of many small temporary events and changes.

Endor, grounded in Social Physics, incorporates the underlying dynamics of human behavior and is therefore better equipped to uncover small groups in the population who are likely to behave in a certain way due to recent, unexpected events.

Endor is therefore uniquely capable of identifying dynamic signals in human behavior data that no other method can sense. This is because traditional Machine Learning and Deep Learning methods would not be able to distinguish between these signals and noise. Without Social Physics, these signals lack any sort of statistical significance.

Social Physics successfully demonstrated novel algorithms to predict anomalous large movements of individuals, and provided a novel marketing campaign allocation methodology

Dr. Nuria Oliver,
Scientific Director, Telefonica

Which one is better for which purpose?

In solving a business query using data science and big data analytics tools, both Machine Learning and Social Physics are viable options. In the table below we try to help you identify the appropriate tool, based on its attributes.

Machine Learning is better for mechanical / physical-driven data

For example: monitoring an oil drill pump’s control data to predict malfunction, face recognition.

Social Physics is better forHuman behavior data

For example: analyzing financial transactions to predict who will purchase a premium service

Why? human behavior is erratic, unpredictable, noisy, complex, and dynamic. Mathematically speaking, in contrast to ”static problems” (such as face recognition), human behavior is dominated by a huge number of “temporal” signals, each affecting a small group of individuals. Hence, it is very hard to “learn” human data and produce consistent, stable models representing it.

Endor uses Social Physics to detect such temporal signals, and therefore is specifically tailored to human-based data.

Here are the primary advantages of using Social Physics as a tool for predictive analytics of human data

Traditional Machine Learning

Deep Learning

(without Social

Physics)

Social Physics

Why?

Small data sets

Able to analyze small data sets, but requires expert data scientists and is a time-consuming process

Requires large amounts of data for every question

Requires very little data to answer any question related to human behavior.
The results are generated automatically (no need for data scientists to be involved)

Endor does not require big data to generate results, since Social Physics already incorporates the underlying dynamics of human behavior driven data. Hence, even with very small data sets it can immediately produce accurate predictions and actionable signals.

Features vs. raw data

Requires a skilled data scientist and / or a domain expert in order to define and select the right features representation of the raw data

Does not need features and can process raw data, but is limited to an extremely narrow type of problems

Does not need features and can process raw data, for any type of predictive problem (for human behavior)

Machine Learning requires a long, often manual, process of transforming raw data into meaningful features. This is typically done from scratch for every problem, and for any new type of data.

Although Deep Learning deals with feature crafting automatically, it still requires large amounts of data, and data requirements increase in line with the complexity of the problem. Therefore, it is limited to “simple behaviors.”

In addition, Deep Learning is usually also confined to “static problems,” as Deep Learning dynamics require a vast amount of data that is usually unavailable at the typical company.

Social Physics automatically transforms any raw human behavior data into a canonical form of human behavioral clusters. Using this canonical representation, Endor is able to contend with all data types and all questions, regardless of data size, and to generate a unified human-behavior data set which then uses the power of Deep Learning to answer any predictive question.

Users and expertise needed

Machine Learning experts, usually with the assistance of domain experts, who help craft semantic features

Deep Learning experts

Business users.
All you need to do is provide an example of “people you want to find more of”

Machine Learning requires “learning” the underlying normal behavior of a large data set, or leveraging prior domain expertise. Endor already incorporates the underlying dynamic of human behavior data.

Pace of data change

Limited to slow-changing data.

Changes in the data requires continuous intervention of domain experts, in order to tweak the features

While it can deal with dynamics, it is limited to slow-changing data (a harsh limitation when it comes to human behavior data!)

Can easily analyze fast-changing data sources, and does so automatically (no need for domain experts)

Endor’s engine is specifically tailored to human behavior data, and hence inherently works on data of a dynamic nature.
As Social Physics is a set of mathematical invariances that are embedded in any human dataset, it can even detect signals representing extremely short time segments.
In other words, it is able to identify emerging changes before they are observable by other techniques.

Scope of analysis

Specific / Limited

Broad / Any question regarding human behavior

For Machine Learning, the learning process must be repeated for each dataset and question, since the automatically-selected model features need to be re-learned. Social Physics is based on underlying human behavior principles which are not question-specific.

Data cleaning

Machine Learning is highly susceptible to noises and gaps in the data. Requires a long and expensive data cleaning process

Deep Learning often requires a careful process of transforming the data to a format acceptable by the Deep Learning tool

No data cleaning required

As both Machine Learning / Deep Learning expert use mathematical patterns that are derived from the data in order to deduce rules, extract signals, and produce predictions, they require a delicate process of data cleaning.

Social Physics, on the other hand, uses external patterns – mathematical invariances that are known to be embedded in every human behavioral dataset. This significantly reduces the effect of data noises.

In addition, Social Physics transforms raw data into behavioral clusters, which further reduces the effect of data gaps and noise, as most are automatically filtered out.

Iterations and Tweaking

Each tweak in the data or the definition of the problem requires the joint work of both a domain expert (business user or analyst) and a Machine Learning/Deep Learning expert. Each iteration can take weeks, and a typical project consists of at least 4-6 iterations

Endor’s interface is designed for use  by a single operator – a business user or an analyst that is able to revise and modify the definition of the predictive challenge as they see fit The results will then be adapted automatically to the new definition if necessary

Social Physics transforms any human data into a canonical representation of Social Physics’ Laws violations. Querying this canonical representation is a fast and efficient process that requires only the definition of an “example”. Therefore, no “technical expert” activity is required in order to define new questions or modify existing ones.

Small data sets

Traditional Machine Learning

Able to analyze small data sets, but requires expert data scientists and is a time-consuming process

Deep Learning

(without Social

Physics)

Requires large amounts of data for every question

Social Physics

Requires very little data to answer any question related to human behavior.
The results are generated automatically (no need for data scientists to be involved)

Why?

Endor does not require big data to generate results, since Social Physics already incorporates the underlying dynamics of human behavior driven data. Hence, even with very small data sets it can immediately produce accurate predictions and actionable signals.

Features vs. raw data

Traditional Machine Learning

Requires a skilled data scientist and / or a domain expert in order to define and select the right features representation of the raw data

Deep Learning

(without Social

Physics)

Does not need features and can process raw data, but is limited to an extremely narrow type of problems

Social Physics

Does not need features and can process raw data, for any type of predictive problem (for human behavior)

Why?

Machine Learning requires a long, often manual, process of transforming raw data into meaningful features. This is typically done from scratch for every problem, and for any new type of data.

Although Deep Learning deals with feature crafting automatically, it still requires large amounts of data, and data requirements increase in line with the complexity of the problem. Therefore, it is limited to “simple behaviors.”

In addition, Deep Learning is usually also confined to “static problems,” as Deep Learning dynamics require a vast amount of data that is usually unavailable at the typical company.

Social Physics automatically transforms any raw human behavior data into a canonical form of human behavioral clusters. Using this canonical representation, Endor is able to contend with all data types and all questions, regardless of data size, and to generate a unified human-behavior data set which then uses the power of Deep Learning to answer any predictive question.

Users and expertise needed

Traditional Machine Learning

Machine Learning experts, usually with the assistance of domain experts, who help craft semantic features

Deep Learning

(without Social

Physics)

Deep Learning experts

Social Physics

Business users.
All you need to do is provide an example of “people you want to find more of”

Why?

Machine Learning requires “learning” the underlying normal behavior of a large data set, or leveraging prior domain expertise. Endor already incorporates the underlying dynamic of human behavior data.

Pace of data change

Traditional Machine Learning

Limited to slow-changing data.

Changes in the data requires continuous intervention of domain experts, in order to tweak the features

Deep Learning

(without Social

Physics)

While it can deal with dynamics, it is limited to slow-changing data (a harsh limitation when it comes to human behavior data!)

Social Physics

Can easily analyze fast-changing data sources, and does so automatically (no need for domain experts)

Why?

Endor’s engine is specifically tailored to human behavior data, and hence inherently works on data of a dynamic nature.
As Social Physics is a set of mathematical invariances that are embedded in any human dataset, it can even detect signals representing extremely short time segments.
In other words, it is able to identify emerging changes before they are observable by other techniques.

Scope of analysis

Traditional Machine Learning

Specific / Limited

Deep Learning

(without Social

Physics)

Social Physics

Broad / Any question regarding human behavior

Why?

For Machine Learning, the learning process must be repeated for each dataset and question, since the automatically-selected model features need to be re-learned. Social Physics is based on underlying human behavior principles which are not question-specific.

Data cleaning

Traditional Machine Learning

Machine Learning is highly susceptible to noises and gaps in the data. Requires a long and expensive data cleaning process

Deep Learning

(without Social

Physics)

Deep Learning often requires a careful process of transforming the data to a format acceptable by the Deep Learning tool

Social Physics

No data cleaning required

Why?

As both Machine Learning / Deep Learning expert use mathematical patterns that are derived from the data in order to deduce rules, extract signals, and produce predictions, they require a delicate process of data cleaning.

Social Physics, on the other hand, uses external patterns – mathematical invariances that are known to be embedded in every human behavioral dataset. This significantly reduces the effect of data noises.

In addition, Social Physics transforms raw data into behavioral clusters, which further reduces the effect of data gaps and noise, as most are automatically filtered out.

Iterations and Tweaking

Traditional Machine Learning

Each tweak in the data or the definition of the problem requires the joint work of both a domain expert (business user or analyst) and a Machine Learning/Deep Learning expert. Each iteration can take weeks, and a typical project consists of at least 4-6 iterations

Deep Learning

(without Social

Physics)

Social Physics

Endor’s interface is designed for use  by a single operator – a business user or an analyst that is able to revise and modify the definition of the predictive challenge as they see fit The results will then be adapted automatically to the new definition if necessary

Why?

Social Physics transforms any human data into a canonical representation of Social Physics’ Laws violations. Querying this canonical representation is a fast and efficient process that requires only the definition of an “example”. Therefore, no “technical expert” activity is required in order to define new questions or modify existing ones.

From one of the world’s leading data scientists, a landmark tour of the new scienice of idea flow.
offering revolutionary insights into the mysteries of collective intelligence and social influence.

Security and Privacy in Social Networks brings to the forefront innovative approaches for analyzing and enhancing the security and privacy dimensions in online social networks, and is the first comprehensive attempt dedicated entirely to this field.

How can we create organizations and governments that are cooperative, productive, and creative? These are the questions of social physics, and they are especially important right now, because of global competition, environmental challenges, and government failure.

Social explorers spend enormous amounts of time searching for new people and ideas—but not necessarily the best people or ideas. Instead, they seek to form connections with many different kinds of people and to gain exposure to a broad variety of thinking.

As we engage more with social networking sites, there is always the danger of a “group think” mentality–when people follow a group consensus rather than critically evaluate information.