Smarter Investing Decisions Using Artificial Intelligence
Group Sentiments Drive Stock Prices
Summary — Using the power of Machine Learning (AI) and Natural Language Processing (NLP) to create an openly accessible web portal (tradinginsights.net), which can determine future stock prices based on investor sentiments expressed on social media. The web portal was created from scratch and uses a variety of AI and Python technologies.
In the image above, the Amazon stock is analyzed in real-time. The graph shows the stock prices and the sentiment value across a certain time period. The chart at the bottom of the image shows people’s comments on social media about the stock, which are analyzed by the algorithm. According to the data in the graph and chart, the algorithm predicts that the stock price for Amazon is likely to increase.
Background
Future stock prices are difficult to predict because they do not follow a predictable pattern that can be modeled through a model or equation. Recent advances in artificial intelligence and growth in computing power can unearth new factors, which can determine future price directions more accurately. One such factor is “group sentiment.” Frequently, people buy and sell stocks based on numerous variables: future outlook, economic situation, peer pressure, and their past performances. Investors’ comments of stocks on social media are analyzed in aggregate across time, which provides a hint of where prices are headed. When presenting the data to the user, the algorithm categorizes the “group sentiment” value based on a scale from 1 (very negative) to 4 (very positive).
Natural Language Processing (NLP), which is based on artificial intelligence models such as neural networks, can be effectively used to understand, interpret, and manipulate the semantic (meaning) of comments made on social media platforms. It can also be used to infer the contents of company reports and stock analyst comments.
Let us look at an example: a retail investor goes on his Facebook page and posts this comment.
“Apple is a great company. I like their products.”
For a human being, this statement above is easy to interpret, knowing that the investor feels good about Apple company and may buy or recommend its stock to others. This statement has significance or intelligence and can be used as a positive indicator for a future price for Apple stock.
For a computer, however, the sentence does not mean much unless each word can be tagged as a part of speech. In other words, each word in the text needs to be mapped to proper names or nouns (“Apple,” “products,” and “company”), verbs (“is” and “like”), and references or pronouns (“their”). Once a sentence is completely tagged, it is then necessary to determine the relationship between the words, thus helping grasp the overall meaning behind the statement.
The above comment will be interpreted as follows in NLP.
First, a sentence is divided into segments (comma ‘,’), and each segment is analyzed to find proper nouns (NNP → Apple) or verbs (VBP → like). In essence, NLP uses its database to classify each word in its corresponding figure of speech.
Also, NLP looks at dependencies. For example, it can understand that the company in the sentence below references to Apple.
Once the data is tagged, a Sentiment Analysis Tree can be used to derive further intelligence.
Sentiment Analysis Tree
Sentiment analysis relies on capturing the overall tone of the sentence, which can be either positive or negative. For example, consider the following sentence.
“Manages to delight without much of a story.”
An NLP analysis of this sentence yields an overall positive sentiment despite part of the sentence having somewhat of a negative sentiment.
The predictive power of NLP is used to create an investment portal (tradinginsights.net) through which anybody can find the likely price direction of any stock included in the S&P 500 Index in real-time.
Main Technologies Used
· Python 3.5
· Numpy/Scipy
· Stanford University NLP engine (https://nlp.stanford.edu/)
· Plotly/Dash/Flask framework
· InfluxDB time series database
· Alphavantage stock price db
· Social chat connectivity/parsing json(comments source → stocktwits.com)
· S&P 500 index breakout from wikipedia
· Apache webserver on Centos Linux
· Github for source code mgmt
· Spyder/Anaconda IDE
Architecture of tradinginsights.net
The application uses extensive python code to manage user interaction through dash/plotly framework. Here is the code to populate the sentiment grid from the data frame, which is procured from the influx database.
@app.callback
[Output(component_id=’sentiment_table’, component_property=’data’),
Output(component_id=’sentiment_table’, component_property=’columns’),
Output(component_id=’selected-stock’, component_property=’children’)],
[Input(component_id=’stock-name’, component_property=’value’)]
)
def update_sentiments(input_data):
if input_data == ‘’ or input_data is None
df= pd.DataFrame({‘sentimentValue’:[],’sentiment’:[],’comment’:[]})
else
qm=query_mgr()
queryResult= qm.getSentimentData(365,input_data)
df = queryResult[‘sentimentEvent’]
eventTimes = []
for et in df.index.values:
eventTimes.append(pd.to_datetime(str(et)).strftime(“%b %d %Y %H:%M:%S”))
df[‘Time’]=eventTime
utils= nlpUtils()
prediction = utils.weightedPricePrediction(df,input_data)
df[“sentiment”] = df[“sentiment”].apply(utils.mapSentimentValue)
df = df.reindex(index=df.index[::-1])
data= df.to_dict(orient=’records’)
columns =[]
columns.append({‘name’: “Time”, ‘id’: ‘Time’})
columns.append({‘name’: “Sentiment”, ‘id’: ‘sentiment’})
columns.append({‘name’: “Social Chat”, ‘id’: ‘comment’}) return data,columns,prediction
The following python code updates the influx database after collecting data from the social media platform, analyzing it, and creating a sentiment value.
from influxdb import InfluxDBClient
import json
import pandas
class db_mgr:
def __init__(self):
self.client = InfluxDBClient(host=’localhost’, port=8086)
self.client.switch_database(‘sentiments’)
def getSentimentFieldsString(self,sentimentVal,commentVal):
return “\”fields\”: {\”sentiment\”:”+str(sentimentVal)+ “,\”comment\”:\””+str(commentVal)+”\”}”
def getTagString(self,symbol):
return “\”tags\”: {\”symbol\”:\””+symbol+”\”}”
def getJSONBody(self,symbol,sentimentVal,commentVal):
return “[{\”measurement\”: \”sentimentEvent\”,”+self.getTagString(symbol)+”,”+self.getSentimentFieldsString(sentimentVal,commentVal)+”}]”
def updateDb(self,symbol,sentimentVal,commentVal):
self.client.write_points(json.loads(self.getJSONBody(symbol,sentimentVal,commentVal)))
Highest Sentiment Stocks
The application also provides the capability to get a daily list of the highest stocks, which mostly have positive sentiments in the last 24 hours and are most likely to increase.
Lowest Sentiment Stocks
Similar to the highest sentiment stocks, the application also provides the capability to get a daily list of lowest sentiment stocks, which mostly have negative sentiments in the last 24 hours and are most likely to decrease.
