Twitter Streaming Importer

2017-01-31

last modified: 2023-04-10

== !

gephi logo 2010 transparent

 

1. Description of the plugin

This plugin is created by Matthieu Totet, who is a long time community member of Gephi (check his Gephi Streamer!)

You can find additional documentation on this plugin on Matthieu’s website.

This plugin enables you to:

  • Collect tweets in realtime, on the topic you choose

  • Get the connections between the users mentionned in these tweets

  • Visualize these connections in Gephi, or just export all the tweets to Excel

2. Installing the plugin

The setup of the plugin will need you to create a Twitter account, if you don’t already have one. Get your Twitter login and password ready. You also need a mobile phone number where your Twitter account can be confirmed.

Setting up a Twitter account and creating a Twitter app is a boring task. It takes just 15 minutes, and you have to do it just once. Then you’ll have Twitter at your fingertips.

Let’s go!

Choose the menu Tools then Plugins
Figure 1. Choose the menu Tools then Plugins

 

Click on the tab Available Plugins
Figure 2. Click on the tab Available Plugins

 

Install the plugin then restart Gephi
Figure 3. Install the plugin then restart Gephi

 

3. Initializing the plugin (to be done just once)

a. Creating a Twitter account

Open a new project in Gephi
Figure 4. Open a new project in Gephi

 

Display the plugin in Gephi
Figure 5. Display the plugin in Gephi

 

We need now to connect to a Twitter account. If you already have a Twitter account and you are logged in, skip to here.

Creating a Twitter account
Figure 6. Creating a Twitter account

 

Confirmation of the sign up
Figure 7. Confirmation of the sign-up

 

You have received an email by Twitter about the sign up. At some point (but now would be a good time), you must click on the link in this email to confirm it.

Enter the verification code you received by sms
Figure 8. Enter the verification code you received by sms

 

Choose a username
Figure 9. Choose a username

 

The next screens of the sign up for Twitter are not important for us. Just follow the steps until you get to the page showing your Twitter profile, meaning the setup is finished.

You now have a Twitter account. We need to create a Twitter app which will authorize us to collect tweets automatically via Gephi

b. Creating a Twitter app

Go to https://developer.twitter.com/en/apps:

Create a new app
Figure 10. Create a new app

 

Fill in the fields
Figure 11. Fill in the fields

 

INFO: Almost there!!

Click on Keys and Access Tokens
Figure 12. Click on Keys and Access Tokens

 

Click on Create my access token
Figure 13. Click on Create my access token

 

Copy paste these codes into Gephi
Figure 14. Copy paste these codes into Gephi

 

Where to copy paste the codes in Gephi
Figure 15. Where to copy paste the codes in Gephi

 

Done!

Well done. This setup had just to be done once. Your credentials are saved and will be loaded at each run. Now we can start using the plugin.

3. Using the plugin

With the Words to follow tab, you will be able to follow one or multiple words. For hashtags , just enter the word without the hash in front of it. (e.g if you want to follow '#Gephi' just add 'Gephi'). You can enter multiple words by separating them with a comma , . E.g word1, word2 .

With the Users to follow tab, you will be able to follow the activity of one or multiple users. Any tweet from this user or retweeting or mentioning the user will be captured. You can also import all users from a twitter list by giviing the user name of the twitter account and the list name to import. You can enter multiple users by separating them with a comma , . E.g totetmatt, gephi .

twitter user list
Figure 16. How to add user from a Twitter List

 

With the Locations to follow tab, you will be able to follow the activity of one or multiple locations. Any tweet that has been geotagged will be captured. You need to delimit the zone to track by its South West Point and the Nord East Point.

geo params
Figure 17. How to add a Locations

 

You can combine the 2 tabs, the tweet collected will be matching either the Words to follow query or the User to follow query

The Load Query File and Save Query File buttons are here to save your search queries in a file or to load it from a file. It’s convenient if you have a long list of words / users.

a. Network Logic

A Network Logic means: what should be done with an incoming tweet? How to transform it as a set of nodes and edges? It’s equivalent to map projection in the cartography world.

Selection of the network logic
Figure 18. Selection of the network logic

 

There are for the moment 4 Network Logics to choose from:

  • Full Twitter Network : This will represent all entities (User, Tweet, Hastags, URL, Media, Symbol etc…​) as a graph.

  • User Network : This will represent the interaction between users. Any mentions, retweets or quotes between 2 users, will be represented. The size of the edge represent the number of interactions between 2 users.

  • Hashtag Network : This will create the network of Hashtags.

  • Emoji Network : Same as Hashtag Network but focused on Emoji characters ( original idea from Neel Shivdasani / @DataNeel ).

  • Bernardamus Projection (based on https://twitter.com/Bernardamus/status/1131334028043411456) : This network represents user network via hashtag present in tweets.

Note for the Emoji Network : By default, the Label of Emoji nodes use the UTF-8 character representation. On the Overview and the Previsualisation panel, the lable display might not work with the default settings (you will see some square instead). To make it work, you need to use a font that support emoji. The Segoe Emoji font should be the one available by default on Win 10.

For the moment, you won’t be able to export the graph as SVG if you keep the UTF-8 character (PNG still works). But you can use the other emoji representation (html or alias) as label to export as SVG and apply post processing on it.

emoji change font
Figure 19. Change font for Emoji display

 

In the following, we use the network logic "User Network":

Adding terms and launching the collection of tweets
Figure 20. Adding terms and launching the collection of tweets

 

Be careful that if you choose very common terms, tweets will arrive fast and in large volumes.

If you don’t have enough memory (RAM) on your computer, this could make it crash.

Click on "disconnect" to stop the collection of tweets.

result plugin 1 en
Figure 21. Users mentioned in or retweeting tweets citing the search terms

 

b. Applying a layout while the tweets arrive.

You can see the users organize spatially in real time, while the tweets are being collected.

Just run Force Atlas 2 in the "Layout" panel. This will not interrupt the collection of tweets.

Running the Force Atlas layout while the tweets are arriving
Figure 22. Running the Force Atlas layout while the tweets are arriving

 

Shift to the data laboratory to view the data collected, in a spreadsheet format:

Switching to the data laboratory view
Figure 23. Switching to the data laboratory view

 

There, you can export nodes and relations ("edges") as csv files by clicking on "Export table".

c. Timeline

When you are finished with your stream, you can use the timeline feature to replay the stream of data and look at a particular time window.

This feature is still experimental and is higly subject to bug.

timeline enable
Figure 24. How to activate the Timeline

 

timeline
Figure 25. Example of Timeline

 

Extra Scripts

They use it !

  • Panteion University - Dept. of Communication, Media and Culture

  • Storyful

  • BBC Monitoring

  • Paris II

  • University Aberdeen

  • Athens university of economics and business / Οικονομικό Πανεπιστήμιο Αθηνών, Department of Management Science and Technology / Τμήμα Διοικητικής Επιστήμης και Τεχνολογίας

You can ping @totetmatt on twitter if you wish to expand the list ;)

to go further

Visit the Gephi group on Facebook to get help,

or visit the website for more tutorials

site stats