Get all tweets of a particular #tag

Nov 4, 2014 at 10:34 AM
Can I get all tweets corresponding to a particular hashtag?
Nov 4, 2014 at 11:19 AM
I am using this
var searchParameter = Search.GenerateTweetSearchParameter("#myblog");
         searchParameter.MaximumNumberOfResults = 10000;
        searchParameter.TweetSearchFilter = TweetSearchFilter.All;
        //searchParameter.Since = new DateTime(2013, 12, 1);
        //searchParameter.Until = new DateTime(2013, 12, 11);
        //searchParameter.SinceId = 399616835892781056;
        //searchParameter.MaxId = 405001488843284480;

        var tweets = Search.SearchTweets(searchParameter);
        tweets.ForEach(t => Console.WriteLine(t.Text));
this is only returning 296 tweets not 10000...
Coordinator
Nov 4, 2014 at 11:38 AM
You need to be aware that Twitter is restricting the Search API so that it only returns the tweets matching your criteria of the past week sometimes extending up to 2 weeks.

Therefore the number of tweets from your search looks perfectly correct to me.
There is absolutely no way to search for tweets older than using the search API.

What you can do though is use the FilteredStream and store all the newly created tweets that matches your tag.

Linvi
Editor
Nov 4, 2014 at 11:39 AM
Edited Nov 4, 2014 at 11:40 AM
Hi,

I don't think it's possible to scrape all tweets corresponding to a particular hashtag.

The search API only goes one week in the past. So if you check the tweets you get for #MyBlog, you'll see the last tweet is actually from 27 Nov.

Ref: https://dev.twitter.com/rest/reference/get/search/tweets

I don't remember where I read the one week thing but from my experience it's true. The API page for search tweets says "Keep in mind that the search index may not go back as far as the date you specify here." for the until parameter.

imlokesh

edit: Linvi beat me to it. :)
Nov 4, 2014 at 12:06 PM
Edited Nov 4, 2014 at 12:10 PM
Thanks guys.. When using filteredStream how can I be sure that the tweets won't be duplicated? Also, are there any other ways to harvest old tweets..
Coordinator
Nov 4, 2014 at 12:19 PM
Edited Nov 4, 2014 at 12:20 PM
When using filteredStream how can I be sure that the tweets won't be duplicated?

The streams in Twitter does not send duplicates. They can send information about a tweet being deleted, but they will not send duplicates.

Also, are there any other ways to harvest old tweets..

Unfortunately I have never heard of such a solution, and I investigated this subject when I was doing research.
You can get old tweets from an external provider (Gnip or Datashift) but you will have to pay to use these services.

Linvi
Nov 4, 2014 at 12:26 PM
Thanks Linvi
Nov 5, 2014 at 10:34 AM
Edited Nov 5, 2014 at 10:34 AM
I am using below code to stream tweets for certain hashtags.. the tweets don't have the twitter handle of the poster..

var filteredStream = Stream.CreateFilteredStream();
        // add tracks 
        string [] split = tweetstostream.Split(',');
        foreach(string item in split)
        {

            filteredStream.AddTrack(item);

        }
        MongoDB db = new MongoDB();
        //filteredStream.AddFollow()
        filteredStream.MatchingTweetReceived += (sender, ts_args) => { db.MongoInsert(ts_args.Tweet.Text); };
        filteredStream.StartStreamMatchingAllConditions();
Nov 5, 2014 at 10:42 AM
got it.. didn't check the ts_args.tweet properties..