How to get a long list of followers

Feb 23, 2015 at 11:29 PM
Hi, I need to get long lists of followers (about 10 lists of more than 1 million followers each).

I tried to follow this](https://tweetinvi.codeplex.com/discussions/573102) but I get only the first X followers.
var maxObjectsToRetrieve= 200;
long lastCursor = -1;
var followerQuery = String.Format("https://api.twitter.com/1.1/followers/ids.json?user_id={0}", userId);


var results =    TwitterAccessor.ExecuteCursorGETCursorQueryResult<IIdsCursorQueryResultDTO>(followerQuery, maxObjectsToRetrieve, lastCursor);
lastCursor = results.Last().NextCursor;

 var returnedIds = TwitterAccessor.ExecuteCursorGETCursorQueryResult<IIdsCursorQueryResultDTO>(followerQuery, maxObjectsToRetrieve, lastCursor).SelectMany(x => x.Ids);
I also tried to wrap the last line with a while (lastCursor > 0) but I get a neverending loop (lastCursor does not change) and the list of followers always contains the first values.

Can you help me to understand what's wrong with this please?

Regards

Luca
Coordinator
Feb 24, 2015 at 1:24 PM
I would strongly advise you to use the new version of Tweetinvi (0.9.5.0) as it contains the RateLimitAwaiter that will help you a lot for big set of UserIds.
RateLimit.UseRateLimitAwaiter = true;
RateLimit.QueryAwaitingForRateLimit += (sender, args) =>
{
    Console.WriteLine("Waiting for RateLimits...");
};

var obama = User.GetUserFromScreenName("BarackObama");

var followerQuery = String.Format("https://api.twitter.com/1.1/followers/ids.json?user_id={0}", obama.Id);
var results = TwitterAccessor.ExecuteCursorGETCursorQueryResult<IIdsCursorQueryResultDTO>(followerQuery);
var followerIds = results.SelectMany(x => x.Ids);
Be aware that will take a lot and a lot of time if you are expecting millions of Ids. I'd suggest to use multiple accounts and aggregate results.

Linvi
Marked as answer by linvi on 2/24/2015 at 5:24 AM
Coordinator
Feb 24, 2015 at 2:00 PM
Please note that you will need the Source Code version of Tweetinvi. There is a bug in the release that I will fix tonight.

Linvi
Coordinator
Feb 24, 2015 at 10:50 PM
In the version 0.9.5.2 that I am about to release you will be able to do the following:
TweetinviEvents.QueryBeforeExecute += (sender, args) =>
{
    var queryRateLimit = RateLimit.GetQueryRateLimit(args.Query);
    RateLimit.AwaitForQueryRateLimit(queryRateLimit);
    Console.WriteLine(args.Query);
};

var obama = User.GetUserFromScreenName("BarackObama");

var followerQuery = String.Format("https://api.twitter.com/1.1/followers/ids.json?user_id={0}", obama.Id);
var results = TwitterAccessor.ExecuteCursorGETCursorQueryResult<IIdsCursorQueryResultDTO>(followerQuery);
var followerIds = results.SelectMany(x => x.Ids);
Linvi
Marked as answer by linvi on 2/24/2015 at 3:00 PM
Feb 25, 2015 at 11:27 PM
Hi, Thanks, the latest version works much better, it awaits and it seems that the cursored query works.
The only problem is that I cannot access to the results:

I am inside a Credentials block:
                    TwitterCredentials.ExecuteOperationWithCredentials(cred, () =>
                          {
I do this (I do not paste all the code, the await and so on)...
                                  var followerQuery = String.Format("https://api.twitter.com/1.1/followers/ids.json?user_id={0}", myPolitician.Id);
                                  var results = Tweetinvi.TwitterAccessor.ExecuteCursorGETCursorQueryResult<IIdsCursorQueryResultDTO>(followerQuery);
                                  var followerIds = results.SelectMany(x => x.Ids);
                                  logger.DebugFormat("Found {0} followers of {1}", followerIds.Count(), myPolitician.ScreenName);

                                  foreach (long f in followerIds)
                                 {
The logger.DebugFormat does not log, the foreach does not start and the program goes on with the instructions outside the ExecuteOperationWithCredentials block, so I completely miss the contents of the result variable.

Do you have any suggestion?
Is it possibile to access to the result varable also in the await event so I could save on db the partial results?

Hello & regards.

Luca
Coordinator
Feb 26, 2015 at 11:36 AM
Edited Feb 26, 2015 at 11:48 AM
Hello Luca,

What you want to do is a bit more complex then. You want to break when the RateLimit have completed their job and go back to the task when you can.
Here is how you can implement this:
// We do not want to await for RateLimits.
RateLimit.UseRateLimitAwaiter = false;

// Though we want the rate limits to be updated so that we can await for them manually.
TweetinviEvents.QueryAfterExecute += (sender, args) =>
{
    var queryRateLimit = RateLimit.GetQueryRateLimit(args.Query);
    if (queryRateLimit != null)
    {
        queryRateLimit.Remaining -= 1;
    }
};

var user = User.GetUserFromScreenName("BarackObama");

long nextCursor = -1;

// Run until you have finished to iterate over the Twitter Results
while (true)
{
    var followerQuery = String.Format("https://api.twitter.com/1.1/followers/ids.json?user_id={0}", user.Id);
    RateLimit.AwaitForQueryRateLimit(followerQuery);

    // Gets all the friends until the RateLimit is reached
    var results = TwitterAccessor.ExecuteCursorGETCursorQueryResult<IIdsCursorQueryResultDTO>(followerQuery, Int32.MaxValue, nextCursor);

    var followerIds = results.SelectMany(x => x.Ids).ToArray();

    // CHANGE HERE : Store the ids into the database
    //StoreFollowers(user.Id, followerIds);

    var previousCursor = results.Last().PreviousCursor;
    nextCursor = results.Last().NextCursor;

    if (previousCursor == nextCursor)
    {
        break;
    }
}
By the way it allowed me to find out that we sometimes want to Track the RateLimit but not await for them.
I have added a Work Item to implement this feature.

Cheers,
Linvi
Coordinator
Mar 13, 2015 at 1:39 AM
Hi Luca,

I just wanted to let you know that the feature for Tracking the RateLimits without awaiting for them is now available in the Source Code.

You can simply select the following option for the RateLimitTracker:
RateLimit.RateLimitTrackerOption = RateLimitTrackerOptions.TrackOnly;
Regards,
Linvi
Marked as answer by linvi on 3/12/2015 at 5:39 PM
Mar 13, 2015 at 8:07 AM

Hi L'invitation,
Thanks for the information.

Btw, do you have experience in severe twitter limits? I am downloading tons of users (the target is about 2mln for a thesis project) and for a while I could get 100 users for each call. In the last days I ask for 100 ids but I get only 3 in the answer, moreover after 3 such query I have to wait about 20 mins before querying again. It seems that I broke some rule and now I am severely capped.

Do you have any advice? I know that there are companies that sell data like that so I suppose there is a way to do

Il 13 Marzo 2015 01:39:39 "linvi" <[email removed]> ha scritto:

From: linvi

Hi Luca,

I just wanted to let you know that the feature for Tracking the RateLimits without awaiting for them is now available in the Source Code.

You can simply select the following option for the RateLimitTracker:
RateLimit.RateLimitTrackerOption = RateLimitTrackerOptions.TrackOnly;
Regards,
Linvi